fa_ent_news_md

Persian named entity recognizer optimized for CPU, with its own internal tok2vec. Labels: PER, LOC, ORG, DAT, MON, TIM, PCT.

Install

pip install https://huggingface.co/Phazel/fa_ent_news_md/resolve/main/fa_ent_news_md-3.8.0-py3-none-any.whl

Use

import spacy

nlp = spacy.load("fa_ent_news_md")
doc = nlp("شرکت ایران خودرو اعلام کرد که تولید خود را افزایش می‌دهد.")

print([(e.text, e.label_) for e in doc.ents])

Accuracy

Scored with spacy benchmark accuracy on the held-out PerDT test split.

Metric	Score
NER precision	76.56
NER recall	72.95
NER F-score	74.71

Per entity label:

Label	P	R	F
DAT	71.79	81.16	76.19
LOC	86.15	82.05	84.05
MON	88.89	80.00	84.21
ORG	72.59	68.06	70.25
PCT	50.00	25.00	33.33
PER	70.91	65.66	68.18
TIM	66.67	66.67	66.67

Other packages in this family

Pipeline	Tier	LAS	ENTS_F	Wheel
fa_dep_news_sm	sm	85.15	-	7.9 MB
fa_core_news_sm	sm	85.15	71.87	13.5 MB
fa_ent_news_sm	sm	-	71.87	5.9 MB
fa_dep_news_md	md	86.34	-	62.6 MB
fa_core_news_md	md	86.34	74.71	68.5 MB
fa_ent_news_md (this one)	md	-	74.71	60.6 MB
fa_dep_news_lg	lg	86.60	-	229.3 MB
fa_core_news_lg	lg	86.60	75.94	235.2 MB
fa_ent_news_lg	lg	-	75.94	227.3 MB
fa_core_news_trf	trf	90.79	82.89	608.2 MB

Tiers: sm hash embeddings, no vectors, md 50k x 300d floret vectors, lg 200k x 300d floret vectors, trf fine-tuned transformer, GPU recommended.

Standalone vector tables, usable as --paths.vectors for your own training:

Vectors	Rows	Used by	Wheel
fa_floret_400k	50,000	md tier	54.5 MB
fa_floret_full_wiki	50,000	no shipped pipeline	54.9 MB
fa_floret_wiki_200k	200,000	lg tier	221.3 MB

Training scripts, configs and evaluation: https://github.com/Fazel94/spacy-persian.

Sources

Source	Author	Licence
UD_Persian-PerDT NER layer (not-to-release/Dadegan with NER tag/)	PerDT authors, tagged with Beheshti-NER (Taher, Hoseini, Shamsfard 2020)	CC BY-SA 4.0
spaCy lang/fa language data (stop words originally from HAZM)	Explosion and spaCy contributors	MIT
fa_floret static vectors (50k rows x 300d, floret mode, 400,000 Persian Wikipedia articles, trained with floret-torch)	Kiyarash Fazeli	CC BY-SA 4.0

Notes

Trained on UD_Persian-PerDT, licensed CC BY-SA 4.0; this pipeline is therefore distributed under CC BY-SA 4.0 with attribution to the treebank authors. The ner component is trained on the NER layer shipped in UD_Persian-PerDT's not-to-release/ directory, so it shares the treebank's genre, tokenization and licence. Those labels are SILVER: the treebank README states they were produced by the BERT-based Beheshti-NER tagger (Taher et al., 2020) with manual corrections to extend recall. They were transferred onto this pipeline's tokenization by difflib alignment at a 99.86% transfer rate (scripts/transfer_perdt_ner.py); spans that could not be aligned exactly were dropped rather than guessed. Labels PER, LOC, ORG and DAT have 1,300 or more training examples each; MON (205), TIM (135) and PCT (121) are thin and their scores in performance.ents_per_type should be read before relying on them. This is the md tier: identical architecture to the sm pipeline plus static floret vectors (50,000 rows x 300 dimensions, minn=maxn=5, hash_count=2) trained with floret-torch on 400,000 Persian Wikipedia articles (fawiki, WikiExtractor --no-templates, spaCy blank('fa') tokenization, ~163M tokens). floret hashes subwords into a fixed table, so there are no out-of-vocabulary tokens and token.has_vector is always True. That matters for Persian, where inconsistent ZWNJ (U+200C) usage splits one word across several surface forms (mi-ravad written joined, with ZWNJ, or with a space) that a classic word-vector table would miss.