NagaYu/tessera-span-lid

Tessera -- span-level multi-label language identification

Labels every span of a document with a set of languages, so low-resource text inside a mixed page survives a corpus filter that would otherwise discard the whole document or file it under one language.

This does not replace GlotLID / OpenLID / OpenLID-v3. They are the comparison, they are the state of the art at document-level LID, and they are Tessera's upstream: their output is an optional prior input. Tessera is one extra pipeline stage.

Model

• 8.68M parameters, 34.7 MB fp32
• 579 labels (479 languages + romanised *_Latn variants + und)
• Hashed char n-grams (1-5) + word unigrams -> EmbeddingBag; a dense script/shape channel; a document-prior channel; a 1-layer BiGRU over the span sequence with a residual skip; sigmoid multi-label head.
• Trained with the prior zeroed on half of all steps, so one checkpoint serves both the with-prior and without-prior settings.

Results

system	micro-F1	macro-F1	recovery rate
A_fasttext176_document	0.2803	0.0505	0.0
B_glotlid_document	0.3031	0.265	0.0
C_glotlid_per_line	0.406	0.4788	0.1713
D_tessera_no_prior	0.7712	0.6487	0.8621
E_tessera_glotlid_prior	0.7795	0.6611	0.8396

recovery rate = of mid/tail-resource spans in documents a document-level baseline would have discarded or kept under the wrong language, the fraction correctly labelled.

Intended use and limits

Intended for corpus curation: deciding which spans of a crawled document to keep and under which language. Not intended as a general-purpose LID API, and not better than GlotLID at labelling a whole document.

Known limits, measured rather than guessed:

• Romanised text remains weak. See the repository's romanised-slice results.
• Above ~80 characters, GlotLID per line is competitive or better. Tessera is built for the short end.
• Passing the document prior slightly lowers the recovery rate even though it raises aggregate F1, because the prior pulls predictions toward the document majority language. If recovery is the goal, do not pass the prior.
• All headline numbers are measured on synthetic data. The reason, and what real data can and cannot show, is documented in the repository.

Ethics

Language identification decides whose text survives a corpus build. Mislabelling a language removes it from the datasets that later models learn from. The dataset card lists every covered language with its known weaknesses; tail-language errors are reported separately rather than averaged away; und is a real label so the model can decline to guess.

Repository: NagaYu/tessera-span-lid.