Rahvusarhiiv/ru_summariser

Russian summariser

Rahvusarhiiv/ru_summariser is an experimental Russian summarization model fine-tuned from RussianNLP/FRED-T5-Summarizer.

This revision replaces the earlier mBART-50 checkpoint. On a matched three-domain evaluation it improves the Ollama judge overall score by 0.47 points over FRED-T5 base, with equal severe factual error counts. It is a research release, not a state-of-the-art claim.

Intended Use

• Summarization of Russian news, encyclopedic, and general informational text.
• Batch or offline summarization where outputs can be checked against the source.
• Not intended as a sole source of factual truth or for unreviewed legal, medical, financial, or archival descriptions.

How to Use

The model was trained with a length-control prefix. medium is the evaluated default. The helper below also applies the source-only salience selection and sentence-boundary cleanup used for the reported release scores.

import re
from collections import Counter

import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

repo_id = "Rahvusarhiiv/ru_summariser"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForSeq2SeqLM.from_pretrained(repo_id)
model.eval()

WORD_RE = re.compile(r"[\w-]+", re.UNICODE)
SENTENCE_RE = re.compile(r"(?<=[.!?])\s+|\n+")


def select_source_evidence(text, max_words=700, trigger_words=760):
    if len(WORD_RE.findall(text)) <= trigger_words:
        return text

    sentences = [" ".join(part.split()) for part in SENTENCE_RE.split(text) if part.strip()]
    tokenized = [[token.casefold() for token in WORD_RE.findall(sentence)] for sentence in sentences]
    frequency = Counter(
        token for sentence in tokenized for token in sentence if len(token) >= 4
    )
    scored = []
    for index, tokens in enumerate(tokenized):
        if not tokens or (len(tokens) < 6 and index >= 3):
            continue
        unique = {token for token in tokens if len(token) >= 4}
        if not unique:
            continue
        lexical = sum(frequency[token] for token in unique) / len(unique)
        lead = max(0.0, 3.0 - index * 0.25)
        length_penalty = 0.02 * max(0, len(tokens) - 45)
        scored.append((lexical + lead - length_penalty, index))

    selected = set()
    total = 0
    for _, index in sorted(scored, reverse=True):
        sentence_words = len(tokenized[index])
        if total and total + sentence_words > max_words:
            continue
        selected.add(index)
        total += sentence_words
        if total >= max_words:
            break
    return " ".join(sentences[index] for index in sorted(selected or {0}))


def remove_incomplete_tail(text):
    text = text.strip()
    if not text or text[-1] in ".!?":
        return text
    end = max(text.rfind(mark) for mark in ".!?")
    return text if end < 0 else text[: end + 1].strip()


text = (
    "Городские власти сообщили о запуске новой программы поддержки общественного "
    "транспорта. По словам чиновников, дополнительные средства направят на обновление "
    "автобусов и повышение доступности маршрутов."
)
prompt = "summary_length: medium\n\n" + select_source_evidence(text)
inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=768)

with torch.inference_mode():
    summary_ids = model.generate(
        **inputs,
        max_new_tokens=96,
        min_new_tokens=0,
        num_beams=4,
        length_penalty=0.7,
        no_repeat_ngram_size=3,
        early_stopping=True,
    )

summary = tokenizer.batch_decode(summary_ids, skip_special_tokens=True)[0]
print(remove_incomplete_tail(summary))

For CUDA inference, move the model and tokenized inputs to the same CUDA device. The checkpoint is also directly compatible with pipeline("summarization", model=repo_id), but the generic pipeline does not apply the evaluated prefix, salience selector, or incomplete-tail cleanup automatically.

Recommended Generation Settings

setting	value
input prefix	summary_length: medium followed by a blank line
maximum input tokens	768
maximum new tokens	96
minimum new tokens	0
beams	4
length penalty	0.7
repeated n-gram block	3
early stopping	true

The checked-in generation_config.json contains the generation defaults. The input prefix and optional preprocessing remain application-side behavior.

Training Data

The training set contains 59,489 real source-summary pairs and the validation set contains 3,000. The mix is predominantly supported Russian news, with a capped general-domain WikiLingua component. No synthetic rows were used in this revision.

source family	training rows
Gazeta	14,500
Russian XL-Sum	10,000
MLSUM	8,000
Taiga-derived news	7,000
WikiLingua	6,716
LR-Sum	6,000
SumNews	5,000
Lenta-derived news	2,273

Training preparation applied:

• source-summary support filtering and low/medium risk bucketing;
• exact and near-source deduplication;
• source-level caps without oversampling;
• source-only evidence selection for long inputs;
• randomized metadata presentation when supported metadata was required;
• short, medium, and long length-control prefixes;
• exact/hash/shingle exclusion against protected Wiki4, GDELT, and supported-news benchmarks.

The final benchmark exclusion audit found zero overlaps across 59,489 training and 3,000 validation rows. Average training-pair lexical support was 0.715.

Evaluation

Automatic metrics use 30 matched source IDs per benchmark. Judge metrics use 10 matched source IDs per benchmark and gemma4:31b through Ollama at temperature zero.

benchmark	ROUGE-L	coverage	source precision	words	judge factuality	judge coverage	judge overall
Wiki4 strict v2	0.431	0.408	0.970	50.2	4.50	3.50	4.00
GDELT news	0.202	0.415	0.955	56.0	5.00	4.30	4.50
Supported news	0.228	0.321	0.812	48.6	4.30	4.30	4.20
Macro average	0.287	0.381	0.912	51.6	4.60	4.03	4.23

Comparison with FRED-T5 base

model	macro ROUGE-L	coverage	source precision	judge overall
This model	0.287	0.381	0.912	4.23
FRED-T5 base	0.270	0.300	0.860	3.77

Across the 30 paired judge examples, this model had 9 wins, 18 ties, and 3 losses. The average overall delta was +0.47, coverage delta was +0.70, and factuality delta was 0.00. Both models had three factuality scores of 2 or lower.

ROUGE and lexical support metrics are proxies. The local LLM judge is useful for matched comparison but should not be treated as an absolute or human-equivalent quality measure.

Limitations and Failure Modes

• The model can hallucinate or misstate entities, numbers, dates, and relationships.
• It is more verbose than FRED-T5 base on short-reference news.
• Long-document quality depends on input selection. The evaluated selector can omit relevant material, while raw tokenizer truncation can overemphasize the beginning.
• The 96-token ceiling is frequently reached. Sentence cleanup avoids an incomplete final clause but may remove useful content from the last generated sentence.
• Length-control prefixes influence training behavior, but exact requested lengths are not guaranteed.
• The data mix is news-heavy; performance may degrade on dialogue, highly technical writing, fiction, or legal text.
• Factual claims should be checked against the source before publication.

Bias, Risks, and Ethical Considerations

The model can reproduce biases, framing, and omissions present in its source corpora. Training sources span different publishers, periods, collection methods, and upstream terms. Users are responsible for checking whether their intended use is compatible with those terms.

CO2 Emissions

Training energy and emissions were not instrumented with CodeCarbon or an equivalent tool, so no numeric CO2 estimate is reported. This revision fine-tuned an existing FRED-T5 checkpoint for 9,500 steps (2.55 epochs) on one local NVIDIA GB10 rather than pretraining a model from scratch.

Model Tree

• Base model: RussianNLP/FRED-T5-Summarizer
• Relationship: full-parameter sequence-to-sequence fine-tune
• This revision: grounded, deduplicated Russian news/general data with length-control prefixes

License

The model card uses license: other because the training mix contains upstream sources with different or unspecified terms. Verify the relevant source terms before redistribution or commercial use.