sbx/KB-bert-swedish_CEFR-classification

Model Card for Model ID

This model was developed as part of the Computational SLA working group at Språkbanken Text. It takes essays written in Swedish by second language learners and assigns them one of the CEFR levels. Of note is that it only uses the first five levels of the scale (A1 to C1), ignoring level C2 due to both lack of training data and it measuring things differently than the other levels do.

Most of the information contained in this Model Card comes from the paper that introduced the present model. Feel free to check it out for more in-depth information.

Model Details

Model Description

• Developed by: Språkbanken Text, as part of the Computational SLA group
• Shared by: Ricardo Muñoz Sánchez (rimusa)
• Model type: BERT for text classification
• Language(s): Swedish
• License: GPL-3.0
• Finetuned from model: KB/bert-base-swedish-cased

Model Sources [optional]

• Paper: Jingle BERT, Jingle BERT, Frozen All the Way: Freezing Layers to Identify CEFR Levels of Second Language Learners Using BERT (link)

Uses

Direct Use

This model is meant for building demos and tools that display an approximate CEFR level of language learner texts. It is important to note that the predictions from this model should be taken as illustrative rather than as autorithative.

Out-of-Scope Use

This model should not be deployed on high-stakes situations, such as actual language assessment or decision-making regarding migration, work, education, etc. It has relatively low performance compared to what would be needed for such situations, not to speak of potential issues regarding accountability.

Bias, Risks, and Limitations

The model has been trained in a heterogeneous dataset of Swedish language learner essays. While this exposed the model to a variety of contexts, it also means that there might be biases in terms of topics and format.

We are currently studying the impact that the essay authors' first language(s) has on these models. This model card will be updated once we have more results on this regard.

Citation [optional]

BibTeX:

@inproceedings{sanchez-etal-2024-jingle,
    title = "Jingle {BERT}, Jingle {BERT}, Frozen All the Way: Freezing Layers to Identify {CEFR} Levels of Second Language Learners Using {BERT}",
    author = "Mu{\~n}oz S{\'a}nchez, Ricardo  and
      Alfter, David  and
      Dobnik, Simon  and
      Szawerna, Maria Irena  and
      Volodina, Elena",
    editor = {Gaillat, Thomas  and
      Mallart, Cyriel  and
      Moreau, Fabienne  and
      Li, Jen-Yu  and
      Drouet, Griselda  and
      Alfter, David  and
      Volodina, Elena  and
      J{\"o}nsson, Arne},
    booktitle = "Proceedings of the 13th Workshop on Natural Language Processing for Computer Assisted Language Learning",
    month = oct,
    year = "2024",
    address = "Rennes, France",
    publisher = "LiU Electronic Press",
    url = "https://aclanthology.org/2024.nlp4call-1.11/",
    pages = "137--152"
}

APA:

Ricardo Muñoz Sánchez, David Alfter, Simon Dobnik, Maria Irena Szawerna, and Elena Volodina. 2024. Jingle BERT, Jingle BERT, Frozen All the Way: Freezing Layers to Identify CEFR Levels of Second Language Learners Using BERT. In Proceedings of the 13th Workshop on Natural Language Processing for Computer Assisted Language Learning, pages 137–152, Rennes, France. LiU Electronic Press.

Model Card Authors

Ricardo Muñoz Sánchez (rimusa)

Model Card Contact

For more information about the model or the present Model Card, you can reach out to:

• Ricardo Muñoz Sánchez ([mailto:ricardo.munoz.sanchez@gu.se])
• Elena Volodina ([mailto:elena.volodina@gu.se])