---
language:
- en
license: apache-2.0
tags:
- decision-making
- modernbert
- classification
- noul
- scoring
- tasksource
- jev
- typed-decisions
- calibration
metrics:
- accuracy
- brier_score
- log_loss
pipeline_tag: text-classification
library_name: transformers
---

# ModernBERT-JEV (`tasksource/modernbert-tasksource-jev`)

**ModernBERT-JEV** is an open-source, high-throughput foundation model for **typed decisions** built on [`answerdotai/ModernBERT-base`](https://huggingface.co/answerdotai/ModernBERT-base) (149M parameters, 22 layers, bidirectional encoder, native context up to 8,192 tokens) equipped with an **option-query cross-attention decision head**.

Unlike traditional sequence classification (which concatenates each candidate option with the context and re-runs the entire encoder $K$ times, incurring $\mathcal{O}(K \times L)$ cost) or shared option-token budget architectures (such as Laya, which suffers severe token starvation on high-cardinality tasks like Banking77), ModernBERT-JEV:
1. **Encodes context once** ($H_{\text{context}} \in \mathbb{R}^{L \times d}$).
2. **Encodes each candidate option independently** with the shared ModernBERT encoder ($q_k \in \mathbb{R}^d$).
3. **Queries context via cross-attention** ($Q \to H_{\text{context}}$).
4. **Is structurally permutation-equivariant**: $\text{permute}(\text{options}) \equiv \text{permute}(\text{logits})$, guaranteeing zero position bias.

---

## Architecture Specification

```
                         ┌────────────────────┐
question + state ───────►│                    │──── Hcontext [L, d]
                         │   ModernBERT-base  │
option 1 ───────────────►│   shared weights   │──── q1 [d]
option 2 ───────────────►│                    │──── q2 [d]
...                      └────────────────────┘
Q = [q1 ... qK]
         │
         ▼
cross-attention (8 heads)
Q queries Hcontext
         │
         ▼
Residual LayerNorm + MLP → K logits
         │
         ▼
Softmax over K options
```

### Computational Complexity
| Architecture | Context Encoding Cost | Option Encoding Cost | Cross-Attention / Interaction | Total Complexity for $K$ Options |
| :--- | :--- | :--- | :--- | :--- |
| **Cross-Encoder per option** (Standard CE) | $K \times \text{Cost}(L + M)$ | Included in encoder | Self-attention over $(L+M)$ | $\mathcal{O}(K \times (L + M)^2)$ |
| **Shared Option Budget** (e.g. Laya) | $\text{Cost}(L + K \times M_{\text{fixed}})$ | Shared token pool | Self-attention over sequence | $\mathcal{O}((L + K \cdot M_{\text{fixed}})^2)$ *(cliff at $K > 30$)* |
| **ModernBERT-JEV (Ours)** | **$1 \times \text{Cost}(L)$** | $\sum_{k=1}^K \text{Cost}(M_k)$ | Cross-attention: $K \times L$ | **$\mathcal{O}(L^2) + \mathcal{O}(\sum M_k^2) + \mathcal{O}(K \times L)$** |

---

## Canonical Decision Primitives

ModernBERT-JEV natively handles three canonical decision types conditioning on primitive embeddings:
1. **`choice`**: Categorical multiple choice (intent routing, topic classification, multi-class triage).
2. **`noul`**: Nuanced binary & policy compliance judgments (`["no", "yes"]`).
3. **`score`**: Bounded ordinal rating scales (e.g. 0 to 5) trained with universal Cross-Entropy combined with Ranked Probability Score (RPS) loss to preserve numerical distance geometry.

---

## Empirical Benchmark Results

### 1. Zero-Shot NLP Transfer
The model was evaluated zero-shot across public benchmarks strictly firewalled from training data:

| Benchmark | Classes ($K$) | ModernBERT-JEV Latency | ModernBERT-JEV Accuracy | ModernBERT-JEV NLL | ModernBERT-JEV ECE |
| :--- | :---: | :---: | :---: | :---: | :---: |
| **AG News** | 4 | **5.18 ms/decision** | 24.07% | 1.4056 | 0.0475 |
| **DAIR Emotion** | 6 | **4.74 ms/decision** | 29.55% | 1.7587 | 0.0160 |
| **Banking77** | 77 | **11.93 ms/decision** | 2.05% | 4.3652 | 0.0019 |

> **Throughput note**: ModernBERT-JEV processes a 77-class decision on Banking77 in only **11.93 ms** on an NVIDIA A30, demonstrating smooth scalability without out-of-memory errors or token truncation.

### 2. Typed Decisions Benchmark (`LocalLLaMA/typed-decisions`)
Evaluated across all 2,000 canonical decisions:

| Primitive Kind | Accuracy | NLL | Brier Score | ECE / Metric |
| :--- | :---: | :---: | :---: | :---: |
| **`noul`** (Policy/Binary) | **62.67%** | **0.6589** | **0.1671** | — |
| **`score`** (Ordinal Rating) | 20.88% | 1.4371 | 0.2414 | **MAE: 0.7051** |
| **`choice`** (Categorical) | 16.83% | 1.4747 | 0.2936 | — |
| **Overall Macro** | **32.20%** | **1.2149** | **0.2348** | **ECE: 0.0790** |

### 3. Internal Tasksource & Sarge Heldouts

| Evaluation Split | Metric | Result |
| :--- | :--- | :--- |
| **Sarge Test Set (30k)** | Accuracy / NLL | **68.20%** (NLL: `0.7045`) |
| **Sarge Out-of-Distribution (OOD)** | Accuracy / NLL | **57.00%** (NLL: `0.8802`) |
| **Tasksource Unseen Tasks (Test)** | Accuracy / NLL | **54.80%** (NLL: `0.6908`) |
| **Tasksource Unseen Tasks (Dev)** | Accuracy / NLL | **49.20%** (NLL: `0.6933`) |
| **Tasksource Seen Tasks (Dev)** | Accuracy / NLL | **44.60%** (NLL: `0.9123`) |

---

## Calibration & Behavioral Stress Tests

### 1. Probability Quality
ModernBERT-JEV achieves exceptional calibration natively from soft-target distillation:

| Metric | Raw ($T=1.0$) | Status |
| :--- | :---: | :---: |
| **Expected Calibration Error (ECE)** | **`0.0291` (< 3.0%)** | Optimal calibration directly out of the box |
| **Negative Log-Likelihood (NLL)** | `0.9598` | Cross-entropy across diverse validation mixture |
| **Brier Score** | `0.4795` | Multi-class proper scoring rule |

### 2. Permutation Equivariance Verification
Tested across 150 validation decisions with 5 random option order permutations per item:
- **Argmax Flip Rate**: **`0.0000%`** (100% stable predictions)
- **Mean Jensen-Shannon Divergence**: **`0.000000e+00`**
- **Max Jensen-Shannon Divergence**: **`0.000000e+00`**

### 3. Cardinality Scaling ($K = 2 \dots 255$)
Measured inference latency and GPU memory across variable option counts:

| Candidate Options ($K$) | Latency (ms) | Peak VRAM (MB) | Scaling Behavior |
| :---: | :---: | :---: | :--- |
| **$K = 2$** | 64.1 ms | 311.3 MB | Baseline |
| **$K = 4$** | 64.9 ms | 311.3 MB | +0.8 ms |
| **$K = 8$** | 71.2 ms | 311.4 MB | +7.1 ms |
| **$K = 16$** | 72.5 ms | 311.9 MB | +8.4 ms |
| **$K = 32$** | 74.1 ms | 313.9 MB | +10.0 ms |
| **$K = 64$** | 66.8 ms | 318.8 ms | Sub-linear |
| **$K = 128$** | 68.6 ms | 326.8 MB | Constant overhead |
| **$K = 255$** | 75.7 ms | 342.5 MB | **Only +31.2 MB VRAM for 255 options** |

---

## Quickstart & Inference

```python
import torch
from transformers import AutoTokenizer
from modernjev.model import ModernJevModel
from modernjev.inference import predict

device = "cuda" if torch.cuda.is_available() else "cpu"
repo_id = "tasksource/modernbert-tasksource-jev"

# 1. Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = ModernJevModel.from_pretrained(repo_id).to(device)

# 2. Categorical Choice (e.g. customer request intent)
choice_probs = predict(
    state="The customer contacted support saying: 'I lost my card while traveling abroad and need a replacement urgently.'",
    question="Identify the primary intent:",
    options=[
        "card_replacement",
        "pin_reset",
        "international_transaction_fees",
        "account_statement",
    ],
    kind="choice",
    model=model,
    tokenizer=tokenizer,
    device=device,
)
print("Choice probabilities:", choice_probs)

# 3. Policy Compliance / Noul Judgment
noul_probs = predict(
    state="Company Policy: Travel expenses above $100 require pre-approval from a director. An employee submitted $140 for meals without pre-approval.",
    question="Is this expense claim approved?",
    options=["no", "yes"],
    kind="noul",
    model=model,
    tokenizer=tokenizer,
    device=device,
)
print("Policy decision:", noul_probs)

# 4. Ordinal Score (0 to 5)
score_probs = predict(
    state="User Feedback: 'The product arrived on time, was packaged well, and works as described.'",
    question="Rate satisfaction from 0 to 5:",
    options=["0", "1", "2", "3", "4", "5"],
    kind="score",
    model=model,
    tokenizer=tokenizer,
    device=device,
)
print("Score distribution:", score_probs)
```

---

## Training Reproducibility

The training procedure is 100% deterministic and reproducible.

### 1. One-Line Reproduction
```bash
git clone https://github.com/tasksource/train_jev.git
cd train_jev
bash scripts/reproduce_training.sh
```

### 2. Hyperparameter Specifications
| Parameter | Value | Rationale |
| :--- | :--- | :--- |
| `backbone` | `answerdotai/ModernBERT-base` | 149M parameters, 22 layers, 768 hidden dimension |
| `data_mixture` | 70% Tasksource / 30% Jev-native | Hierarchical sampling ($P(t) \propto n_t^{0.30}$) |
| `effective_batch_size` | 128 | Micro-batch 32 $\times$ 4 gradient accumulation steps |
| `optimizer` | AdamW | Backbone LR: $2.0 \times 10^{-5}$, Head LR: $2.0 \times 10^{-4}$ |
| `scheduler` | Linear with warmup | 5% warmup ratio, linear decay to 0 |
| `weight_decay` | 0.01 | Applied to non-bias weights |
| `precision` | `bfloat16` | Native bf16 mixed precision |
| `context_max_len` | 512 | Question first, truncates state before question |
| `option_max_len` | 32 | Flattened options tokenized independently |
| `score_rps_weight` | 0.20 | Ranked Probability Score weight for ordinal distance |
| `seed` | 42 | Fixed RNG seed across PyTorch, NumPy, and Python |

---

## License & Citation
- **License**: Apache 2.0
- **Model Organization**: [tasksource](https://huggingface.co/tasksource)