TextEmbedding3SmallSentimentHead

In case you needed a sentiment analysis classifier on top of embeddings from OpenAI embeddings model.

Model Description

• What this is: A compact PyTorch classifier head trained on top of text-embedding-3-small (1536-dim) to predict sentiment: negative, neutral, positive.
• Data: Preprocessed from the Kaggle Sentiment Analysis Dataset.
• Metrics (val): F1 macro ≈ 0.89, Accuracy ≈ 0.89 on a held-out validation split.
• Architecture: Simple MLP head (256 hidden units, dropout 0.2), trained for 5 epochs with Adam.

Input/Output

• Input: Float32 tensor of shape [batch, 1536] (OpenAI text-embedding-3-small embeddings).
• Output: Logits over 3 classes. Argmax → {0: negative, 1: neutral, 2: positive}.

Usage

from transformers import AutoModel
import torch

# Load model
model = AutoModel.from_pretrained(
    "marcovise/TextEmbedding3SmallSentimentHead", 
    trust_remote_code=True
).eval()

# Your 1536-dim OpenAI embeddings
embeddings = torch.randn(4, 1536)  # batch of 4 examples

# Predict sentiment
with torch.no_grad():
    logits = model(inputs_embeds=embeddings)["logits"]  # [batch, 3]
    predictions = logits.argmax(dim=1)  # [batch]
    # 0=negative, 1=neutral, 2=positive

print(predictions)  # tensor([1, 0, 2, 1])

Training Details

• Training data: Kaggle Sentiment Analysis Dataset
• Preprocessing: Text → OpenAI embeddings → 3-class labels {negative: 0.0, neutral: 0.5, positive: 1.0}
• Architecture: 1536 → 256 → ReLU → Dropout(0.2) → 3 classes
• Optimizer: Adam (lr=1e-3, weight_decay=1e-4)
• Loss: CrossEntropyLoss with label smoothing (0.05)
• Epochs: 5

Intended Use

• Quick, lightweight sentiment classification for short text once embeddings are available.
• Works well for general sentiment analysis tasks similar to the training distribution.

Limitations

• Trained on a specific sentiment dataset; may have domain bias.
• Requires OpenAI text-embedding-3-small embeddings as input.
• Not safety-critical; evaluate before production use.
• May reflect biases present in the training data.

License

MIT