Transformers documentation
SarvamMoE
This model was contributed to Hugging Face Transformers on 2026-11-20.
SarvamMoE
SarvamMoE is the architecture of Sarvam-30B, a 30B parameter Mixture of Experts (MoE) language model developed by Sarvam AI. Its larger sibling, Sarvam-105B, uses SarvamMLA.
Key architectural features:
- Grouped-query attention with QK normalization and full rotary position embeddings.
- Sparse Mixture of Experts: 128 routed experts with 6 active per token, plus 1 shared expert. The first layer uses a dense MLP.
- Sigmoid routing with an expert bias correction and a routed scaling factor.
The architecture is the same as GLM-4-MoE, with different parameter and config names. The auto classes load it with the GLM-4-MoE implementation (Glm4MoeModel and Glm4MoeForCausalLM), so only SarvamMoEConfig is specific to SarvamMoE. The released checkpoints are converted to the GLM-4-MoE names when loading and back when saving.
The example below demonstrates how to generate text with the AutoModelForCausalLM class.
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("sarvamai/sarvam-30b")
model = AutoModelForCausalLM.from_pretrained("sarvamai/sarvam-30b", device_map="auto")
messages = [{"role": "user", "content": "Explain the Pythagorean theorem in one sentence."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True)
output = model.generate(**inputs.to(model.device), max_new_tokens=64)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1] :], skip_special_tokens=True))SarvamMoEConfig
class transformers.SarvamMoEConfig
< source >( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: str | torch.dtype | None = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: Literal['regression', 'single_label_classification', 'multi_label_classification'] | None = Nonevocab_size: int = 262144hidden_size: int = 4096intermediate_size: int = 8192num_hidden_layers: int = 19num_attention_heads: int = 64num_key_value_heads: int = 4hidden_act: str = 'silu'max_position_embeddings: int = 131072initializer_range: float = 0.006rms_norm_eps: float = 1e-06use_cache: bool = Truetie_word_embeddings: bool = Falserope_parameters: transformers.modeling_rope_utils.RopeParameters | dict | None = Noneattention_bias: bool = Falseattention_dropout: float | int = 0.0moe_intermediate_size: int = 1024num_experts_per_tok: int = 6output_router_logits: bool = Falsen_shared_experts: int = 1n_routed_experts: int = 128routed_scaling_factor: float = 2.5n_group: int = 1topk_group: int = 1first_k_dense_replace: int = 1norm_topk_prob: bool = Trueuse_qk_norm: bool = Truebos_token_id: int | None = Noneeos_token_id: int | list[int] | None = 1pad_token_id: int | None = 0num_mtp_layers: int = 0head_dim: int = 64 )
Parameters
- vocab_size (
int, optional, defaults to262144) — Vocabulary size of the model. Defines the number of different tokens that can be represented by theinput_ids. - hidden_size (
int, optional, defaults to4096) — Dimension of the hidden representations. - intermediate_size (
int, optional, defaults to8192) — Dimension of the MLP representations. - num_hidden_layers (
int, optional, defaults to19) — Number of hidden layers in the Transformer decoder. - num_attention_heads (
int, optional, defaults to64) — Number of attention heads for each attention layer in the Transformer decoder. - num_key_value_heads (
int, optional, defaults to4) — This is the number of key_value heads that should be used to implement Grouped Query Attention. Ifnum_key_value_heads=num_attention_heads, the model will use Multi Head Attention (MHA), ifnum_key_value_heads=1the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out this paper. If it is not specified, will default tonum_attention_heads. - hidden_act (
str, optional, defaults tosilu) — The non-linear activation function (function or string) in the decoder. For example,"gelu","relu","silu", etc. - max_position_embeddings (
int, optional, defaults to131072) — The maximum sequence length that this model might ever be used with. - initializer_range (
float, optional, defaults to0.006) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices. - rms_norm_eps (
float, optional, defaults to1e-06) — The epsilon used by the rms normalization layers. - use_cache (
bool, optional, defaults toTrue) — Whether or not the model should return the last key/values attentions (not used by all models). Only relevant ifconfig.is_decoder=Trueor when the model is a decoder-only generative model. - tie_word_embeddings (
bool, optional, defaults toFalse) — Whether to tie weight embeddings according to model’stied_weights_keysmapping. - rope_parameters (
Union[~modeling_rope_utils.RopeParameters, dict], optional) — Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value forrope_thetaand optionally parameters used for scaling in case you want to use RoPE with longermax_position_embeddings. - attention_bias (
bool, optional, defaults toFalse) — Whether to use a bias in the query, key, value and output projection layers during self-attention. - attention_dropout (
Union[float, int], optional, defaults to0.0) — The dropout ratio for the attention probabilities. - moe_intermediate_size (
int, optional, defaults to1024) — Intermediate size of the routed expert MLPs. - num_experts_per_tok (
int, optional, defaults to6) — Number of experts to route each token to. This is the top-k value for the token-choice routing. - output_router_logits (
bool, optional, defaults toFalse) — Whether or not the router logits should be returned by the model. Enabling this will also allow the model to output the auxiliary loss, including load balancing loss and router z-loss. - n_shared_experts (
int, optional, defaults to1) — Number of shared experts. - n_routed_experts (
int, optional, defaults to128) — Number of routed experts. - routed_scaling_factor (
float, optional, defaults to2.5) — Scaling factor or routed experts. - n_group (
int, optional, defaults to 1) — Number of groups for routed experts. - topk_group (
int, optional, defaults to1) — Number of selected groups for each token (for each token, ensuring the selected experts is only withintopk_groupgroups). - first_k_dense_replace (
int, optional, defaults to 1) — Number of dense layers in shallow layers(embed->dense->moe->moe…->lm_head). --k dense layers—/ - norm_topk_prob (
bool, optional, defaults toTrue) — Whether to normalize the weights of the routed experts. - use_qk_norm (
bool, optional, defaults toTrue) — Whether to use query-key normalization in the attention. - bos_token_id (
int, optional) — Token id used for beginning-of-stream in the vocabulary. - eos_token_id (
Union[int, list[int]], optional, defaults to1) — Token id used for end-of-stream in the vocabulary. - pad_token_id (
int, optional, defaults to0) — Token id used for padding in the vocabulary. - num_mtp_layers (
int, optional, defaults to 0) — Number of Multi-Token Prediction (MTP) modules available to append after the base transformer model. The released checkpoints do not include MTP weights. - head_dim (
int, optional, defaults to64) — The attention head dimension. If None, it will default to hidden_size // num_attention_heads
This is the configuration class to store the configuration of a Glm4MoeModel. It is used to instantiate a Sarvam Moe model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the sarvamai/sarvam-30b
Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.
Example:
>>> from transformers import AutoModelForCausalLM, SarvamMoEConfig
>>> # Initializing a Sarvam-30B style configuration
>>> configuration = SarvamMoEConfig()
>>> # Initializing a model (which uses the GLM-4-MoE implementation) from that configuration
>>> model = AutoModelForCausalLM.from_config(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config