
Spark 2.0-code-base (pretrained base model)
The base model behind Spark 2.0-code: 322M parameters, trained completely from scratch on Python code by one person on a single RTX 3090. It was pretrained on about 18.4B tokens and has no instruction tuning.
This is not a chat model. It does not follow tasks. You give it the beginning of Python code (a function signature with a docstring, an import block, a class header) and it continues it. For instructions in plain English use Spark 2.0-code or Spark 2.0-code (stage 1).
Why release a base model: to fine-tune it yourself, to study it, or to compare it with the instruction-tuned versions.
Honest summary. It completes functions reasonably (HumanEval 34.8%), but it does not know when to stop: after finishing one function it usually goes on and invents more functions. Cut its output yourself (for example at the next top-level
def/class), or use a stop condition.
Quick start
Needs an NVIDIA GPU (CUDA). The model uses Gated DeltaNet layers from flash-linear-attention, which run on GPU only.
pip install torch flash-linear-attention triton tokenizers safetensors
python generate.py 'def is_prime(n):\n # Return True if n is prime.\n'
generate.py prints your prompt and the continuation (greedy, up to 300 tokens). There is no prompt template: the text
you give is simply continued.
Model
| Parameters | 322M (embeddings tied) |
| Architecture | hybrid: Gated DeltaNet layers + full attention every 4th layer, 24 layers, d_model 1024 |
| Context | 4096 tokens |
| Vocabulary | 16 384 (own BPE tokenizer, Python-focused) |
| Pretraining | about 18.4B tokens of Python (35 070 steps), Muon + AdamW |
| Validation loss | 0.50 |
| Weights | bf16, model.safetensors, 0.65 GB |
Results
My own harness, greedy decoding, code is run in a sandbox. HumanEval is completion-style, which is what a base model does.
| Base (this model) | SFT stage 1 | SFT stage 2 | |
|---|---|---|---|
| HumanEval pass@1 (164 tasks) | 34.8% | 52.4% | 47.6% |
HumanEval and MBPP were kept out of the training data: documents sharing a 13-gram with any task or reference solution were removed.
Limitations
- Not a chat or instruction model; it continues text.
- Does not stop after a function; trim the output.
- It sometimes invents library functions and misses imports.
- It only knows Python.
- A language model: it can write insecure or wrong code. Review before running.
Training data and license
License: CC BY-NC 4.0 (non-commercial, with attribution). The pretraining data includes KodCode (CC BY-NC 4.0), so I do not offer the model for commercial use. For a commercial product, contact the author first.
Licenses are as shown on the Hugging Face dataset pages when I checked.
| Source | License |
|---|---|
| swallow-code-v2 (Python from The Stack v2, rewritten and filtered) | Apache-2.0 |
| OpenCodeInstruct (only solutions that passed all unit tests) | CC BY 4.0 |
| KodCode-V1 | CC BY-NC 4.0 |
| opc-annealing-corpus (synthetic Python snippets) | ODC-By |
Some of these datasets were themselves generated by other language models, so their own terms may apply. This card is my best effort, not legal advice.
Author
Trained from scratch as a personal project. Feedback and bug reports are welcome.
If you like Spark and want to support the next models (GPU rent for bigger ones), you can donate here: https://boosty.to/sparkcode/donate
- Downloads last month
- -