Supra-50M Packs a Heavyweight Punch in a Featherweight Package
SupraLabs released Supra-50M, a tiny 50-million-parameter language model that punches above its weight class. Trained from scratch on 20 billion tokens, it beats much larger models like GPT-2 on specific benchmarks despite being a fraction of the size. The release includes both Base and Instruct versions, optimized for users with limited hardware.
SupraLabs built the model using a modern Llama-style decoder-only transformer architecture. It was fed a curated diet of high-quality educational web text from the fineweb-edu dataset. The result is a compact model that requires minimal resources but still demonstrates surprisingly coherent text generation and reasoning abilities for its size class. The team also trained a custom tokenizer with a 32,000-word vocabulary specifically for this project.
Benchmark performance vs larger models
Key project metrics
- 50 million parameters, single GPU training.
- Outperforms GPT-2 (124M) on BLiMP score 76.3%.
- Scores 77.2% on SciQ beating SmolLM-135M.
- Beats GPT-2 on ARC-Easy and HellaSwag.
- Modern Llama architecture with grouped-query attention.
- Trained on fineweb-edu educational web dataset.
- Available in both Base and Instruct versions.
- Runs comfortably on consumer hardware without GPU.
Privacy-conscious professionals and serious hobbyists will find immediate utility here. The model loads easily on standard computers through Hugging Face pipelines with minimal setup code, making it accessible for those who want local text generation without cloud dependencies. Small agencies and indie developers can experiment with instruction-following and lightweight text completion without needing enterprise infrastructure.
What the developers are saying
The team at SupraLabs views this release as a stepping stone rather than a final product. They have already announced upcoming models in the pipeline, including Supra-124M with experimental reasoning capabilities and Supra-350M targeting coding tasks. Training on a single GPU with bfloat16 precision keeps the barrier to reproduction relatively low for others in the open-source community.
"The main concept of physics is iffy, and the idea that we can make things behave in a certain way. The most important part of physics is called quantum mechanics which states that all particles are made up of energy (energy) and matter (matter)." — Source: Hugging Face
Project Links
