SmolGPT

Visit Tool

smolGPT is an Open Source & Models tool that provides a minimal PyTorch implementation for training small LLMs from scratch. It features efficient training, flash attention, and modern sampling techniques for educational purposes.

Claim this tool

2Views

At a glance

Pricing

Open Source

Free tier

Yes

API

Skill level

Technical

About

What is smolGPT?

smolGPT offers a minimal PyTorch implementation for training small Large Language Models (LLMs) from scratch, designed primarily for educational purposes and simplicity. It boasts a pure PyTorch codebase with no abstraction overhead, incorporating modern architectural elements like Flash Attention (when available), RMSNorm, SwiGLU, and optional Rotary embeddings (RoPE). The tool supports efficient training features including mixed precision (bfloat16/float16), gradient accumulation, learning rate decay with warmup, weight decay, and gradient clipping. It also includes built-in TinyStories dataset processing and SentencePiece tokenizer training integration, making it a comprehensive yet accessible platform for learning LLM development.

Best used for

Ideal for students and educators who need to understand the inner workings of LLMs and train their own small models. Especially valuable for those looking to experiment with modern architectural components like Flash Attention and various sampling techniques in a simplified environment.

Common actions

train LLMs

learn LLM architecture

implement deep learning models

experiment with AI

deepfakeopen-sourceworkflowslow-code/no-codeface swapping"AI Agents"github copilotcollaborationautomated workflow

Capabilities

Key features

Minimal PyTorch codebase
Flash Attention support
RMSNorm and SwiGLU
Efficient top-k/p/min-p sampling
Rotary embeddings (RoPE)
Mixed precision training
Gradient accumulation

Target Audience

studentteacher

Integrations

Not yet documented

Pricing & Plans

Open Source

Free

FAQs

What are the hardware requirements for running smolGPT?

smolGPT recommends Python 3.8+, PyTorch 2.0+ with CUDA, and a modern GPU for optimal performance. The provided training rig specs used an NVIDIA L4 Tensor Core GPU with 24 GB VRAM, 16 vCPUs, and 64 GB RAM.

Can I use smolGPT to train models on my own custom datasets?

Yes, smolGPT includes built-in dataset support and SentencePiece tokenizer training integration, allowing users to process and train models on their own custom datasets in addition to the provided TinyStories dataset.

Does smolGPT support pre-trained models?

Yes, smolGPT provides an option to use pre-trained models. Users can download a pre-trained checkpoint and tokenizer model, then run inference with their own prompts without needing to go through the full training cycle.

Trending

Subcategories trending in Coding & Development

Code Assistants DevOps & Infrastructure No-Code / Low-Code Testing & QA Backend & APIs Prompt Engineering

Trending

Also listed in

This tool also appears in

AI Agents & Automation › AI Frameworks & Infra Research & Education › Scientific Computing Research & Education › Study Assistants

Explore

Browse AI tools by category

Content & Design Productivity & Business Coding & Development AI Agents & Automation Research & Education Wellness & Lifestyle Career Development Marketing & Growth Data & Analytics Customer Support & CX Finance E-commerce