Training Gym SDK

📖 Documentation · API Reference

Modal Training Gym is a Python SDK for RL post-training on Modal.

The library takes care of infrastructure concerns such as Cluster topology, Ray/NCCL bring-up, volume mounts, checkpointing, and serving for eval and rollouts, so you can focus on training models.

Quickstart

Requirements:

• Python 3.12

Install with pip:

pip install -q git+https://github.com/modal-projects/training-gym.git@main

Or pin it in pyproject.toml for uv:

training-gym = { git = "https://github.com/modal-projects/training-gym.git", branch = "main" }

Authenticate with Modal:

modal setup

Then, it’s as easy to use as:

from modal_training_gym import (
    HuggingFaceDataset,
    Qwen3_4B,
    Qwen3_4b_Recipe,
    TrainConfig,
)


class MathDataset(HuggingFaceDataset):
    hf_repo = "zhuzilin/dapo-math-17k"
    input_key = "prompt"
    label_key = "label"
    output_format = "jsonl"
    apply_chat_template = True


config = TrainConfig(
    model=Qwen3_4B(),
    dataset=MathDataset(n_rows=120),
    recipe=Qwen3_4b_Recipe(
        gpu_type="H100",
        actor_num_nodes=1,
        actor_num_gpus_per_node=8,
        tensor_model_parallel_size=1,
        sequence_parallel=False,
        rollout_num_gpus=8,
        rollout_num_gpus_per_engine=1,
        colocate=True,
        num_rollout=1,
        n_samples_per_prompt=4,
        rollout_batch_size=8,
        rollout_max_response_len=2048,
        max_tokens_per_gpu=4096,
        sglang_mem_fraction_static=0.6,
        rm_type="deepscaler",
    ),
)
run = config.launch()
print(run.training_run_id)

While that’s running, you can view the run in the dashboard:

training-gym setup  # first-time deploy
training-gym open  # opens in your browser

Agent set-up

Install the Training Gym skill bundle into your project:

training-gym skills install

This installs all bundled skills—agent-driven-training, example-validation, modal-infrastructure, model-support, and training-gym-overview—under the cross-tool .agents/skills/ directory. It also links each skill into .claude/skills/ when that path is safe to manage. Canonical and Claude-side conflicts are preserved with a warning without blocking unrelated skills. Run the command again with --force to replace existing canonical copies or manageable Claude child paths.

This repository also includes an AGENTS.md and the source skills/ directory that teach agents how to use and develop the framework.

Agents browsing the docs can start at the llms.txt.

Models

Models with a built-in training recipe, with the ModelConfig and recipe classes to import for each.

Model	Framework	ModelConfig	Recipe
Qwen/Qwen3-0.6B	slime	Qwen3_0_6B	Qwen3_0_6b_Recipe
Qwen/Qwen3-1.7B	slime	Qwen3_1_7B	Qwen3_1_7b_Recipe
Qwen/Qwen3-4B	slime	Qwen3_4B	Qwen3_4b_Recipe
Qwen/Qwen3-8B	slime	Qwen3_8B	Qwen3_8b_Recipe
Qwen/Qwen3-ASR-1.7B	slime	Qwen3_ASR_1_7B	Qwen3_ASR_1_7b_Recipe
Qwen/Qwen3-VL-8B-Instruct	slime	Qwen3_VL_8B	Qwen3_VL_8b_Recipe
Qwen/Qwen3.5-0.8B	slime	Qwen3_5_0_8B	Qwen3_5_0_8b_Recipe
Qwen/Qwen3.5-2B	slime	Qwen3_5_2B	Qwen3_5_2b_Recipe
Qwen/Qwen3.5-4B	slime, miles	Qwen3_5_4B	Qwen3_5_4b_Recipe, Qwen3_5_4b_Miles_Recipe
Qwen/Qwen3.5-9B	slime	Qwen3_5_9B	Qwen3_5_9b_Recipe
Qwen/Qwen3.6-27B	slime	Qwen3_6_27B	Qwen3_6_27b_Recipe
Qwen/Qwen3.6-35B-A3B	slime	Qwen3_6_35B	Qwen3_6_35b_Recipe
Qwen/Qwen3.8-27B	slime	Qwen3_8_27B	Qwen3_8_27b_Recipe
google/gemma-4-26B-A4B-it	miles	Gemma4_26B_A4B	Gemma4_26B_A4B_Recipe
moonshotai/Moonlight-16B-A3B-Instruct	miles	Moonlight_16B_A3B_Instruct	Moonlight_16B_A3B_Recipe
zai-org/GLM-4.7	slime	GLM_4_7	GLM_4_7_Recipe

Tutorials

The fastest path through the API is the tutorials. Each one ships as a runnable .py and a paired .ipynb narrated cell-by-cell.

Difficulty is a rough signal for where to start:

• Beginner: single-node, introduces one framework concept.
• Intermediate: 1–2 nodes, or wires up something non-default (custom reward, external script).
• Advanced: ≥2 nodes with non-trivial parallelism (tensor-parallel, colocated RL, long context); assumes familiarity with the underlying framework (i.e, slime/miles).

RL

Tutorial	Summary	Difficulty	Framework	Launch
000_rl_basics	Writing correct haikus	Beginner	slime
001_sandboxes	Code RL with Harbor sandboxed evals	Intermediate	slime
002_multiturn	Multi-turn number-guessing RL with custom generate and reward functions	Intermediate	slime
003_on_policy_distillation	Teacher-student (OPD) distillation	Intermediate	slime
005_dapo	Large-scale RL for everyone	Advanced	slime
006_audio_asr	Automatic Speech Recognition	Intermediate	slime
007_param_sweep	Easy grid search	Intermediate	slime
008_computer_use	GUI grounding to predict click coordinates	Advanced	slime
009_cross_tokenizer_distillation	Cross-tokenizer agentic distillation on BFCL v3 multi-turn	Advanced	slime

Agents

Tutorial	Summary	Difficulty	Framework	Launch
000_agent_sandbox	Build an LLM agent harness with a self-hosted model and Modal Sandbox tool execution	Beginner	Modal Sandbox

Multinode

Tutorial	Summary	Difficulty	Framework	Launch
002_glm_4_7	Multi-node full-weight training	Advanced	slime

Multi-node access

Important

Single-node training is open to everyone. Multi-node clusters — required for larger models — are still in Beta. Contact us on Slack for access.