Torchfeather · training log

Small models, real multi-GPU runs.

An interactive look at two decoder-only Mixture-of-Experts pretraining experiments I ran on four RTX 3090 GPUs. The plots are reconstructed from the local W&B run logs; choose a metric and toggle runs to explore the learning curves.

How to read this comparison: these runs differ in model size and training budget, so this is an engineering and learning record, not a controlled scaling-law study. The samples are included as diagnostics, not as polished model outputs.
Latest model150.0M parameters
Training data processed≈1B byte tokens
Accelerators4 × RTX 3090
Wall-clock training≈7 hours

Explore the learning curves

The x-axis uses processed byte tokens, calculated as step × global batch (32) × sequence length (512). Training loss is logged every 10 steps; validation is evaluated every 500 steps. Throughput and TFLOPs are per-GPU logged measurements.

Interactive experiment metric chartToggle runs and select a metric. Hover over a curve to inspect the value.

Run summary

ExperimentParametersData processedStepsDurationVal. lossVal. perplexityFull run

Both use byte-level tokens and a 512-token context. Validation loss and perplexity are the final logged values; perplexity is exp(validation loss). See each W&B run for full metadata and charts.

End-of-run greedy samples

Prompt used by the training script: The meaning of distributed training is. Greedy decoding is deterministic. These short samples are useful for checking that checkpoint loading and inference work, but they still show repetition and weak coherence.

What I take from these runs

The training and validation losses fall substantially in both runs, and the larger run reaches a lower final validation loss after seeing more data. That is consistent with learning the byte-level training distribution. It does not establish that the 150M model is better solely because it is larger: it also saw more than three times as many tokens, and the experiments were not designed as a controlled comparison.

The generated text is a useful reality check. Despite the lower validation loss, the sample still loops on a phrase and does not sustain a coherent explanation. This makes a good next experiment clear: improve the evaluation protocol and compare size or data changes while holding the other variables fixed.