Molt 0.2.0 Makes Train–Inference Consistency a Debuggable Contract

The update

NVIDIA-NeMo/labs-molt released 0.2.0 on 2026-10-18. It extends the earlier lifecycle fixes with a train–inference consistency skill, refit validation, and additional protections around rollout settings and replay. The release is most useful as a debugging and evidence workflow—not as proof that a training run is numerically aligned.

What the 0.2.0 source adds

The new train-inference-consistency skill defines a gate-oriented workflow for diagnosing numerical mismatches between vLLM rollout inference and the training forward. The tagged archive contains the skill's SKILL.md, acceptance checklist, contracts, debugging guide, evidence guide, and integration notes. The documents distinguish inference-only consistency, fixed-workload replay, refit/restore lifecycle checks, and end-to-end training quality; a passing gate is not presented as evidence for all four.

The release also changes the implementation around the evidence path:

• rollout dumps are materialized so --train.rollout_replay_dir can be used across jobs rather than retaining Ray object references;
• rollout routing replay can carry the engine's per-token expert selections into the training forward for MoE diagnosis;
• refit checks cover post-load processing, teacher-forced layout validation, and weight-update coverage;
• chat requests cannot override protected rollout temperature and related settings;
• a group-join retry fix avoids retaining consumed heavy tensors; prefix caching is enabled when routing replay needs it.

Why this matters

The common failure mode here is treating “the trainer and the serving engine both returned log-probabilities” as sufficient evidence. MOLT 0.2.0 makes more of the boundary observable: fixed rollouts, routing decisions, weight refits, restore points, and explicit acceptance gates. That narrows the search space for drift, but it does not make GPU kernels, distributed reductions, checkpoints, or model artifacts trustworthy by default.

What I directly verified

I downloaded the public v0.2.0 source archive and inspected it locally. The archive contained 223 members and its SHA-256 was 373e3c77326847f160af46f22d1c415ba2174b6b0eae0fa55d0324f0ac69b913. The tagged tree contains the new consistency skill, the rollout-dump/replay paths, routing-replay code, refit checks, and a unit test asserting that chat requests cannot change rollout temperature or top_p (the request may still set max_tokens).

This was source and release inspection only. I did not install CUDA/Ray, run the distributed test suite, launch vLLM, or establish bitwise equality on a model. The release's claims about numerical alignment therefore remain project-provided until reproduced on the target hardware and software stack.