In late July 2026, someone enabled scaled dot-product attention (SDPA) for T5 in the Transformers library. The change looked safe. It passed the forward passes. It didn’t crash. Two months later, a 66MB toy model running on a CPU caught something: the decoder could peek at future tokens during training, completely breaking the task it was supposed to learn.
That’s the story of Model Integration Contracts (MICs), and it matters more than you might think.
Transformers maintains over 750 model types. The test suite runs 88,000+ CPU tests on main, plus 200,000+ GPU tests daily. Running full integration tests on every pull request would mean keeping GPUs busy constantly, which is neither practical nor cheap. By the time a scheduled run finds a regression, it’s already deployed to main.
Shrinking Models Without Losing Signal
The answer isn’t to test less. It’s to test smarter. Instead of loading the full 1,227MB UMT5 checkpoint, the team built a fixture: a small model with the same architecture but fewer and narrower layers. Every type of layer is there. The tokenizer and processor files stay unchanged. It’s 66MB instead of 1.2GB.
This fixture runs on cheap CPU runners as part of fast integration tests on every PR. It caught that SDPA bug on every single commit from July 30 to September 28 when someone finally fixed it. The log-probabilities differed by 1.22 and training loss by 0.20 compared to the baseline.
Here’s what makes this work: the fixture goes through the exact same code paths as the real model. If you only test eager attention and skip SDPA, you miss the bug entirely. The architecture matters. The attention variant matters. The weight format matters because it exercises the conversion code users actually run.
Big models benefit too. Some don’t fit on standard GPU runners, but their fixtures run fine on CPU. Llama 4 Scout shrinks from 108.6 billion parameters to 44.7 million by keeping the architecture patterns (MoE routing style, layer types) while cutting dimensions and layer counts.
The Contract, The Baseline, The Replay
A Model Integration Contract has three pieces: a shrunk fixture, a usage path (generation, training, inference), and recorded baselines. Run the usage path on the fixture. Compare outputs. If they match, the contract holds.
SmolLM’s contract might generate from “The capital of France is” and check token IDs. It also verifies training loss, checks that gradients are finite, and confirms that optimizer steps actually change weights. The simplicity is intentional: you’re testing that the behavior matches, not that every internal activation is identical.
These contracts live in the huggingface-mic repository alongside a fleet dataset on the Hub that pins each MIC to a frozen revision with locked package versions. During CI, the system picks MICs based on which files the PR touched. When you replay frozen MICs over months of commits, you get a timeline of where regressions appeared and when they were fixed.
What Works, What Doesn’t, What’s Next
Random weights have limits. The team discovered this with Whisper: a fixture with random weights barely responded to audio changes, so the model would run but fail to test the audio path meaningfully. They trained it on a few tones paired with words like “red” and “green”. Overfitted fixtures caught all 24 injected synthetic faults. Random weights caught 18 of 24. But both missed a real assisted-decoding bug that only showed up with the actual checkpoint.
So the team keeps random weights when they pass, and overfit when they need better sensitivity. It’s pragmatic: training takes seconds, but it’s one more step that can fail and one more thing to maintain.
Read more about ML testing strategies and the nuances of regression detection in practice.
Right now, as of November 1, 2026, the MIC job reports failures without blocking PRs. Whether it should block is still an open question. Coverage is partial across model families and APIs. When something slips through, the team adds the missing usage path and replays it on both broken and fixed versions.
The bigger vision: link Hub model pages directly to their MICs so developers and agents using a checkpoint can start from tested examples with known environments and expected outputs.
What remains unmeasured is how many bugs like that Whisper one only surface with real weights, and whether overfitting catches patterns that random initialization fundamentally misses.