I find it fascinating how the same resource allocation problems that plague shared systems have plagued humanity for centuries. AI2’s approach to GPU scheduling is essentially a modern take on Garrett Hardin’s “tragedy of the commons,” and their solution offers lessons that extend far beyond machine learning clusters.
The core problem they faced is one many AI labs are hitting right now: demand for GPU time vastly exceeds supply. At AI2, they have 2-3x more workload requests than available capacity at any given moment. Under their old priority-based system, this created predictable failure modes. Teams would park no-op workloads to claim GPU time (squatting). Priority levels inflated until 100% of jobs claimed HIGH priority, collapsing the entire priority system. Engineers spent most of their time negotiating shutdowns rather than fixing infrastructure.
The root cause was misaligned incentives. Users had every reason to game the system because the resource was free to reserve. The solution wasn’t better rules or stricter policing, it was restructuring incentives entirely.
From Monopolies to Time Budgets
Their first instinct, assigning GPU monopolies to teams, failed because research is seasonal. Some teams need massive capacity for weeks, then nothing for months. Static allocations meant GPUs sitting idle while other teams waited. They were solving a knapsack problem manually.
The breakthrough came when they reframed the problem from “how do we distribute hardware” to “how do we distribute time.” More importantly, they moved from operational scheduling to strategic budgeting. Instead of fighting over individual workload priorities, leadership now debates research strategy and allocates GPU time accordingly. Project leads get a percentage claim on total capacity, regardless of what else is queued.
This is elegant because it aligns with how organizations actually think. Executives understand resource allocation as a strategic investment problem. They’re comfortable debating whether robotics RL deserves 15% or 20% of capacity. What they shouldn’t be doing is approving individual training jobs.
The Scheduler as Policy Enforcer
Budgets only matter if they’re actually enforced. AI2 built a hierarchical fair-share scheduler that tracks occupancy over a 7-day window and prioritizes underutilized allocations. This approach isn’t novel, it dates back to the Hadoop Fair Scheduler in 2009. What matters is applying it to a tree of actual organizational units with budgets set by managers.
The clever part is distinguishing allocated versus unallocated occupancy. Workloads charged to a budget get preemption protection (up to a minimum runtime). Unallocated workloads are free but preemptible. This keeps GPUs busy while preventing waste. If a team’s allocation sits unused, another team’s free workloads run. When the first team’s paid job arrives, the free workload gets preempted. No GPU sits idle, but budgets still mean something.
One researcher reported this made it feel like they had an extra 30% compute. They could burst beyond their allocation on idle capacity, then get preempted when needed. For bursty workloads, this is massive.
The Minimum Runtime Contract
Long-running training jobs create scheduling nightmares. Once a job starts, it might run for weeks, blocking any rebalancing. This is what enabled GPU squatting in the first place.
AI2’s solution: workloads declare a minimum runtime (minimum progress needed), and they’re protected from preemption during that window. After that, the scheduler can preempt and requeue them, enabling fair-share to converge. Users can set minimum runtime to zero to run entirely unallocated (and free).
This is a contract: we’ll give you uninterrupted time to make progress, then we reserve the right to pause you if others need their allocation. It’s reasonable from both sides. More surprisingly, it reduced human-in-the-loop repairs by 74%. Infrastructure maintenance can now fully drain workloads automatically instead of negotiating with job owners.
What This Means for the Industry
I think this matters because most AI labs are facing the same scaling challenges. As GPU clusters grow from hundreds to thousands of devices, operational scheduling becomes untenable. You can’t have humans debating priority for every workload.
The budget model is also more defensible politically. It’s transparent and strategic rather than appearing arbitrary. Teams know exactly what they’re getting and why. This is how resource allocation should work in shared infrastructure.
The technical innovation is modest, the organizational innovation is significant. By shifting the debate from operations to strategy, AI2 turned a management nightmare into a budgeting exercise. The scheduler is just the enforcement mechanism.
I’m curious whether this model scales to the era of truly massive training runs, where a single job might demand thousands of GPUs for months, or whether it breaks down entirely.