Databricks Scales AI Research Using AI Runtime
Databricks AI Research is using its internal AI Runtime infrastructure to manage model training across hundreds of GPUs, expanding automated agent-driven training jobs by 8x.
Databricks is deploying its proprietary AI Runtime (AIR) infrastructure as the foundation for its AI Research division. The team uses AIR to build, train, and fine-tune models powering products like Genie, Smart Routing, and AI Functions, which include tools such as ai_parse_document and ai_extract. The infrastructure supports supervised fine-tuning (SFT), offline reinforcement learning (RL), and online RL across model scales ranging from a few billion parameters to nearly a trillion. Research efforts relying on AIR have already yielded breakthroughs, including making search 3x faster, improving offline reinforcement learning, and advancing document extraction quality.
To handle these massive workloads, AIR provisions hardware such as NVIDIA H100 and B300 GPUs, scaling up to runs involving hundreds of GPUs over days or weeks. For long-running projects like KARL—which requires post-training models with more than 100 billion parameters—AIR provides health monitoring and preflight checks. The system automatically detects issues like NCCL communication failures, thermal or power throttling, and GPU driver errors, cordoning off faulty nodes to prevent progress loss. Researchers control execution using the AIR CLI, custom Docker containers local to development, the Databricks Artifact Registry, and Unity Catalog FUSE mounting for seamless data governance. Job tracking is managed automatically via MLflow.
The system has enabled a shift toward agent-driven research, a concept popularized by Andrej Karpathy as autoresearch. By using AI agents to propose changes, launch parallel trials, evaluate metrics, and update plans autonomously, training job volume across AI Functions has grown by 8x. Agents interact directly with the environment via the Databricks CLI and databricks-air-skills to handle concurrent workloads. Workload prioritization and preemption allow teams to maximize GPU utilization without manual coordination.
For practitioners, this integrated architecture demonstrates how combining agentic workflows with resilient GPU infrastructure can remove operational bottlenecks. By automating node management, container deployment, and MLflow logging, developers can move from manual job launching to managing autonomous research loops that dramatically accelerate model development.
This is our own summary of reporting by Databricks AI
