NVIDIA Releases AI Agent Skills to Accelerate Apache Spark

NVIDIA released new agentic workflow tools, called the cuDF skills for Apache Spark, that allow AI agents to automatically rewrite custom user-defined functions (UDFs) for GPU processing. In traditional Apache Spark pipelines, custom UDFs break declarative abstractions, forcing GPU-accelerated queries to fall back to slow, row-by-row CPU execution and incurring heavy data transfer overhead. By converting these functions into GPU-native code, data engineering teams can eliminate fallback bottlenecks and achieve massive performance improvements.

The workflow offers three distinct translation strategies depending on complexity: replacing functions with standard Spark SQL expressions, implementing the RapidsUDF interface using the Java API for cuDF via evaluateColumnar, or calling custom CUDA kernels using the Java Native Interface (JNI). Powered by model guidance like NVIDIA Nemotron, the cuDF skills guide autonomous coding agents through four structured feedback loops covering testing, conversion, benchmarking, and optimization.

To demonstrate the impact, a major consumer technology company used this agent-assisted process to convert over 30 production UDFs. In one pipeline affecting more than 3,000 jobs, rewriting a single UDF for a 600-billion-row workload reduced operator execution time from 274.55 hours to just 12.78 hours, delivering a 22x speedup. Across various workloads, NVIDIA reports that rewrites routinely yield over 20x speedups.

For data practitioners, these tools lower the barrier to GPU acceleration by automating vectorized programming, reducing refactoring cycles from weeks to hours without compromising correctness. Developers can deploy the skills directly via GitHub or using the command npx skills add NVIDIA/cudf-spark --skill '*' to begin modernizing enterprise analytics workflows.