LangChain Finds GLM-5 Matches Closed Frontier Models
New evaluations from LangChain demonstrate that open-weight models like GLM-5 rival top closed systems on core AI agent tasks while dramatically reducing latency and operational costs.
Framework developer LangChain has updated its Deep Agents evaluation suite to assess 52 models across seven categories, including tool use, file operations, retrieval, memory, conversation, summarization, and unit tests. The testing harness revealed that open-weight architectures such as GLM-5 and MiniMax M2.7 perform comparably to closed proprietary systems on core agentic duties. Outside benchmarks like SWE-Rebench and Terminal Bench 2.0 reinforce these findings, proving that open models deliver the predictable instruction following and tool calling required for production deployments.
The primary drivers for adopting open models are cost and speed, especially as closed frontier options remain 8–10x more expensive for high-throughput applications. Running an application outputting 10 million tokens daily costs roughly $250 per day on Claude Opus 4.6, compared to approximately $12 per day using MiniMax M2.7 on Ollama cloud—an annual difference of around $87,000. Additionally, hosting GLM-5 on Baseten yields an average response latency of 0.65 seconds at 70 tokens per second, outperforming Claude Opus 4.6, which averages 2.56 seconds latency at 34 tokens per second.
LangChain tracks performance using four core metrics: Correctness, Solve rate (measuring expected steps over wall clock seconds), Step ratio, and Tool call ratio. In continuous integration runs across providers like Baseten, Fireworks, Groq, OpenRouter, and Ollama, models run at default thinking settings—high for Gemini 3+, medium for OpenAI models, and disabled for Claude. Models like Nemotron-3-Super are also tested alongside GLM-5 and MiniMax M2.7 to grade execution efficiency, where a step ratio of 1.0 represents an on-budget workflow.
For software engineering teams, integrating these models into Deep Agents requires a single line of code, such as create_deep_agent(model="baseten:zai-org/GLM-5"). The harness automatically handles context compression and model identity injection. Additionally, the open-source Deep Agents command-line interface introduces ConfigurableModelMiddleware to enable mid-session model switching via the /model command. Developers can now orchestrate high-level planning with expensive frontier models before handing off routine execution steps to cheaper open subagents.
This is our own summary of reporting by LangChain Blog
