Benchmark Fatigue, Context Rot, and the Push for Verifiable Agents

Research

METR's Latest Task-Length Evals Show Agents Still Falling Off a Cliff Past a Few Hours

METR keeps refining its horizon-length methodology and the pattern holds: agents crush short tasks but degrade hard once you push past multi-hour work. If you're building anything long-running, this is your reminder that autonomy claims need a time axis attached, not just a pass rate.

New Paper Maps Failure Modes When Agents Handle Long-Context Multi-Step Workflows

The core finding is that context rot is real, agents lose track of earlier constraints as the conversation or task history grows, even with huge context windows. This is a bigger practical bottleneck than raw model capability right now for anyone shipping agents that run for a while.

Study Benchmarks Agent Exploration Strategies Against Static Tool-Use Baselines

Shows that agents given room to explore and backtrack outperform rigid tool-call pipelines on ambiguous tasks, which is a useful data point if you're deciding between a strict workflow graph and something more open-ended. Tradeoff is cost and latency, so this isn't free.

Analysis

Faithfulness Under Fire: Probing Whether Agent Chain-of-Thought Actually Reflects Its Actions

This digs into the gap between what an agent says it's doing and what it actually executes, which matters a lot once you're trusting CoT logs for debugging or safety review. Turns out the reasoning trace is often a post-hoc story, not a real-time plan.

METR Flags Gaps Between Lab-Reported Capabilities and Independent Agent Evals

Another entry showing independent evals consistently lag behind what labs claim in their own benchmarks, especially on tasks requiring sustained tool use. Worth internalizing before you greenlight an agent build based on a vendor's demo numbers alone.

Tools

Composio Ships More Auth and Tool-Connection Primitives for Agent Stacks

Composio keeps chipping away at the boring but critical plumbing problem: getting agents authenticated and connected to real tools without every team reinventing OAuth flows. Unsexy infrastructure, but it's exactly the kind of leverage that saves builders weeks.