Agent Optimizers Tested Twice: Only RELAI's Own Method Compounded
RELAI.ai researchers optimized a GPT-5.5 terminal agent in two rounds on hard Terminal-Bench 2.0 tasks. GEPA overfit, Meta Harness stalled, and the company's own RELAI-VCL kept improving.
TL;DR
- In a two-round test on 22 hard Terminal-Bench 2.0 tasks, GEPA's gains failed to transfer and Meta Harness stalled when optimized again [^1].
- Only RELAI-VCL, from the authors' company RELAI.ai, kept improving; they credit its rule of rejecting changes that break previously solved tasks [^1].
Background
An agent harness is everything around a language model that makes it an agent: prompts, tools, memory and code [^1]. Agent optimizers edit that harness automatically and keep the changes that raise a benchmark score. According to the paper, most reported gains are one-shot: an agent is optimized once against a fixed benchmark [^1]. Deployed agents meet new failures, so optimization is repeated. The authors relate this to catastrophic forgetting, where a model updated on new data loses performance on earlier tasks [^1].
What happened
Wenxiao Wang, Priyatham Kattakinda and Soheil Feizi of RELAI.ai asked whether optimizer gains compound: can an agent optimized once be optimized again on new tasks without losing what the first round achieved [^1]? They built a two-phase test from Terminal-Bench 2.0, whose 89 tasks include 30 rated hard. Phase 1 uses 12 hard tasks, and Phase 2 adds 10 more, for 22 [^1]. Every method started from the same baseline terminal agent running on GPT-5.5 and got 200 rollouts per phase. Each task was run twice per agent [^1]. The three optimizers were GEPA, which evolves the agent's prompts using feedback on its runs; Meta Harness, where a coding agent edits the harness code; and RELAI-VCL, which rejects any change that improves new tasks at the cost of tasks the agent could already solve [^1].
After Phase 1, all three beat the baseline's 62.5 percent on the first 12 tasks: GEPA reached 70.8, Meta Harness 66.6 and RELAI-VCL 79.2 percent [^1]. The picture changed on all 22 tasks before any further optimization. GEPA fell to 54.5 percent, below the unoptimized baseline at 56.8, while Meta Harness reached 68.2 and RELAI-VCL 72.7 percent [^1]. The authors found sample-specific information from the Phase 1 tasks apparently hardcoded in GEPA's optimized prompt [^1].
In Phase 2, each method got a second budget on all 22 tasks. GEPA recovered to 72.7 percent. Meta Harness found no candidate better than its existing agent, and its score fell to 59.1. RELAI-VCL reached 77.3 percent [^1]. Averaged over the three stages, RELAI-VCL scored 76.4 percent, GEPA 66.0, Meta Harness 64.6 and the baseline 58.7 [^1].
Why it matters
The study shows how a single score can hide different failures. GEPA and Meta Harness ended with similar averages, 66.0 and 64.6 percent, yet one overfit the first tasks and the other could not build on its own changes [^1]. For anyone tuning agents, that argues for measuring three things separately: the gain on the tasks that drove the change, the result on tasks the optimizer never saw, and whether a second round still helps. The authors interpret RELAI-VCL's lead as a sign that regression control inside the search loop filters out narrow shortcuts, but they call this an interpretation tied to this evaluation, not a general law [^1].
The evidence has clear limits. The authors are testing their own company's method against two others. The test covers 22 tasks, two trials each and two rounds [^1]. The authors also note that Terminal-Bench 2.0 tasks are only loosely related, while real failures cluster in one domain, and that their protocol assumes tasks can be rerun and checked by reliable verifiers, which is rarely true in production [^1].
Practical example
Say you maintain an internal coding agent and run an optimizer every month on the tickets it failed. Before you accept a change, you rerun the tasks the agent already solved and reject any edit that breaks one, the rule RELAI-VCL builds into its search [^1]. You also read the new prompt for details copied from specific tickets, the kind of sample-specific information the authors found in GEPA's prompt [^1]. Finally, you keep a set of tickets the optimizer never sees, since unseen tasks were where GEPA dropped below the unoptimized agent [^1].