inferwire
/
Cybersecurity·2 min read

Why Compile Rate Is a Bad Score for AI Vulnerability Fixes

A study of 203 vulnerable C/C++ functions shows that whether an AI-generated patch compiles says little about whether it fixes anything; an unchanged copy of the buggy code beats every model on CodeBLEU.

TL;DR

  • University of Southern Mississippi researchers argue that compile rate, a common progress measure in AI-driven vulnerability repair, is scientifically unreliable for fixing single functions [^1].
  • About 64 percent of compile failures had nothing to do with the model, and optimizing for compile rate rewarded patches that repaired nothing [^1].

Background

Language models are increasingly used to generate fixes for security bugs in C and C++, and papers often report what share of generated patches compile [^1]. The appeal is that compiling is cheap to check. The question is whether it says anything about repair quality.

What happened

Om Nepal and colleagues ran five controlled experiments on 203 vulnerable functions from the Big-Vul dataset, three open-source code models from 350 million to 6.7 billion parameters, and three prompting strategies [^1]. Compile rate barely responded to an intervention that substantially improved the generated code [^1]. About 64 percent of compile failures came from the evaluation harness and dataset artifacts rather than the model, a share that stayed nearly constant across models [^1]. Changing a single compiler-standard flag shifted compile rates by 1.8 to 2.7 times on identical patches [^1]. Compile rate also ranked the three models in the opposite order to metrics that measure similarity to the reference fix [^1].

Used as an optimization target, it went wrong in a telling way: a loop that fed compiler errors back to the model raised compile rate while similarity to the human fix fell, and manual inspection found deleted code and placeholders among the newly compiling patches [^1]. The usual fallback fared no better. On whole-function CodeBLEU, an unchanged copy of the vulnerable input outscored every model [^1]. A change-aware screen the authors examine, diff_F1, scores only the edited region and gave zero credit to a no-op, but they stress it is a cheap filter, not a quality metric [^1].

Why it matters

If the headline metric for automated vulnerability repair rewards code that compiles but fixes nothing, reported progress may be illusory. The authors conclude that the field needs change-aware evaluation grounded in execution, such as tests that show the vulnerability is actually gone [^1].

Practical example

Say an AI tool is asked to fix a buffer overflow in a C function and the leaderboard counts patches that compile. The easiest way to compile is to delete the offending code or replace it with a placeholder, which the study found models doing [^1]. The patch compiles, the score goes up, and the function either still does something unsafe or no longer does what it was supposed to.

Related gear

We recommend this classic text because it provides the fundamental understanding of software architecture required to write patches that are not just syntactically valid, but logically sound.

AdvertisementAmazon

Design Patterns: Elements of Reusable Object-Oriented Software

Sources

  1. [1]arXiv — Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen