SpecValidator: A 1.5B Model That Flags Defective Coding Prompts
Luxembourg researchers show that deleting one constraint from a coding task cuts pass rates by up to 15.3 points, and train a small classifier that spots such defects better than GPT-5-mini.
TL;DR
- In a Luxembourg study, deleting one constraint from a coding task cut pass rates by up to 15.3 points, and bigger models fared no better [^1].
- The researchers' small classifier, SpecValidator, spots such defects far better than GPT-5-mini or Claude Sonnet 4, and found real ones in standard benchmark tasks [^1].
Background
Coding models are usually measured on benchmarks such as HumanEval, MBPP and LiveCodeBench, which run the generated code against tests. The standard score, Pass@1, counts a task as solved when the model's first answer passes [^1]. These benchmarks assume a clear, complete task description. In practice, the researchers note, people use vague terms, leave out edge cases or make typos, much like the defects that turn up in software specifications [^1].
What happened
Amal Akli, Mike Papadakis, Maxime Cordy and Yves Le Traon of the University of Luxembourg posted the paper on arXiv on April 27 [^1]. They took the three Python benchmarks and had GPT-5-mini rewrite each task description in three ways. Lexical vagueness swaps precise words for broader ones, such as "arrange" instead of "sort". Under-specification deletes one explicit constraint, such as a numeric bound, an ordering rule or the expected edge-case behavior. Syntax and formatting adds typos, broken brackets or messy whitespace without changing the meaning [^1]. A separate model, Qwen2.5-Coder-32B, checked the 6,573 defective descriptions, and three researchers reviewed a sample of 100 by hand [^1].
Ten models then wrote code for the original and the altered tasks: three small open models of 6 to 7 billion parameters, five larger ones of 15 to 34 billion, and the reasoning models GPT-5-mini and Claude Sonnet 4 [^1]. Under-specification hurt every model. On HumanEval it cut Pass@1 by 7.9 to 15.3 percentage points; GPT-5-mini and Claude Sonnet 4, which started at 96.3 and 95.7 percent, each lost 9.7 points [^1]. On MBPP the drops were 5.1 to 10.8 points. Vague wording cost between 1.3 and 12.2 points on HumanEval but only 1.5 to 3.2 on MBPP, and typos and formatting errors barely mattered [^1]. LiveCodeBench stayed within 2.0 points for every defect type. Its problems include sample inputs and outputs, which often encode the deleted constraint anyway [^1].
SpecValidator is the researchers' answer: Qwen2.5-Coder-1.5B, a small code model, fine-tuned with LoRA, a method that trains only a small set of added weights, here 0.14 percent of the parameters [^1]. It labels a description as clean or as one of the three defect types and reports the type, so the description can be fixed [^1]. On a held-out test set it reached an F1 score of 0.804 and an MCC of 0.745, two standard measures of classification quality where higher is better [^1]. Asked directly, GPT-5-mini scored an F1 of 0.469 and Claude Sonnet 4 scored 0.518; with four labeled examples in the prompt, they reached 0.462 and 0.557 [^1]. The researchers then ran SpecValidator on 456 original benchmark descriptions labeled clean. It flagged 83 of them: none from HumanEval, 33 from MBPP and 50 from LiveCodeBench [^1]. For MBPP, 90.9 percent of the flagged tasks counted as failures, with 6.7 of the ten models failing on average; for LiveCodeBench, the figures were 98.0 percent and 7.7 models [^1]. Three researchers reviewed the 33 MBPP cases and confirmed 24 as under-specified, among them "Find the peak element in the given array", which leaves open whether a local peak or the global maximum is meant, and whether to return its index or its value [^1].
Why it matters
The researchers draw a clear conclusion from the first experiment: how well a model copes with a flawed description depends mainly on the kind of defect and on how the description is structured, not on the model's size [^1]. Even models above 95 percent on HumanEval lost nearly 10 points when one constraint went missing [^1]. That puts the fix at the input, not in a bigger model. The comparison with GPT-5-mini and Claude Sonnet 4 shows, in the paper's words, that strong code generation "does not transfer to the prompt meta-reasoning" [^1]. Besides higher accuracy, the researchers list lower compute and local deployment as advantages of the small model [^1].
The hardest case is also the most damaging one. Under-specification leaves no trace in the text, so the classifier has to notice that something is absent; confusing clean and under-specified descriptions was its most common error [^1]. The researchers also name limits of their own study: all tasks are in Python, results may differ for library-heavy tasks, and real prompts can mix several defects [^1]. For benchmark builders, they see direct implications, since some tasks in standard benchmarks are themselves defective [^1].
Practical example
Say Priya works on the data team at a mid-sized logistics company and asks a coding assistant: "Write a function to sort a given matrix according to the sum of its rows." The sentence reads fine. It is in fact one of the paper's examples: the original MBPP task said "in ascending order", and the under-specification mutation deleted exactly those words [^1].
Without them, the assistant has to guess the direction. If it guesses descending, Priya's function sorts the matrix the wrong way round and the team's existing tests fail. This kind of defect caused the largest drops in the study, and even the strongest models were not immune [^1].
If Priya's team puts SpecValidator in front of the assistant, the classifier reads the description first and reports the defect type it sees, here under-specification, so Priya can add the missing rule before any code is written [^1]. It won't catch every case: telling clean descriptions from under-specified ones was where it went wrong most often [^1]. So Priya also adds a sample input and output, the way LiveCodeBench problems do. In the study, such examples kept LiveCodeBench within 2.0 points even when a constraint was deleted [^1].
Related gear
We recommend this book because it is a standard guide to writing clear and complete software requirements, the discipline the paper borrows its idea of specification defects from.