Common Errors When Scoring AI Outputs and How to Avoid Them

The most common mistake when using any evaluation engine is running it on empty or near-empty input. If you paste nothing into both text areas, there is nothing to compare, and if you paste a reference but forget the candidate, the engine cannot score anything. The Eval Matrix Engine guards against both cases with inline error messages, but the real lesson is procedural: always confirm that a candidate output was actually captured before you grade it. An empty candidate that sneaks into an automated pipeline will score near zero and silently trip every regression alarm.

A second frequent error is treating a single metric as the whole verdict. New evaluators look at the similarity percentage and ignore BLEU-1, BLEU-2, and keyword overlap, then get confused when a high-similarity candidate is still rejected by a reviewer. Each column of the matrix measures something different. Similarity reflects exact character closeness, BLEU-1 reflects vocabulary coverage, BLEU-2 reflects word-pair ordering, and keyword overlap reflects concept retention. A candidate that rephrases everything will score poorly on similarity yet well on keywords, and that is not a bug in the tool, it is a signal that you need to read the whole matrix.

Stopwords are the classic silent killer of keyword scores. If the engine did not filter words such as "the", "will", "and", and "within", a candidate could match the reference purely on filler and post a deceptively high overlap. The tool removes stopwords and tokens shorter than three characters precisely to avoid this inflation. The mistake users make is judging the raw overlap number without looking at the matched-keyword chips. If the intersection is dominated by generic business terms and none of the domain-specific nouns match, the score is not actually good, regardless of the percentage shown.

Short strings break n-gram scoring in ways that surprise people. BLEU-2 requires at least two tokens in the candidate to compute a bigram precision, so a one-word candidate always reports a BLEU-2 of zero. Similarly, a candidate that is a fraction of the reference length produces a misleadingly low BLEU-2 simply because it has few bigrams to match. When you evaluate terse outputs, such as one-line status messages or short labels, prefer BLEU-1 and keyword overlap, and never use BLEU-2 as a pass/fail gate on strings under a dozen tokens.

Mixing normalisation levels is another trap. The engine lower-cases both sides before computing distance and n-grams, which is a deliberate choice. If you then compare those scores against numbers generated by a different tool that kept original casing, every comparison will be subtly off. Pick one normalisation convention, document it, and keep it constant across runs. The same logic applies to number formatting: "5-7 business days" versus "5 to 7 business days" is a legitimate wording difference, but it will lower the similarity score, so decide in advance whether paraphrasing of numbers should be acceptable in your domain.

Users also misread the direction of the distance metric. A larger Levenshtein distance means worse agreement, while a larger similarity percentage means better agreement. With four numbers on screen, it is easy to glance at the distance, see a high number, and assume the result is good because the other bars are high. Always check that the distance column is moving in the opposite direction from the similarity and keyword columns before drawing conclusions from a run.

Over-normalising punctuation hides genuine errors. If your pipeline strips all punctuation before scoring, then a candidate that adds a comma to a critical address field can score as perfect even though the extra character changes the meaning. The tool computes character counts on the raw strings so you can spot length inflation, and it runs edit distance on trimmed, lower-cased text that still preserves punctuation. When a structured field is involved, do not rely on aggregate scores alone, inspect the exact characters with a diff-style read of the two strings.

Regression logs that lack context are their own error. Saving only the score without the reference, candidate, and model version makes the number meaningless three weeks later. When a score drops, you cannot tell whether the model changed, the prompt changed, or the reference set changed. Export the full report from the copy button and store it with its inputs so every score remains audit-trail-ready.

Finally, do not conflate correlation with a solved evaluation. A matrix score of 95% similarity does not prove the candidate is factually correct, it proves it is textually close to the reference. If the reference itself contains an error, the candidate faithfully reproduces it and still scores perfectly. Periodic expert review of both the references and the borderline outputs is what catches this class of problem, no automated metric can substitute for it.

The way to master the matrix is to know what each column can and cannot tell you, and to treat the error messages and guardrails as part of the workflow rather than annoyances. When inputs are validated, stopwords are respected, short strings are handled with the right metric, and scores are logged with context, the Eval Matrix Engine becomes a reliable early-warning system instead of a source of confusing numbers.