Eval Matrix Engine for AI Outputs
Compare a reference (gold) text against a candidate AI output with Levenshtein distance, similarity, BLEU-1/2, and keyword-overlap scores.
Comparison Matrix
Professional Insights & Guide
Learn how the Eval Matrix Engine scores text, which formulas drive each metric, and the edge cases that can mislead you.
Core Use Case scenario
LLM engineers and ML evaluators must measure whether an AI output matches an expected gold response. The engine compares two strings and reports four families of evidence: a Levenshtein distance over characters, a similarity percentage derived from it, BLEU-1 and BLEU-2 word precision scores, and a Jaccard keyword-overlap score. Together they expose exact edits, lexical coverage, and topic retention in one matrix.
How Each Metric Is Computed
- Levenshtein distance: the minimum number of single-character insertions, deletions, or substitutions needed to turn one string into the other, found with dynamic programming.
- Similarity %: 100 × (1 − Levenshtein ÷ max(reference length, candidate length)), so 100% means identical text.
- BLEU-1 / BLEU-2: the clipped precision of matching word unigrams and bigrams, i.e. matched n-grams divided by candidate n-grams.
- Keyword overlap: stopwords and very short words are removed, then Jaccard = |ref ∩ cand| ÷ |ref ∪ cand| × 100.
Troubleshooting & Edge-Case Failure Points
- Blank input: both fields empty shows a hint; a missing reference blocks scoring so the candidate cannot be graded in isolation.
- Case and punctuation: the distance and n-gram scores run on normalized lower-case text, so trivial casing changes do not inflate the distance.
- Very short strings: BLEU-2 returns 0 when the candidate has fewer than two tokens; interpret the n-gram columns together, never alone.
- Stopword noise: common words such as "the" and "will" are filtered before the keyword overlap so filler cannot dominate the score.
Detailed Step-by-Step Instructions
- Paste the expected or gold answer into the Reference text area, or press Load Sample Pair to see a realistic refund-response example.
- Paste the candidate output produced by your LLM or agent into the second text area.
- Click Compare Outputs (or edit either box to re-score live) and inspect the comparison matrix.
- Read the score bars: similarity, BLEU-1, BLEU-2, and keyword overlap each turn green, amber, or red by threshold.
- Review the matched, reference-only, and candidate-only keyword chips to see exactly which concepts drifted.
- Press Copy Report to paste the metric table into a test run, ticket, or evaluation log.
Related Web Utilities (Silo Hub)
How to Use the AI Eval Matrix Engine
Scores AI outputs against a gold reference across exact, lexical, and semantic dimensions - the fast eval layer production teams run first.
- Paste the gold-standard output and the candidate output.
- Read the matrix: similarity, BLEU-1/2, Levenshtein, keyword overlap.
- Compare variants; ship the prompt whose worst metric improves.
What BLEU and Levenshtein Tell You
Levenshtein counts single-character edits between strings - good for exact-ish tasks (code, extraction). BLEU measures n-gram precision vs the reference. Keyword overlap catches factual misses the others average away. Honest practice: no single number evaluates language; a matrix does. Score 5-10 test cases, watch the worst case, and remember lexical metrics are blind to meaning - "buy" vs "purchase" scores as difference.
AI Eval Matrix Engine FAQ
What is a good BLEU score?
0.3-0.5 is respectable on prose with one reference; extraction/code can exceed 0.7. Judge against your baseline.
Why does Levenshtein fail on long text?
It is global per-character: one shifted sentence inflates distance. Use it for short exact outputs.
How many test cases do I need?
5-10 minimum including boundary and adversarial inputs. Review the worst rows, not the mean.