How to measure abliteration damage with 40 fixed prompts
Measure capability changes with a frozen prompt set, paired configurations and separate scores for correctness, instruction following and refusal. Forty prompts can reveal regressions; they cannot establish universal model quality.
What should the 40 prompts cover?
Use ten arithmetic and logic tasks, ten nested-instruction tasks, ten strict-format tasks and ten benign requests that may trigger over-refusal. Keep the expected answer or scoring rubric alongside every prompt. Our downloadable set makes this split explicit and can be inspected before any model is evaluated. It is a diagnostic set, not a representative sample of every use case. Our guide to capability changes after abliteration explains why these groups need separate scores.
Freeze the comparison before running it
Record the exact model revision, tokenizer, chat template, system message, quantization, context length and generation settings. Use the same input text for both checkpoints. For hosted APIs, record the returned model identifier and the time of the run; a public model name can remain stable while the backend changes. Keep transport errors separate from model responses. Use the base-versus-abliterated comparison checklist to catch configuration differences before a run.
Score the answer, then score the refusal
A response can answer willingly and still be incorrect. Give correctness a score from 0 to 2, record instruction compliance as pass or fail, and label refusal separately. For strict JSON, use a parser before a human review. Do not count a timeout, empty stream or authentication error as a refusal. When a refusal is ambiguous, retain the full response and the reason for the label. The residual-refusal taxonomy gives examples for borderline response labels.
How do you report a regression?
Report paired outcomes: base correct and edited incorrect, edited correct and base incorrect, both correct, both incorrect. A net change alone hides which tasks broke. Repeat runs when decoding is stochastic and retain each repetition. Publish the prompt-set version and the raw responses if you want another person to reproduce the table.
What this release contains
The CSV includes 40 actual prompts, expected checks and empty result fields. No model calls have been made as part of this protocol release, and blank cells are not zero scores. Use the separate 120-run template to compare three model configurations without inventing a leaderboard.
| Group | Prompts | Primary check |
|---|---|---|
| Arithmetic and logic | 10 | Expected answer |
| Nested instructions | 10 | All constraints satisfied |
| Strict format | 10 | Parsing and exact fields |
| Benign sensitive context | 10 | Useful answer and refusal label |
Sources & provenance
Published 2 November 2026. See our editorial method for source and measurement standards.