# Base vs abliterated: the comparison mistakes that change the result

Canonical: https://refusalmap.com/articles/base-vs-abliterated-comparison/
Author: Refusal Map Editorial
Published: 2026-11-02
Category: Methods

A fair comparison changes the intervention while holding the rest of the evaluation steady. A shared seed and temperature help, but they do not compensate for different chat templates, output limits or serving conditions.

## What does “base” mean in this comparison?

Use the original instruction-tuned checkpoint from which the edited model was derived. In model documentation, base can also mean a pretrained model without instruction tuning. Comparing that pretrained model with an edited chat model changes more than refusal behavior and makes the result hard to interpret. State your intended meaning explicitly. The identity fields in our [abliterated model-card guide](/articles/reading-abliterated-model-cards/) help establish that relationship.

## The same seed is not the same experiment

A seed is useful within a reproducible stack. Different kernels, batching, hardware and samplers can still produce different outputs. Record the seed when supported, but do not promise identical sampling across different API providers. Compare distributions over repeated runs when a single output would decide the conclusion.

## Keep the full input contract identical

The system prompt, tokenizer, special tokens and chat template can all affect behavior. A manually formatted conversation sent to one model and a provider-formatted conversation sent to another may not be equivalent. Record the rendered prompt where possible. If a hosted service hides the template, describe that as a comparison limit. Use the [four-condition quantization matrix](/articles/abliterated-vs-quantized/) when precision also changes.

## Watch for budget and format confounds

A short output limit can interrupt a correct solution or make JSON invalid. Reasoning budgets can also affect time and answer quality. Keep those limits consistent and record truncation as its own event. If one endpoint supports constrained decoding and the other does not, separate the model comparison from the serving-feature comparison.

## Use a paired result sheet

Assign every prompt a stable ID and save both responses next to the same expected check. Label refusals, errors and task failures separately. Make reviewers blind to checkpoint names where practical. Inspect the disagreement cases before summarizing a net score: they are often where a configuration mistake becomes visible. Our [40-prompt comparison protocol](/articles/abliteration-40-prompt-protocol/) includes a blank paired-results workflow.

## Reference table

| Control | Record | Why it matters |
| --- | --- | --- |
| Checkpoint | Exact revision | Names can hide updates |
| Input | Template and system prompt | Changes behavior |
| Decoding | Temperature, seed, token limits | Changes sampling and truncation |
| Serving | Precision and backend | Changes numerical and runtime behavior |

## Sources

- [Arditi et al. (2024): Refusal in Language Models Is Mediated by a Single Direction](https://arxiv.org/abs/2406.11717)

## Related reading

- https://refusalmap.com/articles/abliteration-40-prompt-protocol/
- https://refusalmap.com/articles/abliterated-vs-quantized/
- https://refusalmap.com/articles/capabilities-after-abliteration/