Frozen Models Can Drift from Their Dataset Hashes

While building the contracts for an LLM evaluation harness, I tried changing a dictionary inside a frozen Pydantic model. Attribute assignment failed. Editing the dictionary succeeded. The dataset had changed, even though the object was supposed to represent a fixed version.

That matters when a run manifest identifies its dataset by a hash. If the examples change after the version is recorded, the manifest can describe a different evaluation from the one that actually ran. Giving the model frozen=True does not close that gap.

The design I kept stores the version as a field and recomputes it during validation. This catches disagreement between the content and its recorded identity when loading a record. It still permits live mutation, and its creation path hashes the content twice. Both limits deserve an explicit explanation; the second also deserves a measurement.

This article is for readers using Pydantic v2 to persist records, with some familiarity with BaseModel and validators. The examples simplify the evaluation contracts to string inputs and answers. The probes and benchmarks below ran on Python 3.14.3 and Pydantic 2.13.5.

A computed version follows the mutation

My first approach exposed the version through @computed_field. A small model shows the problem without the rest of the dataset schema:

import hashlib
import json

from pydantic import BaseModel, ConfigDict, computed_field


class ComputedDataset(BaseModel):
    model_config = ConfigDict(frozen=True, extra="forbid")
    inputs: dict[str, str]

    @computed_field
    @property
    def version(self) -> str:
        data = json.dumps(self.inputs, sort_keys=True).encode("utf-8")
        return "sha256:" + hashlib.sha256(data).hexdigest()


ds = ComputedDataset(inputs={"question": "2+2?"})
before = ds.version
ds.inputs["question"] = "3+3?"  # succeeds
assert ds.version != before

The computed value remains consistent with the current content. What it cannot preserve is the identity of the earlier content. That earlier identity needs to be recorded somewhere else if a later comparison is going to detect a change.

There was also a persistence problem. With this configuration, passing the result of model_dump_json() to model_validate_json() fails: the dump includes version, but validation treats it as an extra input because it is not a declared field.

Excluding computed fields fixes this round trip. In the tested version, both exclude_computed_fields=True and round_trip=True omit this computed field. That is a valid design if the expected version lives elsewhere, such as in the manifest or a content-addressed filename. I wanted the dataset record itself to carry a version that could be checked on load, so I made it an ordinary field. See Pydantic’s dump options.

What frozen actually protects

Pydantic documents this as faux immutability: frozen=True blocks ordinary attribute reassignment. It does not recursively freeze the objects stored in those attributes.

There are three separate things to consider. A frozen model protects a field binding. A tuple protects the membership and order of that container. An annotation such as Mapping exposes a read-only interface to the type checker. None of these makes a dictionary nested inside an example immutable.

In the original probe, Mapping[str, str] accepted a dictionary and Sequence[Example] accepted a list; the resulting objects were still a dictionary and a list. Those abstract annotations were not a request for immutable containers. Switching the dataset’s examples to tuple[Example, ...] prevents appending or replacing examples, but the inputs dictionary inside each example can still be edited.

I also tried a validator that returned MappingProxyType(dict(value)). It blocked item assignment, but both json.dumps and Pydantic’s JSON serializer rejected the resulting proxy without a custom serializer. The copy matters: a proxy over a shared dictionary reflects changes made through the original reference. This is a serialization integration cost, not an impossibility; a field serializer can convert the proxy back to a dictionary. It also only protects that mapping, not arbitrary mutable values inside it.

For the contracts I was building, I kept ordinary mappings and a stored hash. That choice detects inconsistency at a validation boundary. It does not prevent inconsistency from arising between boundaries.

Decide what “the same dataset” means

A dataset version represents content identity. A run ID represents an event: executing the same inputs twice still produces two runs. The two identifiers answer different questions.

For the dataset model below, the hash includes provenance and the ordered examples. Each example contributes its ID, inputs, and expected answer. Dataset name and creation time are excluded. The resulting policy is precise:

• Changing an answer, an example ID, or provenance changes the version.
• Reordering examples changes the version; changing dictionary insertion order does not.
• Renaming the dataset or recreating it at another time leaves the version unchanged.

I want both reordering and provenance edits to create a new version; these are deliberate choices in this contract. A dataset treated as an unordered set would need a different ordering rule before hashing. A benchmark that cares only about questions and answers might keep provenance outside content identity.

The hash recipe sorts dictionary keys, fixes JSON separators and Unicode escaping, and encodes the result as UTF-8. These JSON options make serialization deterministic for the restricted payload shown here. They do not define a universal canonical JSON format: visually identical Unicode strings can still differ, and a richer numeric schema needs explicit rules for values such as non-finite floats. The sha256: prefix identifies the digest algorithm, not the schema or serialization recipe.

Store the hash and verify the normalized content

Here is the complete module used for the measurements. create accepts already-built examples, computes a version, and passes it to normal construction. The after-validator computes it again from the validated fields. Loading a JSON record supplies its stored version and runs the same check once.

import hashlib
import json
from collections.abc import Mapping, Sequence
from datetime import UTC, datetime
from typing import ClassVar

from pydantic import AwareDatetime, BaseModel, ConfigDict, Field, model_validator


def canonical_hash(payload: object) -> str:
    canonical = json.dumps(
        payload,
        sort_keys=True,
        separators=(",", ":"),
        ensure_ascii=False,
    ).encode("utf-8")
    return f"sha256:{hashlib.sha256(canonical).hexdigest()}"


class FrozenModel(BaseModel):
    model_config: ClassVar[ConfigDict] = ConfigDict(frozen=True, extra="forbid")


class Example(FrozenModel):
    id: str
    inputs: dict[str, str]
    expected: str


# Nested mappings remain mutable. Validation checks the stored hash;
# serialization and live mutation do not run that check.
class DatasetVersion(FrozenModel):
    dataset_id: str
    version: str
    provenance: Mapping[str, str]
    examples: tuple[Example, ...]
    created_at: AwareDatetime = Field(default_factory=lambda: datetime.now(UTC))

    @staticmethod
    def compute_version(
        provenance: Mapping[str, str],
        examples: Sequence[Example],
    ) -> str:
        return canonical_hash(
            {
                "provenance": dict(provenance),
                "examples": [e.model_dump(mode="json") for e in examples],
            }
        )

    @classmethod
    def create(
        cls,
        *,
        dataset_id: str,
        provenance: Mapping[str, str],
        examples: Sequence[Example],
    ) -> DatasetVersion:
        return cls(
            dataset_id=dataset_id,
            version=cls.compute_version(provenance, examples),
            provenance=provenance,
            examples=examples,
        )

    @model_validator(mode="after")
    def check_version(self) -> DatasetVersion:
        expected = self.compute_version(self.provenance, self.examples)
        if self.version != expected:
            raise ValueError(
                f"version mismatch: expected {expected}, got {self.version}"
            )
        return self

The method parameters accept a Sequence so callers can provide either a list or tuple of examples; the model field is a tuple. This factory is deliberately narrower than Pydantic’s constructor: it expects Example objects, not raw example dictionaries.

An after-validator sees the fields after Pydantic has processed them. That is the representation the stored hash must agree with. The important round trip now works:

ex = Example(id="q1", inputs={"question": "2+2?"}, expected="4")
ds = DatasetVersion.create(
    dataset_id="arith",
    provenance={"source": "handwritten"},
    examples=(ex,),
)
assert DatasetVersion.model_validate_json(ds.model_dump_json()) == ds

Change expected in the dumped JSON while keeping version, and loading raises a version-mismatch validation error. The check compares content with the stored digest; it is independent of the filename and JSON whitespace.

How much does hashing twice cost?

“Compute the hash” includes more than SHA-256 here. Each pass dumps every example to a dictionary, builds the payload, sorts and encodes JSON, allocates UTF-8 bytes, and finally computes the digest. The creation path repeats that entire pass.

The benchmark uses synthetic ASCII questions with the schema above, on an AMD Ryzen 7 5800H running Linux, with Python 3.14.3, Pydantic 2.13.5, and pydantic-core 2.46.5. Example construction and fixture generation happen outside the timed region. Each operation gets three warmups, then nine batches in a seeded shuffled order, with garbage collection enabled. Results are median milliseconds per call across batches; batches use 100 calls for the 20- and 100-example fixtures, 20 for 1,000 short questions, 3 for 10,000 questions, and 2 for 1,000 long questions. These are local microbenchmarks, not disk-read timings or measurements of an LLM evaluation run.

“Validate” below constructs the same model with an already supplied version and checks it once. “Create” uses the factory and hashes twice. Their difference estimates the duplicate pass under these conditions; validation with a supplied version is not a complete replacement for creating a new identity.

For twenty examples with 1,024-byte questions, a full hash pass took 0.11 ms, while SHA-256 over prebuilt bytes took 0.011 ms. At 10,000 examples, those figures were 60.65 ms and 5.55 ms. Across the fixtures, SHA-256 accounted for roughly 6–13% of the full pass. Most of the measured work was preparing and serializing the content. Example count matters as well as byte size, which is why these results should not be reduced to a single cost per MiB.

Loading prebuilt JSON, including parsing, model construction, and one hash check, took 0.14 ms for twenty 1,024-byte questions and 85.55 ms for 10,000. Avoiding the factory’s duplicate work would not remove that load-time check.

My expected datasets are around twenty examples. Their input lengths are not fixed here, so the benchmark covers 256-byte, 1,024-byte, and 16,384-byte questions. The duplicate pass adds about 0.06, 0.11, and 1.27 ms respectively on this machine. That supports keeping the explicit factory for this scale. The larger fixtures show why I would revisit the choice if the datasets or creation frequency grow. These synthetic string-only records do not measure all the fields in the full evaluation schema.

Moving hash generation into a mode="before" validator is not by itself a fix. If the after-validator remains, creation still hashes twice. If it is removed, hashing raw input can disagree with the normalized content that Pydantic produces. A design that hashes once needs to validate the content first, then either generate a missing identity or compare a supplied one, while deciding whether persisted records are allowed to omit their version. That is a change to the construction contract, not just a decorator swap.

To reproduce the measurements, save the model, benchmark, and tests in one directory. The raw results include every batch timing and its range.

uv run --python 3.14.3 --with pydantic==2.13.5 benchmark.py
uv run --python 3.14.3 --with pydantic==2.13.5 --with pytest pytest -q

A passing check has a time and a scope

The stored hash establishes agreement when the check runs. Ordinary JSON dumping does not run the validator. This remains possible:

ds.examples[0].inputs["question"] = "changed"
text = ds.model_dump_json()  # succeeds, carrying the stale version
DatasetVersion.model_validate_json(text)  # raises: version mismatch

My harness rechecks the dataset against the manifest before a run starts. That matters because a load check alone does not cover mutations between loading and execution. The remaining condition is that the evaluator uses the content that was checked: rechecking and then continuing to share mutable objects still leaves room for a later mutation. An immutable snapshot is one way to keep the check and execution tied to the same values.

For a small record, a checked serialization path can validate the exact text it is about to write:

text = ds.model_dump_json()
DatasetVersion.model_validate_json(text)
# Write this exact text only after validation succeeds.

That adds parsing and hashing work, but it checks the snapshot being written. It does not retroactively establish which data an earlier evaluation used.

Nor is the stored digest proof of authenticity. Someone who changes both the content and the hash can produce a self-consistent record. When retrieving a dataset for a run, its version also needs to match the expected version recorded independently in that run’s manifest. Authenticating the origin of the record would require a separate trust mechanism.

Pydantic has deliberate validation bypasses too: model_construct skips validation, and model_copy(update=...) trusts its updates. The downloadable tests demonstrate that a copied model can carry a wrong version until it is explicitly validated. See the model construction documentation and copy API.

Tests should preserve the limits too

The accompanying tests check JSON round trips, changed content with a stale hash, the identity policy, a pinned expected digest, and the failure of the computed-field and mapping-proxy approaches. They also keep the less comfortable cases visible: nested mutation succeeds, unchecked copying bypasses the invariant, and replacing both content and hash passes the internal check.

A golden digest guards the byte recipe across changes to the code and its dependencies. If serialization changes, old records may fail validation and newly created records may get different versions. That is a reason to investigate the change, not update the fixture automatically. As the schema grows, a named hash-format version and migration policy can distinguish recipes that all happen to use SHA-256.

Content identity is also separate from model equality and Python hashability. Two records can share version while differing in dataset_id or created_at. And although frozen Pydantic models can generate a Python __hash__, dictionary fields make these instances unhashable. The content digest string is the identifier used here.

The practical constraint is now clearer than “make the model frozen”: the bytes or values used for an evaluation must still agree with the identity recorded for that evaluation. A stored hash gives me a check I can place at that boundary. The measurements tell me what each check costs; shallow freezing does not make the check unnecessary.