Can Transformers Reliably Anonymize Legal Documents?

Courts publish their decisions so that the law can be read, cited and criticized. Those same decisions name the people involved: litigants, witnesses, lawyers, sometimes children. Before a judgment can be released, someone has to find every identifying mention and replace it, without destroying the legal reasoning around it.

That sounds like a solved problem. Named entity recognition (NER) models have tagged people, organizations and places in news text for two decades [5]. But a judgment is not a news article. Here is a typical sentence from EU legal text, with the entities our model is asked to find:

Most of the entities in that sentence are not private at all. They are the legal content. An anonymization system therefore has two jobs: recognize everything, then mask only what identifies a person. If the first step fails on a name, the document leaks. If it confuses a regulation with an organization, the masking policy can no longer be trusted.

In this post I go through LegNER, a model my coauthors and I built for this problem [1], organized around two questions:

1. Does adapting a transformer to legal language improve legal entity recognition over existing general-purpose and legal-domain models?
2. Does high recognition accuracy translate into reliable redaction?

The short answers: the first is a clear yes in our setting, and the second is “not automatically”, for reasons that are worth spelling out.

Why legal text breaks general-purpose NER

Standard NER benchmarks such as CoNLL-2003 define four entity types: person, organization, location and miscellaneous [5]. Legal text needs more. A judgment refers to statutes (“Article 6(1) of Regulation 2016/679”), to other cases (“C-123/20”), to institutions with long compositional names, and to the same party under several roles (“the applicant”, “the claimant”, “Ms Thompson”). Sentences run for many clauses, and entities nest inside one another: “the Court of Justice of the European Union” contains “the European Union”.

There is also a quieter problem at the level of the vocabulary. BERT splits text into WordPiece units drawn from a fixed vocabulary [2]. Legal identifiers are rarely in that vocabulary, so they arrive at the model already broken into fragments:

A model can still learn to recognize “c - 123 / 20” as a case reference, but it has to learn it from context, using a vocabulary built for Wikipedia and books. That is the gap domain adaptation is meant to close.

Building a corpus where entities carry legal meaning

We built the dataset from EUR-Lex, the EU’s official legal database [4]. We queried court decisions published between 2015 and 2023 on data protection, administrative law and competition policy. We kept only documents with stable identifiers, complete rulings and machine-readable XML, then removed duplicates and purely procedural notes. That left 1,542 full-text judgments, used in their English versions.

Each judgment was annotated for six entity types: PERSON, ORGANIZATION, LOCATION, DATE, LAW and CASE_REFERENCE. Annotation ran in two stages. About 600 documents (39%) were annotated in full by trained legal experts. The remaining 942 were pre-labelled automatically, using legal lexicons, regular expressions and weak NER baselines, and every suggestion was then manually checked and corrected. The corpus contains roughly 34,000 sentences, 1.2 million tokens and 287,000 entity mentions:

Annotations are stored as character-offset spans rather than word labels. This matters later, when the text is cut into subword tokens whose boundaries rarely match word boundaries:

{ "text": "On March 15, 2023, Judge Alice Thompson ruled in favor of the European Commission in case C-123/20 concerning GDPR compliance in Berlin.",
  "entities": [
    { "type": "DATE",           "text": "March 15, 2023" },
    { "type": "PERSON",         "text": "Alice Thompson" },
    { "type": "ORGANIZATION",   "text": "European Commission" },
    { "type": "CASE_REFERENCE", "text": "C-123/20" },
    { "type": "LAW",            "text": "GDPR" },
    { "type": "LOCATION",       "text": "Berlin" } ] }

The illustrative annotation from the paper, shortened here: each entity also stores its start and end character offsets.

We split the judgments by document, stratified so that every entity type appears in proportion: 1,079 for training, 231 for validation (early stopping and tuning) and 232 for the final test. Splitting by document rather than by sentence keeps sentences from the same case out of both training and test.

Adapting BERT to legal language

LegNER starts from BERT-base: 12 encoder layers, a hidden size of 768, 12 attention heads and about 110 million parameters [2]. Domain adaptation enters at three points.

• Vocabulary. The 30,522-entry vocabulary was extended with 1,245 legal terms taken from EUR-Lex and curated legal dictionaries. This targets the fragmentation shown in Figure 2.
• Pre-training. Masked language modelling on unannotated EUR-Lex text teaches the encoder how statutes, citations and institutions tend to co-occur before it sees a single label.
• Span-aligned fine-tuning. The model is trained as a token classifier with BIO tags. When a word is split into several subword pieces, the first piece inherits the word’s tag and the rest are marked as continuations, so an entity boundary never falls inside a word.

Under BIO tagging, “Judge Alice Thompson” contributes B-PERSON I-PERSON I-PERSON if the title is part of the span, and every other token is O. The whole system sits inside the loop shown in Figure 4. Documents flow into pre-training, models are fine-tuned on the corrected annotations, and their errors are fed back into the annotation guidelines when performance is not yet acceptable.

Setup

• Backbone: BERT-base, 12 layers, hidden size 768, 12 heads (~110M parameters)
• Vocabulary: 30,522 WordPiece entries + 1,245 legal terms
• Objective: token classification over BIO tags, cross-entropy loss
• Optimizer: AdamW, learning rate 3×10⁻⁵ with linear decay, weight decay 0.01
• Training: batch size 16, 8 epochs, early stopping on validation F1, max sequence length 512
• Hardware: one NVIDIA A100 (Google Colab Pro)

We evaluate with strict span matching: a prediction counts only if both its boundaries and its type are correct. A model that tags “Court of Justice” but misses “of the European Union” gets no credit for that entity. Precision, recall and F1 are computed over entities:

For anonymization, recall is the number that matters most. A false positive masks a word that did not need masking. A false negative leaves a name in a published judgment.

What the experiments show

The first thing to establish was whether all this adaptation buys anything over models that are already legal-domain. We compared LegNER with five baselines that each represent a different design choice: Legal-BERT, pre-trained on legal corpora [3]; a legal RoBERTa, with byte-pair tokenization [8]; SpanBERT, with span-level pre-training [7]; a ContractNLI model trained on contracts [6]; and LegalNER.pt, a BERT model trained on Portuguese legal text. Every model was fine-tuned on the same 1,079 training judgments and scored on the same 232 test judgments.

What we observed. LegNER had the highest precision, recall and F1, with precision and recall balanced (99.5 vs 99.3). It was also the fastest model in this setup, at 12.3 documents per second against 8.6–11.4 for the baselines, at a comparable parameter count. Among the baselines, the two models pre-trained on legal corpora, Legal-BERT and RoBERTa-Legal, came first, and the Portuguese model came last.

How we read it. Two things seem to matter. Vocabulary and pre-training close to the target documents help, and the Portuguese model’s 10-point gap suggests that “legal” is not one domain: legal language from another jurisdiction and language transfers only partly. The accuracy column, by contrast, says little. Most tokens in a judgment are outside any entity, so token-level accuracy sits above 95% for every model and hides the differences F1 exposes.

A final score does not show whether training was stable, so we also looked at how the metrics moved from epoch to epoch. The interesting part is the first two epochs, where precision and recall move in opposite directions.

What we observed. Precision rose from 85.2% to 98.5% between the first and second epochs while recall fell from 99.5% to 92.3%. By epoch 3 both had recovered to 99.5% and 99.3%, and F1 stayed at 99.4% for the remaining epochs. The loss fell from 0.022 to 0.001 after the second epoch and stayed below 0.001.

How we read it. The pattern is what boundary learning should look like. Early on the model marks too much as entities, then becomes conservative, then finds where spans start and stop. The flat curves after epoch 3 suggest that eight epochs were more than enough, which fits early stopping on validation F1. What the curves cannot tell us is how the model behaves on documents unlike the training distribution. A plateau at 99.4% on in-distribution data is a statement about this corpus.

An aggregate score can hide a weak class, and for anonymization the classes are not equally important. Missing a date is a nuisance. Missing a name is a disclosure. So we broke the test results down by entity type.

What we observed. Dates were the easiest class (99.5% F1), followed by case references (99.0%) and people (98.9%). Organizations (97.1%) and statutes (98.0%) were the hardest. Organization recall, at 96.8%, is the lowest single number in the table.

How we read it. The hard classes are the long, compositional ones. “European Commission Directorate-General for Competition” can be cut in several plausible places, and statutes are cited in many formats (“Directive 95/46/EC”, “Article 6(1)(f)”, “the GDPR”). Under strict matching, a boundary error is a full miss. Dates and case numbers follow regular patterns, which is presumably why they are easiest. The last column is ours, not the paper’s. It is a reminder that what gets masked is a policy decision made after recognition. For an organization, it depends on whether it identifies a private party or is an institution like the Commission.

Recognition is only the first half of anonymization. The second half replaces each sensitive span with a placeholder, and two things can go wrong there that F1 does not measure. The same person can get different placeholders in different paragraphs, which makes the judgment unreadable. Or the replacement can damage the sentence enough that the legal reasoning no longer comes through. We therefore defined two anonymization-specific measures. The consistency rate is the share of replacements that agree with the placeholder already assigned to the same entity:

The second is readability: legal experts scored the anonymized documents on a five-point Likert scale, from 1 (incoherent) to 5 (fully coherent).

What we observed. In a manual review of 12 anonymized case documents, LegNER reached a consistency rate of 98.7%. Legal experts gave the anonymized documents an average readability score of 4.8 out of 5.

How we read it. Numbered, typed placeholders are a good compromise. “[PERSON-1] v [ORGANIZATION-1]” keeps the structure of an argument that bare black bars would destroy. But note what the example also shows: roles, the court type and the procedural posture all survive. That is deliberate, and it is also exactly the information that can re-identify a party in a small jurisdiction or a well-reported case.

This last question goes beyond the paper, but it is the one a court or a data-protection officer would ask. NER metrics count mentions. A redaction succeeds or fails per document, and one surviving name is enough for it to fail.

A rough calculation shows why the distinction matters. Take the reported PERSON recall of 99.1%, so a miss rate of \(m = 0.009\) per mention. If a judgment contains \(n\) person mentions and misses were independent, the probability that at least one survives is

The independence assumption is wrong in both directions. Misses cluster: an unusual foreign name, an OCR-damaged paragraph or a name in a footnote tends to be missed every time it appears, so real documents fail less often than the formula says. But when they fail, they fail more completely. Consistent replacement also cuts both ways. It spreads a correct detection to every repetition of a name, and it can spread a miss just as easily. What the calculation does show is that mention-level recall in the high nineties does not imply document-level safety. The number that would settle the question is the share of test documents with no residual identifiers, a metric we did not report.

A second gap is conceptual. Masking names, organizations, places and dates removes direct identifiers. It does not remove quasi-identifiers, combinations of facts that point to one person. Under the GDPR, data that can still be attributed to a person using additional information is pseudonymised, not anonymous, and remains personal data [11]. Work on text anonymization makes the same point empirically: NER-based de-identification and anonymization are different tasks, and evaluating the latter needs annotations of what actually discloses identity in context [9][10].

Where the approach is likely to fail

The paper names a few recurring failure types. Most were identified through the feedback loop of Figure 4.

• Very long, clause-heavy sentences. The remaining errors concentrate there. Sentence structure, more than vocabulary, seems to drive the difficulty.
• Nested and compositional entities. “Court of Justice of the European Union” inside a longer reference, or multi-part directorate names, where a boundary error turns into a full miss under strict matching.
• Ambiguous abbreviations that can name a court, a party or a statute depending on context.
• Formatting noise. OCR artifacts, repeated headers and footers were filtered out during corpus construction. Scanned archives would bring them back.

Two further limits come from the evaluation design rather than from the model. All documents come from one source (EUR-Lex) in one language (English), so performance on national courts, other languages, contracts or scanned historical archives is untested. And 61% of the corpus was pre-labelled by lexicons, regular expressions and weak models before human correction. Human review limits the risk, but a model trained and tested on such labels can be partly rewarded for reproducing the pre-labellers’ conventions. Scoring the expert-only documents separately would show how large that effect is.

What we learned

1. Domain adaptation helped, and “legal” is not one domain. Under the same split and the same strict metric, the adapted model beat every legal-domain baseline. The model trained on another jurisdiction’s language was furthest behind.
2. The hard cases are long, compositional entities. Organizations and statutes, not names, set the floor of the per-entity results.
3. Recognition and redaction are different questions. High mention-level F1 is necessary for safe anonymization but does not guarantee it. Document-level leakage and quasi-identifiers need their own evaluation.
4. What to mask is a policy decision. The same recognizer serves privacy (persons, places, dates) and legal structure (statutes, case references). Separating the two decisions makes both easier to audit.

Limitations and open questions

• Generalization. How does LegNER perform on national court decisions, other EU languages, and OCR-scanned documents?
• Attribution of gains. How much of the improvement comes from the vocabulary extension, the domain pre-training and the span alignment individually? An ablation would answer this.
• Variance. Repeated runs with different seeds, and confidence intervals on the test metrics, would show which differences between models are stable.
• Annotation quality. The paper reports no inter-annotator agreement and no separate results for expert-annotated versus semi-automatically annotated documents.
• Document-level privacy. What fraction of test documents is left with no direct identifier at all? How often do the remaining roles, dates and facts allow re-identification? Benchmarks built for text anonymization, such as TAB [10], are designed to measure this.
• Deployment. Nothing here certifies GDPR compliance or production readiness. Any real redaction pipeline would need human review, especially for judgments involving minors or other vulnerable parties.

@article{karamitsos2025legner,
  title   = {{LegNER}: A Domain-Adapted Transformer for Legal Named
             Entity Recognition and Text Anonymization},
  author  = {Karamitsos, Ioannis and Roufas, Nikolaos and
             Al-Hussaeni, Khalil and Kanavos, Andreas},
  journal = {Frontiers in Artificial Intelligence},
  volume  = {8},
  pages   = {1638971},
  year    = {2025},
  doi     = {10.3389/frai.2025.1638971}
}