How do you process documents at scale with semantic operators? | Scorable

Updated: 2026-03-22
By: Ari Heljakka

Short answer

Semantic operators are language-model-powered versions of classical data-processing primitives: semantic map (extract structured information from a document), semantic filter (decide whether a document matches a criterion), semantic reduce (summarize a group of documents). They are composable. They scale to large corpora when paired with task cascades (cheap models filter, expensive models adjudicate) and rewrite directives (decompose a heavy operator into a chain of lighter ones). They produce credible results only when every operator is treated as a versioned evaluable component scored on a calibrated ground-truth dataset.

Key facts

• Definition: Language-model-powered extensions of classical data primitives (map, filter, reduce) for unstructured document processing.
• When to use: Large-scale document workflows where deterministic rules are insufficient: extraction across heterogeneous formats, semantic deduplication, thematic summarization, policy compliance scans.
• Limitations: Cost and latency scale linearly with the number of operator calls; quality is bounded by the calibration of each operator's judge.
• Example: A team processes 200,000 legal contracts by chaining a cheap semantic filter (relevance) with a small-model semantic map (clause extraction) and a frontier-model semantic reduce (jurisdictional summary), scoring each stage against a ground-truth set.

Key takeaways

• Semantic operators are the right abstraction when "rule plus regex" stops working. Code rules first; reach for operators when the criterion is irreducibly semantic.
• Composition is where the leverage is. Chain operators across model tiers; cheap models filter, expensive models adjudicate.
• Optimization is not free. Task cascades and rewrite directives cut cost only when the decision boundary is calibrated.
• Every operator is an evaluable component with a version, a rubric, a ground-truth dataset, and a calibration agreement metric.
• Steerability beats raw accuracy. The right operator is the one a domain expert can iterate on by adjusting a rubric, not by retraining a model.

Definition

A semantic operator is a data-processing primitive whose work is performed by a language model rather than deterministic code. The three canonical operators mirror classical functional primitives.

• Semantic map. Applies an operation to each document independently and emits a structured result. Examples: extract clauses, classify sentiment, identify entities, summarize per document.
• Semantic filter. Decides whether each document satisfies a natural-language criterion and emits true or false. Examples: "is this contract subject to GDPR," "does this transcript mention a refund request," "does this document contain personal health information."
• Semantic reduce. Aggregates a group of documents into a single output. Examples: synthesize a thematic summary across a corpus, list all unique policies, produce a consolidated report.

Operators compose. A typical pipeline is a chain: filter to a relevant subset, map to extract per-document structure, group by a key, reduce to a per-group output. The chain is itself a versioned artifact; reproducing a result requires the operator versions, the model versions, the rubrics, and the dataset version.

When this matters

Semantic operators earn their keep when at least one of the following holds.

• Document heterogeneity exceeds what regex can handle. Contracts from twenty law firms have twenty layouts; semantic extraction generalizes where regex breaks.
• The criterion is inherently semantic. "Does this document discuss regulatory risk" cannot be reduced to a keyword list without losing recall.
• The corpus is large. Hundreds of thousands of documents need a substrate that scales; pairwise human review does not.
• The pipeline must evolve. Domain experts iterate by adjusting a rubric in plain English; engineers do not need to retrain a model every time the criterion shifts.
• Auditability requires per-document evidence. Each operator emits a verdict and a justification that an auditor can inspect.

How it works

A defensible semantic-operator pipeline has five components.

Component 1: Pick the operator for each step

Each step of the pipeline is one operator. The right operator depends on the cardinality.

• Map for per-document work. One input, one structured output.
• Filter for inclusion or exclusion. One input, one boolean.
• Reduce for cross-document aggregation. Many inputs, one output.

Some pipelines also use semantic join (match documents from two sets on a semantic criterion) and semantic group-by (cluster documents on a semantic key). These are sugar over the three core operators.

Component 2: Compose operators in a pipeline

The most common pattern is filter then map then reduce.

• Filter narrows the corpus to the relevant subset (cheapest if the criterion is well-specified).
• Map extracts the structured properties from each relevant document.
• Reduce aggregates the mapped outputs by a key or theme.

The chain has explicit data shapes between stages; each stage's output is the next stage's input. Type discipline between operators is what makes the pipeline reasonable to debug.

Component 3: Optimize with task cascades and rewrite directives

Cost and latency scale linearly with operator calls. Two optimizations matter.

• Task cascades. Run a cheap small model first; route the ambiguous cases to a more capable model. The decision boundary is calibrated against a ground-truth set: where the small model disagrees with the large model above a threshold, the small model defers. Most filter and simple map steps cascade well.
• Rewrite directives. Decompose one heavy operator (a complex extraction with twenty fields) into a chain of lighter ones (extract entities, then extract relationships, then validate cross-field consistency). The chain is often cheaper and more accurate than the monolithic operator.

Both optimizations need their own calibration. Cascade boundaries that look fine on a small probe regress at scale; revalidate periodically.

Component 4: Treat every operator as an evaluable component

Each operator has:

• A version (the prompt or program, the model, the rubric).
• A ground-truth dataset: hand-labeled inputs with expected outputs.
• An evaluator: deterministic checks where possible, an LLM judge where not.
• A calibration agreement metric (Matthews correlation for binary filters, rank correlation for graded scores; operational thresholds Matthews 0.6 or rank correlation 0.7).

The pipeline-level score composes the per-operator scores. A regression at the pipeline level decomposes into the operator that caused it; without per-operator scoring, the regression is opaque.

Component 5: Steerability through versioned rubrics

The point of semantic operators is that the criterion lives in a rubric, not in code. Domain experts iterate by editing the rubric. Engineering treats the rubric as a versioned artifact:

• Each rubric edit produces a new version.
• The new version is scored against the ground-truth dataset before promotion.
• Rubric changes are gated the same way prompt changes are gated.

Without this discipline, "iteration" devolves into rubric drift; the same operator scores differently from week to week because nobody is sure which rubric is in effect.

Example

A team processes a corpus of 180,000 legal contracts to answer a board question: "Which contracts include a data-residency clause that requires storage in the European Union, and how does coverage vary by counterparty type?"

Pipeline.

1. Semantic filter on the full corpus. Criterion: "does this contract include any data-residency or geographic-storage clause?" Cascade: a small model on 100 percent of traffic; a large model on the 11 percent the small model scores in the ambiguous band. Calibration set: 200 labeled contracts; current Matthews 0.71.
2. Semantic map on the filtered subset (about 24,000 contracts). Extract: jurisdiction required, exceptions clauses, counterparty type. Schema-validated output. Calibration set: 120 labeled contracts; current rank correlation 0.74 on jurisdiction extraction, 0.68 on exceptions extraction.
3. Semantic group-by on counterparty type. Deterministic grouping on the mapped field.
4. Semantic reduce per group. Synthesize a one-paragraph summary of data-residency coverage. Calibration set: 30 expert-labeled group summaries; current rank correlation 0.66.

Optimization.

• The filter cascade cuts cost by 78 percent versus running the large model on every contract; the calibrated decision boundary holds at Matthews 0.69 after the cascade.
• The map operator is rewritten from one monolithic extraction into three sub-operators (jurisdiction, exceptions, counterparty); the chained version is cheaper and scores higher on exceptions extraction (rank correlation 0.74 versus 0.65).

Evaluation.

• Each operator scores on a versioned ground-truth dataset.
• The pipeline-level score is the composed per-operator scorecard, tied to the tuple (filter v2.1, map v3.4, reduce v1.7, model versions, rubric versions, dataset v3).
• A drift alert fires when filter recall drops on the legal-services counterparty slice; root cause is a new contract template introduced two weeks earlier. Calibration set is extended; recall recovers.

The board question is answered. The pipeline is reproducible. The next time the criterion shifts (the EU adds a new data-residency requirement), only the rubric changes; the rest of the pipeline is unchanged.

Limitations

• Cost and latency scale linearly. Large corpora can be expensive. Task cascades and rewrite directives help; uncalibrated cascades hurt.
• Quality is bounded by judge calibration. An operator scoring high on a stale calibration set is not actually scoring high. Recalibrate after model updates and rubric changes.
• Composability has limits. Some operators are not order-independent (filtering after mapping can lose context the filter needed). Pipeline design matters.
• Steerability requires discipline. Rubric drift is the dominant failure mode; rubrics must be versioned and gated like any other artifact.
• Per-document justifications are not free. Storing every operator's verdict and justification at scale is a storage and access-control concern; treat the verdict log as production data.

Evidence and sources

• DocETL: A Declarative System for Scalable Document Processing with LLMs, arxiv.org/abs/2410.12189
• Lotus: Enabling Semantic Queries with LLMs over Tables and Documents, arxiv.org/abs/2407.11418
• Hugging Face Datasets for document processing pipelines, huggingface.co/docs/datasets

FAQ

How are semantic operators different from prompt chains?
A prompt chain is an ad-hoc sequence of model calls. A semantic operator is a typed primitive with a defined input and output shape, a versioned rubric, and a ground-truth dataset. Operators compose with type discipline; chains often do not.

Do I need an LLM for every operator?
No. Where a criterion can be codified deterministically (regex, schema, exact match), use a rule. Reserve semantic operators for what rules cannot capture. Many production pipelines mix both.

How do I choose the model for each operator?
Start with the cheapest model that passes the calibration threshold. Use a task cascade to route hard cases to a stronger model. Revisit the boundary periodically; model capability shifts as new versions ship.

What about latency-sensitive pipelines?
Semantic operators are batch-oriented by default; latency-sensitive applications require careful design (parallelism, caching, smaller models). For real-time use, rules and small classifiers are usually faster than even the smallest LLM operator.

How do I avoid rubric drift?
Version every rubric. Gate every rubric change against the ground-truth dataset. Tag every evaluation run with the rubric version. Without this, the same operator scores differently from week to week and nobody can tell why.