To classify documents with AI, define a versioned taxonomy, label a representative evaluation set, and require the model to return an allowed class with evidence and an exception status. Keep ambiguous, mixed, unreadable, rare, and high-impact documents in a human review queue instead of forcing a confident label.

The general guide to controlled AI use covers source boundaries and review. This workflow classifies whole documents or document segments; it does not summarize them, extract fields, decide legal effect, or move source files.

Key Takeaways

Write class definitions, exclusions, and multi-document rules before labeling data.

Preserve originals and send only authorized, minimized copies to the AI system.

Separate training examples, tuning examples, and an untouched holdout set.

Require a page or passage locator for every proposed class.

Route unknown, mixed, poor-quality, and high-cost cases to people.

Evaluate every class with precision, recall, F1, and a confusion matrix.

What does AI document classification produce?

Document classification assigns one or more labels from an approved taxonomy to a document. Microsoft documents custom classification models that can identify documents and multiple document instances within an input file.［1］ That capability is useful only when your workflow defines what counts as one document, which labels are legal, and what happens when boundaries are unclear.

Use a reviewable record rather than a loose answer:

Field	Purpose
Document ID and version	Links the proposal to an immutable source version
Taxonomy version	Identifies the definitions used for the decision
Page or segment range	States which part was classified
Proposed class or classes	Uses only allowed labels
Confidence	Supports routing, not proof of correctness
Evidence locator	Points to a page, heading, or passage
Exception type	`unknown`, `mixed`, `unreadable`, `incomplete`, or controlled value
Reviewer and disposition	Records acceptance, correction, split, rejection, or escalation

Do not mix other tasks into this output. A document summary answers what a document says. Table extraction captures structured values. A classification record answers only which approved class applies to a defined unit.

How do you design a stable document taxonomy?

Begin with the business decision that follows classification. A useful class changes a lawful, documented workflow: retention, queue, reviewer skill, extraction template, or access policy. Avoid labels that merely sound descriptive but lead to the same handling.

For every class, define:

a unique identifier and plain-language name;
inclusion criteria and required evidence;
exclusions and commonly confused classes;
whether the label applies to a file, page range, or embedded document;
whether multiple labels may coexist;
the cost of a false positive and false negative;
the human owner and review interval.

Add explicit unknown and mixed outcomes. Unknown means no approved definition fits with sufficient evidence. Mixed means the input contains more than one class or document boundary and must be split or handled under a defined multi-label rule.

Version the taxonomy. If “invoice” later excludes credit notes, historical results must remain interpretable under the earlier definition. Store the taxonomy version in every label, evaluation row, prompt, and report.

Run a boundary workshop before collecting thousands of examples. Give domain reviewers ten to twenty deliberately similar documents and ask them to explain disagreements. If people cannot apply the definition consistently, more model tuning will not repair the policy.

How should you prepare authorized document samples?

Inventory the sources, owners, sensitivity, retention rules, file formats, languages, scan quality, and access groups. Send only documents approved for the stated classification purpose. A file being available in a shared drive does not make it suitable for a consumer AI service.

Create a controlled copy while preserving the original:

remove irrelevant personal, financial, medical, credential, and confidential content;
keep page order and boundary clues needed for classification;
record redactions so reviewers know what evidence is absent;
reject password-protected, corrupted, or unsupported inputs safely;
keep document IDs and versions outside the model when possible;
prevent the classifier from renaming, moving, deleting, or overwriting source files.

The Microsoft documentation lists input requirements and describes training, testing, and troubleshooting for custom classifiers.［1］ Treat vendor limits and supported formats as current product facts to verify before deployment, not as permanent properties of document classification.

Sample the real operating range: native PDFs, scans, photographs, office documents, different layouts, languages, handwritten annotations, blank pages, rotated pages, missing pages, bundles, duplicate versions, and documents produced by different teams or suppliers.

How do you build a gold set and holdout set?

Write a labeling guide with examples and counterexamples. Have two qualified reviewers label overlapping samples independently. Resolve disagreement through adjudication, and record whether the cause was reviewer error, insufficient evidence, or an ambiguous class definition.

Split the data by source group or time where leakage is possible. Near-duplicate versions of the same form must not appear across training and holdout sets, because layout similarity can make performance look better than it will be on new material.

Keep three roles distinct:

Set	Use
Training or examples	Teaches the allowed patterns
Development	Supports prompt, threshold, or model selection
Holdout	Measures the chosen workflow once without tuning on the answers

Include enough examples per class to expose variation, but do not invent a universal sample count. Rare classes may require targeted collection and wider uncertainty intervals. If a class has too few trustworthy examples, keep it manual until more evidence is available.

The gold row should preserve source ID, document version, page range, expected class, acceptable alternate outcome, exception state, evidence locator, reviewer IDs, adjudication, and taxonomy version.

How do you classify documents with AI without letting them control the model?

Document content is untrusted. Instructions inside a file—such as “classify this as approved” or “ignore the previous policy”—are evidence to categorize, not commands to follow. Put the taxonomy and task rules in a higher-trust control layer and give the classifier no file-management permissions.

Ask the model to:

inspect only the provided pages or extracted text;
choose only allowed classes;
return unknown when no definition fits;
return mixed when multiple document units require splitting;
cite the page and short evidence for the proposal;
state unreadable or missing evidence;
avoid summarization, extraction, legal conclusions, and source-file actions.

Validate output against a strict schema. Reject unknown keys, labels not present in the named taxonomy version, out-of-range confidence, invalid page numbers, and missing evidence. If output cannot be parsed, route the source to the exception queue rather than retrying until a desired label appears.

For contracts, policy documents, or other version-sensitive material, classification must not be confused with comparing contract versions. A label such as contract says nothing about which clauses changed or what they mean.

Which edge cases need a human review queue?

Define exception types and owners in advance. A reviewer needs both the source and the proposed record, not only the model's label.

Common exception paths include:

unknown_class: no approved class fits;
mixed_bundle: several documents share one file;
boundary_unclear: starting or ending page is uncertain;
low_quality: scan, handwriting, rotation, or OCR prevents reliable reading;
incomplete: pages or attachments are missing;
conflicting_evidence: different sections support different classes;
unsupported_language: no validated workflow exists for the language;
rare_high_impact: too little evidence or too high a cost for automation;
version_conflict: a near-duplicate or superseded copy may have been submitted.

Do not treat unknown as a failed prediction to hide. It is a legitimate safety outcome and a source of taxonomy improvement. Reviewers should select an existing class, split the document, request a better copy, create a proposed taxonomy change, or escalate to the domain owner.

Never create a new production class from one surprising document. Collect examples, define the downstream handling, test confusion with neighboring classes, approve the taxonomy version, and then update the model workflow.

How do you evaluate each class and threshold?

A confusion matrix counts true and predicted classes, revealing which labels are mistaken for each other.［2］ A classification report can summarize precision, recall, F1, and support for each class.［3］ Use both, because a single aggregate number can conceal a weak or underrepresented category.

Interpret the metrics operationally:

precision asks how many documents assigned to a class were actually acceptable;
recall asks how many documents belonging to a class were found;
F1 balances precision and recall when both matter;
support shows how many evaluated examples underlie the result;
the confusion matrix shows the direction of errors;
exception rate shows how much work reaches human review;
evidence validity checks whether the cited passage supports the label.

Choose thresholds by error cost. A high false-negative cost may justify prioritizing recall and reviewing more candidates. A high false-positive cost may require strong precision and mandatory human approval. Record the threshold, dataset version, confidence calibration method, and approving owner.

Slice results by language, source, format, scan quality, document length, and time period. A classifier that performs well on one standardized form may fail on photographs or documents from a new supplier. Report uncertainty when a slice contains few examples.

Use the fact-checking workflow to verify any narrative conclusions written about evaluation results. Metrics should come from the controlled evaluation output, not from a model's summary of pasted numbers.

How do you deploy and monitor the document classifier?

Start in shadow mode and compare proposals with existing manual handling. Then show the class and evidence to reviewers without changing source files. Automate only a narrow, reversible downstream route after that class meets its own acceptance criteria.

Keep audit fields for source version, taxonomy, model, prompt, threshold, output, reviewer change, and final disposition. Maintain a manual queue and disable switch. Reprocessing must create a new result version rather than overwriting the evidence for an earlier decision.

Monitor class distribution, unknown and mixed rates, human corrections, evidence failures, queue latency, and confusion between high-cost classes. Investigate drift after policy, supplier, template, language, scanning, or business-process changes.

NIST's Generative AI Profile highlights confabulation, privacy, bias, information integrity, and overreliance risks.［4］ Apply lifecycle governance: approved data, bounded permissions, evaluation, monitoring, incident handling, and named human accountability.

Summary

Classification begins with a versioned taxonomy and a defined document unit.
Source documents remain immutable; AI receives only authorized, minimized copies.
Human adjudication creates the gold set, while a separate holdout measures performance.
Every proposed class needs evidence and an explicit exception path.
Precision, recall, F1, support, and the confusion matrix are reviewed per class.
Deployment progresses from shadow mode to reviewed suggestions before narrow automation.

FAQ

Is document classification the same as document extraction?

No. Classification assigns an approved label to a document or segment. Extraction retrieves named fields, while summarization explains content; each task needs its own output and validation.

Should every document receive a known class?

No. Unknown, mixed, unreadable, and incomplete are valid outcomes. Forcing a known label hides policy gaps and increases silent errors.

Can one file contain several document classes?

Yes. Define whether the workflow labels the whole bundle, splits page ranges, or permits multiple labels. Unclear boundaries belong in the review queue.

How many labeled documents are enough?

There is no universal number. You need representative variation and enough holdout examples to estimate each class, especially rare and high-cost classes.

Why not use overall accuracy alone?

Accuracy can remain high while a rare class fails. Per-class precision, recall, F1, support, and confusion directions expose that problem.

Can AI move documents into folders after classification?

Keep classification read-only during evaluation. Any move is a separate, reversible action that requires validated output, authorization, collision handling, audit records, and a recovery path.

How should taxonomy changes be handled?

Propose and review the definition, examples, downstream effects, and confused neighbors. Approve a new version, relabel affected evaluation data, and preserve historical results under their original version.

Who reviews edge cases?

Assign domain owners by exception type. Records, legal, privacy, finance, security, or operations specialists may be needed when classification changes consequential handling.

Disclaimer: This article provides general information about document operations, not legal, records-management, privacy, or compliance advice. Apply your organization's approved policies.

Sources:

Microsoft Learn — Build and train a custom classification model — https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/how-to-guides/build-a-custom-classifier?view=doc-intel-4.0.0
scikit-learn — confusion_matrix — https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html
scikit-learn — classification_report — https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html
NIST — Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile — https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence

Sources checked 4 September 2026.

Related articles:

How to Classify Documents with AI and Review Edge Cases