An AI interview scorecard should start from an approved job analysis, essential functions, and job-related competencies—not the job description alone. AI can organize evidence into consistent questions, permitted probes, and draft behavior anchors, but trained interviewers must validate the scorecard, provide accommodations, record evidence, resolve disagreements, and make every employment decision.

This is a design workflow for the interview instrument; it does not process candidate material. The responsible AI guide explains the broader evidence, privacy, and human-approval boundary.

Key Takeaways

Treat the job description as one input, not the sole definition of success.

Trace every assessed competency to current job-analysis evidence and essential work.

Ask candidates the same core questions under a documented structured process.

Define observable behavior anchors before anyone is interviewed.

Remove protected characteristics, proxies, personality guesses, and irrelevant preferences.

Calibrate raters with synthetic responses and preserve not observed as a valid outcome.

Keep candidate data, scoring, ranking, accommodation, and hiring decisions outside AI.

Why is a job description not enough?

A job description may summarize duties, reporting lines, and qualifications, but it can be outdated, promotional, incomplete, or written before the assessment strategy. It may mix essential functions with preferences or use broad traits such as “executive presence” that are difficult to observe consistently.

OPM describes job analysis as a systematic process for gathering and documenting the content, context, and requirements of work, including the relationship between job tasks and the competencies needed to perform them. It is a foundation for structured interviews and other assessments.［1］ The scorecard therefore needs a trace from each competency and question back to approved job evidence.

If the description itself needs correction, use the separate job-description drafting workflow and obtain approval before deriving an assessment.

Step 1: Freeze the role evidence and governance boundary

Record the role version, location or jurisdiction, job-analysis owner, approval date, essential functions, critical tasks, entry requirements, assessment policy, scorecard owner, and effective date. Name what is out of scope, including culture-fit impressions, medical information, protected characteristics, and competencies measured by another approved assessment.

Create an evidence header:

Field	Synthetic example
Role/version	`Service Operations Analyst / v4`
Job analysis	`JA-204`, approved current version
Essential functions	Referenced by IDs `EF-01` to `EF-05`
Jurisdiction	Confirmed by HR; not inferred by AI
Assessment owner	Talent Assessment role
Interview scope	Competencies `C-02`, `C-04`, `C-07`
Excluded data	Resumes, recordings, demographic and medical data
Approval	HR, hiring owner, and legal review as locally required

Do not ask a model to determine which law applies or whether a task is legally essential. Those are governed decisions based on local facts and advice.

Step 2: Select observable, job-related competencies

Start with competencies linked to critical tasks in the approved job analysis. Confirm that each is needed on entry or is appropriately assessed at the selected stage. Avoid interviewing for a competency already measured more validly elsewhere unless the assessment strategy explains the overlap.

OPM says structured interviews measure job-related competencies through systematic questions about past behavior or proposed behavior in hypothetical situations. It also emphasizes predetermined questions and common rating standards.［2］ Its development guide begins with job analysis, competency selection, question format, rating scales, probes, pilot testing, interviewer guidance, and documentation.［3］

Use a competency map:

Field	Record
Competency ID/name	Stable, approved label
Definition	Job-specific meaning, not a dictionary stereotype
Linked tasks/functions	Evidence IDs from job analysis
Entry requirement	Why it must be assessed now
Observable behavior	What an interviewer can hear in a response
Exclusions	Traits or proxies that are not part of the competency
Assessment method	Interview, work sample, or another approved method
SME confirmation	Reviewer and date

Prefer observable actions and decisions over personality adjectives. “Explains a service interruption to a nontechnical stakeholder using accurate, proportionate language” is easier to assess than “charismatic communicator.”

Step 3: Build the scorecard structure before questions

Define one controlled template so every interviewer records the same kinds of evidence:

Scorecard field	Purpose
Role and scorecard version	Connects ratings to the approved instrument
Competency and rationale	Shows job-related basis
Question ID and wording	Preserves consistency
Allowed probes	Clarifies permitted follow-up
Evidence indicators	Observable content relevant to the competency
Behavior anchors	Predefined differences between rating levels
Rating scale	Approved labels and meanings
`Not observed`	Prevents guessing when evidence is absent
Rater evidence notes	Candidate's relevant actions/results, not impressions
Accommodation record	Process reference without medical detail
Disagreement rule	How independent ratings are discussed and resolved
Approval/change log	Owner, effective date, pilot, and revision history

Keep administrative eligibility checks outside the competency score. Do not add a “gut feeling” row.

Step 4: Draft consistent core questions

Choose behavioral questions about prior experience or situational questions about a relevant hypothetical, according to the approved method. Each question should be open, clear, job related, readable, and capable of eliciting evidence for one primary competency.

Avoid questions about family, health, age, nationality, religion, disability, or other protected information. Also remove indirect proxies that are not necessary for the job, such as unexplained requirements for continuous employment, particular hobbies, prestige employers, accent, or “native” communication when a defined job-related language level is what matters.

Step 5: Define allowed probes without changing the test

Probes help interviewers obtain enough detail while preserving comparable treatment. Define neutral prompts such as “What was your specific role?”, “What action did you take?”, and “What was the result?” Allow clarification of an ambiguous term without coaching toward the preferred answer.

Document when a probe is used, but do not treat the number of probes as a performance score unless the validated method says so. A candidate may need clarification because of question wording, interpretation, disability, or communication style; probe count is not a safe proxy for competence.

Interviewers should not improvise harder follow-ups for one person or offer hints only to another. Log novel clarifications and consider a controlled revision after the session.

Step 6: Draft behavior anchors from supplied evidence

Behavior anchors describe observable differences in a response. They should reflect the job context and approved competency definition, not labels such as “excellent attitude.” Build them before seeing candidates.

Synthetic example for incident communication:

Rating state	Behavior anchor
Evidence insufficient	No relevant action or result can be identified
Basic evidence	States the issue and gives a factual update to the intended audience
Effective evidence	Adapts detail, confirms understanding, and explains next steps accurately
Strong evidence	Also handles uncertainty, stakeholder impact, escalation, and feedback proportionately
Not observed	Question or response did not provide assessable evidence

Labels and levels come from the approved method. An anchor is not a keyword-matching model answer; it helps raters compare observed behavior with a common standard.

For a general framework on building criteria and anchors, see grading-rubric design, but do not import classroom assumptions into an employment assessment.

Step 7: Draft the AI interview scorecard as a bounded instrument

Provide only the approved role evidence, competency map, template, writing constraints, and synthetic examples. Do not provide resumes, interview recordings, candidate answers, names, contact details, or demographic information.

Using only the supplied approved job-analysis evidence, draft scorecard rows.
Trace each competency and question to supplied task or essential-function IDs.
Keep core questions consistent and list only neutral permitted probes.
Draft observable behavior anchors without personality, emotion, disability,
protected traits, employment-history proxies, or cultural-fit language.
Do not add qualifications, score candidates, rank people, or recommend hiring.
Mark missing job evidence, policy, and legal review as [VERIFY].

AI does not score candidates in this workflow. It also does not infer whether an answer is truthful, detect emotion, evaluate facial or vocal behavior, or predict future performance. Those uses introduce legal, accessibility, and privacy questions and are outside the scorecard-drafting scope.

Step 8: Screen for protected traits and proxy criteria

Review every competency, question, probe, anchor, and note field with HR and legal owners. Ask whether it measures essential work, whether a less exclusionary method is available, and whether the language could invite protected information or reward a proxy unrelated to performance.

The EEOC and U.S. Department of Justice warn that AI and software used in employment decisions can disadvantage people with disabilities. Their guidance highlights reasonable accommodation, risks of screening out qualified people, and disability-related inquiries.［4］ This is U.S. guidance; employers must determine all applicable local requirements.

Establish an accommodation process before interviews begin. Tell candidates how to request one through an approved route, train interviewers on adjusted delivery, and avoid putting diagnosis or medical details into the scorecard. An accommodation may change presentation or response conditions without changing the job-related competency being assessed.

Review language, technology, timing, location, and sensory demands. The interview must not add barriers the job itself does not require.

Step 9: Pilot with synthetic responses and calibrate raters

Create synthetic answers that cover clear, borderline, contradictory, and insufficient evidence. They must not imitate real applicants. Have raters score independently, cite the words or actions that support each rating, and use not observed when evidence is absent.

Compare whether raters interpreted the competency differently, an anchor combined too many behaviors, a question failed to elicit evidence, or a probe changed difficulty. Check whether style, confidence, accent, similarity, prestige, or an avoidable access barrier influenced the result. Revise the instrument and version until authorized reviewers consider it usable; do not manufacture a reliability statistic from a small synthetic pilot.

NIST's generative-AI profile identifies confabulation, privacy, harmful bias, and human-AI configuration risks.［5］ Human review, data minimization, and traceability are controls, not proof that an assessment is fair or valid.

Step 10: Train interviewers and control live use

Train every interviewer on the role evidence, competencies, exact questions, permitted probes, anchors, note taking, accommodations, conflicts, privacy, independent scoring, disagreement resolution, and prohibited inferences. Provide one versioned interviewer guide.

During interviews, ask the approved questions and record job-relevant evidence. Do not paste answers into AI or use it to summarize tone, personality, disability, emotion, honesty, or “fit.” Store scorecards under approved access and retention rules.

Raters complete independent notes before panel discussion. Resolve differences from anchors and evidence, preserve dissent, and follow the escalation rule.

This article is not candidate coaching; candidates looking to prepare can use the separate interview-practice guide, which has a different audience and data boundary.

Step 11: Approve, monitor, and revise the scorecard

The scorecard owner, HR, hiring owner, assessment specialist, legal adviser, accessibility owner, or other required reviewers approve the instrument according to local policy. Record effective date, eligible roles, training version, pilot evidence, known limits, and change history.

Trigger review when the job changes, essential functions are revised, a question performs poorly, an accommodation issue appears, policy or law changes, raters interpret anchors inconsistently, or monitoring identifies a fairness concern. Do not silently edit questions during an active candidate cohort.

For the broader privacy boundary around sensitive prompts and automated inference, review AI privacy risks before choosing any tool or integration.

What should the final scorecard review confirm?

Summary

Begin with approved job analysis and essential work, using the description as supporting evidence.
Select a small set of observable, job-related competencies.
Standardize questions, permitted probes, evidence notes, and behavior anchors.
Use AI only to organize supplied instrument-design evidence.
Pilot with synthetic answers, calibrate raters, and prepare accommodations.
Approve, monitor, and version the scorecard under local HR and legal governance.

Frequently asked questions

Can I build the scorecard from the job description alone?

No. The description is one input and may be incomplete or outdated. Confirm critical tasks, essential functions, entry requirements, and competency linkages through the approved job analysis and subject-matter review.

How many competencies should one interview assess?

Use only the number that can be assessed with enough questions, evidence, and time under the validated design. Move other competencies to more appropriate methods rather than giving every desirable trait one shallow score.

Must every candidate receive the same questions?

A structured process normally uses the same predetermined core questions and common rating standards. Document permitted neutral probes and accommodation-related delivery changes so consistency does not become inflexibility or an access barrier.

What makes a probe fair?

A fair probe requests missing context, action, or result without suggesting a preferred answer or adding a new competency. Define probes in advance, train interviewers, record use, and review novel clarifications after the session.

How detailed should behavior anchors be?

They should be detailed enough for trained raters to distinguish observable evidence at each level, but not scripts that reward specific phrases. Tie them to approved job behavior and test them with varied synthetic responses.

What should a rater do when evidence is missing?

Use not observed or the approved insufficient-evidence state. Do not infer capability from confidence, speaking style, resume prestige, similarity, or assumptions about what the candidate probably meant.

How does interview rater calibration work?

Raters independently score the same synthetic responses, cite evidence, compare differences, and revise unclear questions or anchors. Calibration aligns use of the instrument; it does not authorize sharing real candidate data with AI.

How should accommodations affect the scorecard?

Follow the approved accommodation process and preserve the job-related competency while adjusting unnecessary barriers in presentation or response conditions. Keep medical details out of the scorecard and consult qualified local owners.

Can AI score candidates or make the final hiring decision?

Not in this workflow. AI does not receive candidate data, does not produce ratings or rankings, and does not make recommendations. Trained raters and authorized decision makers apply the approved process, review evidence, resolve disagreements, and remain accountable.

Disclaimer: This article provides general information, not legal, employment, accessibility, or assessment advice. Requirements vary by jurisdiction and organization; consult qualified HR, legal, assessment, and accessibility professionals.

Sources

U.S. Office of Personnel Management, What Is a Job Analysis? — https://www.opm.gov/frequently-asked-questions/assessment-policy-faq/job-analysis/what-is-a-job-analysis/
U.S. Office of Personnel Management, Structured Interviews — https://www.opm.gov/policy-data-oversight/assessment-and-selection/structured-interviews
U.S. Office of Personnel Management, Structured Interview Guide — https://www.opm.gov/policy-data-oversight/assessment-and-selection/structured-interviews/guide.pdf
U.S. EEOC and U.S. Department of Justice, Warning Against Disability Discrimination — https://www.eeoc.gov/newsroom/us-eeoc-and-us-department-justice-warn-against-disability-discrimination
NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile — https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence

Sources checked 4 September 2026.

How to Turn a Job Description into an AI Interview Scorecard