Human in the Loop
For each decision in an AI-assisted process, human-in-the-loop design sets whether a person steps in, at what moment, with what information and who answers for the result. The analyst breaks each decision into four functions: acquire the information, analyse it, select the decision, implement it. For each function the analyst sets a control mode from the cost of an error and the measured reliability of the system. The analyst then names the reviewer and the accountable person and has the oversight tested before go-live. The deliverable is a decision register.
Goal
Human-in-the-loop design places a human decision in a process where an AI system reads, classifies, proposes or acts. For each function of a decision it settles whether to automate and in which mode before the team writes the solution requirements. It also names the person who answers for the result when an error slips past the control.
The deliverable is a decision register. Each row holds one function of one decision in the process. Its columns give the cost of an error, the measured reliability, the control mode, the time per case, the reviewer and the accountable person. The register feeds the solution requirements and the RACI matrix of the process. A test plan for the oversight and a list of indicators for the human review go with it.
The technique replaces a common phrase in specifications: "a person validates the result". The phrase leaves open which cases, with what and how much time. It produces two known defects. The reviewer approves out of habit what a reliable system shows them. The human signature makes a failing system acceptable and shifts the blame onto the person who signs. The register turns the phrase into a design that the team can test.
Usage
When to use
- New AI-assisted process: set where a person decides before specifying the solution requirements.
- Agent that acts in a system: place the approval before the write, the send or the payment that the agent triggers.
- Decision that affects a person: reimbursement, job application, credit; provide for the review that the person can request.
- System classified as high-risk: design the human oversight that the AI Act requires of the deployer.
- Human review that approves almost everything: check that it still decides something.
- Model update or change of supplier: retest the oversight on the new system.
- Volume rising without extra reviewers: recalculate the time available per case and revisit the mode.
When not to use
- Personal assistant with no effect beyond its user: the author rereads the draft; a usage policy is enough.
Description
The decision and its four functions
Parasuraman, Sheridan and Wickens (2000) describe automation by function rather than by system. Their model distinguishes four classes of functions. The example follows a health insurer that processes a physiotherapy invoice.
- Information acquisition: the system reads the invoice and extracts the tariff items, the number of sessions and the date of the prescription.
- Information analysis: it compares these data with the prescription and the insured person's history, then estimates the probability of an error.
- Decision and action selection: it decides to pay the invoice or to send it for checking.
- Action implementation: it triggers the payment or sends the letter that refuses part of the amount.
Each function sits on a continuum that runs from manual to automatic. One system can automate the four functions to different degrees. The breakdown shows where the error arises and where a person catches it at the lowest cost. A reading error shows up in a sample. An unfounded refusal letter can be corrected before it is sent.
The control modes
No standard names the control modes. They range from approving every case to no human step per case. The person who does all the work remains an option: the system then receives no function.
| Mode | What the person does | Timing | Fits when |
|---|---|---|---|
| Approval before action | Accepts, corrects or rejects each case that the system proposes | Before the effect for all cases | The error is costly or cannot be undone or the reliability has not been measured. The reviewers handle the volume in the time available |
| Referral of flagged cases | Handles the cases that the system flags or places below its confidence threshold | Before the effect for flagged cases | The system flags its doubtful cases. The share of errors flagged at this threshold has been measured |
| Sample review | Reviews cases drawn at random from those that the system handled alone | After the effect | The error can be undone. The volume is high. The reliability is measured |
| No human step per case | Reads the indicators and stops the system when they drift | Continuously | The error costs little and can be undone. Automatic checks detect it |
Modes combine on the same decision. Referral of flagged cases almost always calls for a sample review among the unflagged cases. Without it, nobody sees the errors that the system does not flag. You will find an example of this defect in the AI Canvas technique.
Choosing the mode: the cost of an error and reliability
Parasuraman and his co-authors give two levels of criteria. The primary criterion is the effect of the automation on the person's work. The secondary criteria are the reliability of the automation and the cost of the consequences of a decision or an action.
The cost of an error is what a wrong case costs when it slips past the control: an amount in CHF, a delay, harm to a person. The analyst also asks whether the effect can be undone before it does harm. A document placed in the wrong queue is transferred in two minutes. A refusal letter sent to an insured person cannot be taken back.
Reliability is measured on a sample of real cases. When the threshold must hold for each language, region or document type, it is measured for each one. Reliability that nobody has measured does not count. The decision then stays in approval before action until it is measured. The Measuring the success of an AI solution page shows you how to establish these figures.
The primary criterion checks that the mode leaves the person work that they can do. The calculation multiplies the number of cases per day by the time to read one case, then compares the result with the reviewers' time. A health insurer receives 5'000 documents a day. At 30 seconds per document, approving each routing takes almost 42 hours a day. The three intake clerks have 24 hours of review a day. The calculation rules out this mode before go-live. If no mode leaves feasible work, the decision to automate goes back to risk analysis.
The risk level of the whole system sets a floor. The register chooses the modes function by function above that floor. The Assessing AI risks page shows you how to establish this level. The AI Act: classifying the risk page tells you when the law requires human oversight.
The reviewer and the accountable person
Subcategory GOVERN 3.2 of the NIST AI RMF expects policies that define and differentiate the roles in human-AI configurations. The NIST Playbook separates the people who oversee the system from those who use it. You will find an overview of the framework on the ISO/IEC 42001 and NIST AI RMF page. NIST calls this function oversight. The AI Act uses the same term: human oversight. The register names two roles per row.
The reviewer examines a case or a sample and decides: accept, correct or reject. MAP 3.4 cites physicians as an example. These domain experts work with AI without knowing machine learning. The reviewer judges the content of the case. The reviewer does not judge the model.
The accountable person answers for the result of the decision. This person chooses the mode, sets the thresholds, reads the indicators of the human review and can stop the system. The accountable person also answers for errors in the cases that nobody reviewed. The reviewer answers only for the cases that they have seen. A RACI matrix records these two roles: the reviewer is R and the accountable person is A.
The reviewer holds the right to refuse. The text of GOVERN 1.5 lists appeal and manual override of a system decision among the processes that leave a person to decide. A reasoned rejection counts as normal work. The reviewer need not justify a rejection at greater length than an approval.
What the reviewer receives
MAP 3.5 asks for documentation that lets oversight staff make informed decisions. The review by Goddard, Roudsari and Wyatt (2012) includes 74 studies from several fields and focuses on clinical decision support. It studies automation bias: the tendency to follow a system's proposal without checking it. Among its mitigators it lists training, user accountability, the position of the recommendation on screen, an updated confidence level and information given in place of a recommendation. From a small sample of workshops, Faas and his co-authors (2025) add that reviewers must understand their task and see what the AI bases its output on.
The review file holds the input, the facts that the system extracted with their source and the system's proposal. It shows the system's confidence when that confidence is measured. The reviewer can reject with a reason. The facts come before the proposal: the reviewer reads the invoice and the prescription before reading "Send for checking". You will find the errors that the reviewer looks for on the Hallucinations and Bias pages.
Time is part of the file. Goddard and his co-authors list workload, task complexity and time pressure among the factors that strengthen this bias. The register sets a time per case and the review queue respects it.
The steps
- List the decisions in the process. Start from the process model and mark each point where the system reads, classifies, proposes or acts. Break each decision into four functions. Decision modelling supplies the list when it exists.
- Put figures on the cost of an error and on reliability. The business prices the cost. The data team measures reliability on a sample. A missing value stays marked "to be measured".
- Choose the mode for each function. Apply the secondary criteria, then check the primary criterion by calculating the review time.
- Name the reviewer and the accountable person. Enter them in the register and in the RACI matrix of the process.
- Specify the review file. Write what the reviewer sees, in which order, with what time per case and which rejection reason.
- Train the reviewers. GOVERN 3.2 and MAP 3.5 call for training the people who oversee the system. The training covers system performance, the context of use, known limitations and known negative impacts. The Change management page shows you how to prepare the teams.
- Test the oversight before go-live. MAP 3.5 calls for testing the oversight under conditions close to operation and for evaluating its validity and reliability. For a critical or high-risk system, the Playbook places this evaluation before deployment.
- Monitor the human review in operation. GOVERN 1.5 calls for monitoring risk management continuously, reviewing it at a set frequency and defining the responsibilities for it. The accountable person reads the indicators every month. After each change of model, data or supplier, the accountable person has the test run again.
- Document. The register, the test results and the decisions of the accountable person form the documentation that GOVERN 3.2 and MAP 3.5 ask for.
The order of the steps is a synthesis. Steps 1 to 3 come from Parasuraman and his co-authors. Steps 4 to 7 and 9 come from the NIST Playbook. Step 8 applies GOVERN 1.5. Its indicators and the retest after a model change are our own judgement. Goddard and his co-authors supply part of the review file in step 5. The From prototype to production page shows you where the step 7 test fits.
Testing the oversight on known-error cases
An oversight test measures whether the reviewers catch the errors. The accountable person slips known-error cases into the review queue: an invoice that exceeds the prescription, a misfiled document, a letter that cites the wrong article. The reviewers do not know which cases these are. The share of these cases that the reviewers reject measures the oversight. The accountable person sets the minimum share before the test. The Acceptance and Evaluation Criteria technique shows you how to write this threshold. A small share of these cases stays in the queue in operation.
Monitoring the human review
A human review degrades silently. The indicators of the human review make the decline visible. No source sets an alert threshold. The accountable person sets their own from the first weeks of operation.
| Indicator | Question | Signal |
|---|---|---|
| Share of cases approved without change | Does the review still change a decision? | Close to 100% for weeks |
| Median time per case | Did the reviewer read the file? | Below the time needed to read the input |
| Rejection reasons | Do the rejections say what is wrong? | Reasons empty or always the same |
| Errors found afterwards | Do approved cases contain errors? | Complaints or sample checks that find errors in approved cases |
| Share of known-error cases rejected | Does the review catch a known error? | Share falling from month to month |
What makes the exercise fail
Approval out of habit
Bainbridge (1983) describes the irony of automation. The system exists because it does the work better than the operator. Yet the operator must check that it works. The vigilance studies that she cites show that a person cannot keep attention for more than about half an hour on a source where little happens. Approving every case of a reliable system therefore produces approvals without reading. The remedy cuts the number of approval points, reserves them for costly cases and adds known-error cases.
The human as alibi
Green (2022) examines 41 policies that prescribe human oversight of government algorithms. He identifies two flaws. The evidence suggests that people do not perform the expected oversight functions. The policies make flawed algorithms acceptable and let vendors and agencies shed their responsibility. The study covers the public sector. The mechanism holds wherever a person signs off the results of a system whose use nobody has justified. The register makes an accountable person other than the reviewer answer for the result. The known-error test measures whether the review catches errors.
Loss of skill
Bainbridge notes that manual skills deteriorate when they are no longer used. The operator who takes over in an abnormal situation needs more skill. Yet that operator has lost the practice. A reviewer who no longer handles a whole file can no longer judge the doubtful one. The register provides for a share of cases handled by hand or a rotation between review and manual processing.
The reviewer without the means to judge
A reviewer who sees the proposal without the source, the time or the training approves what they cannot check. MAP 3.5 and the needs that Faas and his co-authors list target this defect. The review file and the time per case correct it. A rejection that demands three screens of justification brings it back.
The misplaced checkpoint
Anthropic describes agents that pause for human feedback at checkpoints or when they hit a blocker. Placing an approval at every step of an agent produces approval out of habit. The Chatbot, assistant, agent page shows you where to place the validation of an agent.
AI considerations
A conversational assistant produces a first list of the decisions from a description of the process. It proposes the breakdown of each decision into four functions and fills in the descriptive columns of the register. It drafts known-error cases from anonymised real cases: an invoice that exceeds the prescription, a letter that cites a wrong provision. It groups the rejection reasons of one month of review and flags those that recur.
The cost of an error remains an input from the business. The AI Canvas technique tells you why. The assistant chooses neither the reviewer nor the accountable person. It does not know the workload of the teams or the time they have. The confidence that a language model expresses about its own answer does not replace reliability measured on a sample.
We count a second model that checks the first among the automatic checks. The register then enters the function under "No human step per case". When a model explains its own proposal, we treat that explanation as a recommendation. Following the mitigator of Goddard and his co-authors, the review file shows the extracted facts and their source rather than this argument. The real cases and the rejection reasons contain data on insured persons. The analyst submits them only to an assistant that the organisation has approved.
Examples
A health insurer in French-speaking Switzerland sorts its incoming documents and checks its invoices with models. The register shows two decisions of the same routing model with two different modes. Rows 3a to 3d repeat the four functions of the grid. The values are invented.
| No. | Decision | Function given to the system | Cost of an error | Measured reliability | Control mode | Time per case | Reviewer | Accountable |
|---|---|---|---|---|---|---|---|---|
| 1 | Route an ordinary document to its queue | Decision and action selection | Low: a two-minute transfer | 94% routed correctly out of 5'000 documents | Sample review: 300 documents a month | 30 seconds | Intake clerk | Head of intake |
| 2 | Recognise an objection | Decision and action selection | High: several days of delay before someone notices it | At the 90% confidence threshold: 98% of French and 96% of German objections recognised | Referral of flagged cases: every document below 90% confidence goes to manual sorting. Review of 50 unflagged documents a week | 1 minute | Intake clerk | Head of legal services |
| 3a | Process a physiotherapy invoice | Information acquisition: read the invoice | Low: the check of the total and the analysis in row 3b stop a wrong amount | 99% of items read correctly | No human step per case: automatic check of the total | Not applicable | None | Head of benefits |
| 3b | Process a physiotherapy invoice | Information analysis: estimate the probability of an error | Medium: a wrong probability puts the invoice on the wrong side of the threshold | Probabilities calibrated on three years of checked invoices | Sample review: 2% of unflagged invoices | 5 minutes | Benefits officer | Head of benefits |
| 3c | Process a physiotherapy invoice | Decision and action selection: pay or send for checking | Medium: CHF 120 per erroneous invoice paid. A check costs CHF 15 | 85% of erroneous invoices flagged at the 12.5% threshold | Referral of flagged cases above 12.5% (15 ÷ 120) | 2 minutes | Benefits officer | Head of benefits |
| 3d | Process a physiotherapy invoice | Action implementation: send the partial refusal letter | High: a letter sent to the insured person cannot be taken back | To be measured | Approval before action: the benefits officer reads and signs each letter | 3 minutes | Benefits officer | Head of benefits |
| 4 | Answer an objection | Action implementation: draft the letter | High: a letter with legal effect sent to the insured person | To be measured | Approval before action: the lawyer reads and signs each letter | 10 minutes | Lawyer | Head of legal services |
The head of legal services also answers for the objections that nobody reviewed. You will find the review of an automated individual decision that the FADP grants to the insured person on the Specifying the requirements of an AI solution page.
Visualisations
The decision register is a table. Each row describes one function of one decision. Each column compares the modes or the accountable persons. The function grid needs a drawing: it shows that one decision occupies four different positions on the axis from manual to automatic. The reviewer's screen is drawn too. The order of its blocks carries a mitigator of automation bias: the facts before the proposal.
Cost
| Phase | Level | Justification |
|---|---|---|
| Preparation | Medium | List the decisions from the process model, obtain the reliability measurements and bring together the business, the data team and the lawyer. |
| Execution | High | One workshop per process for the register. The known-error test needs prepared cases and several days of real review. |
| Documentation | Medium | Maintain the register, read the indicators every month and repeat the test after each model change. |
Tooling
A spreadsheet is enough for the register and for calculating the review time. In a BPMN modelling tool, each approval becomes a user task. A timer boundary event on this task escalates the case that waits beyond the deadline that the accountable person sets. The time per case in the register becomes the task duration when the tool simulates the process.
The review queue is a workflow or ticketing tool. It must record the decision, the rejection reason and the time spent per case. Without this log, the indicators of the human review cannot be calculated.
Agent development kits provide the approval mechanism. In the OpenAI Agents SDK, a tool marked as needing approval pauses the run. The pending call shows the agent, the tool and its arguments. The person approves or rejects the call. A rejection can carry a message. The run state is saved and resumes later: the reviewer can decide hours after the pause.
Sources
No body of knowledge describes this technique under a single name. The procedure is a synthesis of primary sources. The model of Parasuraman, Sheridan and Wickens gives the place and the degree of control. The NIST AI RMF and its Playbook give the roles, the test, the monitoring and the documentation. The names of the control modes, the order of the steps, the known-error test, the monitoring indicators and the retest after a model change are our own judgement.
- Parasuraman, R., Sheridan, T. B. and Wickens, C. D. (2000), A model for types and levels of human interaction with automation, IEEE Transactions on Systems, Man, and Cybernetics, Part A, 30(3), pp. 286-297: the four classes of functions, the continuum from manual to automatic, the primary criterion and the secondary criteria.
- NIST (2023), Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1: the framework whose actions the Playbook details.
- NIST, AI RMF Playbook, GOVERN: GOVERN 3.2 for the roles, the training and the human-AI configurations; GOVERN 1.5 for continuous monitoring, periodic review and appeal against a system decision.
- NIST, AI RMF Playbook, MAP: MAP 3.5 for defining, testing and documenting human oversight; MAP 3.4 for the domain experts who work with AI.
- Goddard, K., Roudsari, A. and Wyatt, J. C. (2012), Automation bias: a systematic review of frequency, effect mediators, and mitigators, JAMIA, 19(1), pp. 121-127: the factors of automation bias and its mitigators; the review covers mostly clinical decision support.
- Bainbridge, L. (1983), Ironies of automation, Automatica, 19(6), pp. 775-779: monitoring a reliable system and the loss of manual skills in the process industries.
- Green, B. (2022), The flaws of policies requiring human oversight of government algorithms, Computer Law & Security Review, 45, 105681: the two flaws of human oversight policies in the public sector.
- Faas, C., Kerstan, S., Uth, R., Langer, M. and Feit, A. M. (2025), Design considerations for human oversight of AI, arXiv:2510.19512, preprint: the needs of reviewers according to four co-design workshops.
- Anthropic, Building effective agents: the human checkpoints of an agent.
- OpenAI, Agents SDK, Human-in-the-loop: pausing a run that awaits approval and resuming it.