AI Canvas
The AI Canvas is a one-page grid that breaks down a decision taken under uncertainty. The top row describes the decision: the machine supplies the prediction, people own the judgment and the action, the outcome measures success. The bottom row describes the data that trains, feeds and corrects the prediction. Ajay Agrawal, Joshua Gans and Avi Goldfarb published it in 2018 in the Harvard Business Review. Louis Dorard's Machine Learning Canvas (2015) complements the AI Canvas: its ten blocks specify the predictive system that will supply the prediction. The deliverable is one filled-in page per decision. Its empty blocks show what the project does not know yet.
Goal
The AI Canvas frames an artificial intelligence project from the decision it must serve. Its authors observe that recent progress in AI lowers the cost of prediction. For them, predicting means producing the information you lack from the information you have. A cheaper prediction changes a decision only if someone knows what they will do with it. The canvas forces the team to write this down: what each error costs, which action follows the prediction, how success will be recognised. Before any development, the canvas sets the human share of the decision (judgment and action) on the same page as the machine's share.
The Machine Learning Canvas takes over once the decision is framed. For Dorard, it frames a machine learning project on one page: what to predict, the decision the prediction feeds, the data needed and what an error costs. It also documents an existing predictive system. Different authors designed the two canvases independently of each other. The AI Canvas decides whether the project starts. The Machine Learning Canvas tells the data team what it is building.
The deliverable is one filled-in canvas per decision. The empty blocks list what remains to be established before a budget is committed.
Usage
When to use
- Use case selected before any budget: fill in the AI Canvas to check that a prediction changes a named decision.
- Workshop with management or the business: use the AI Canvas. It talks about decisions and error costs without data science vocabulary.
- Specification with the data team: move to the Machine Learning Canvas. It sets the task, the data, the features, the timing and the evaluation.
- Predictive system already in production: document it on the Machine Learning Canvas to see which block nobody owns.
- Human decision to automate: fill in the Judgment block first to put a figure on what each type of error costs.
- Vendor offer to assess: fill in the AI Canvas before the demonstration. Then ask the vendor which blocks its solution covers.
When not to use
- Decision based on a known and stable rule: model it in a decision table.
- Use case not chosen yet: one canvas per idea scatters the workshop. List and prioritise the use cases first.
- Choice between building and buying: the canvas does not settle it. A make-or-buy analysis compares the options.
Description
The AI Canvas
Agrawal, Gans and Goldfarb designed the canvas while teaching MBA students at the Rotman School of Management in Toronto. They fill it in for every decision whose outcome depends on an uncertainty. The four top blocks describe the decision. The three bottom blocks describe the data.
The Prediction block names the information that is missing at the moment of decision: is this transaction fraudulent, how many parcels will reach the sorting centre tomorrow. Write it as a question that a model answers with a value or a probability.
The Judgment block puts a figure on what each error costs. A false alarm and a missed alert rarely cost the same. The authors reserve this block for human judgment. It also weighs factors that are hard to put a figure on. Privacy is one of them. Judgment can change the model itself: if a missed alert costs more than a false alarm, the team accepts more false alarms to miss fewer alerts.
The Action block names what happens depending on the prediction. The action can be binary: block the transaction or let it through. It can also offer a wider choice: block, ask the customer to confirm or let it through.
The Outcome block says how you will know, after the fact, that the action was the right one. The authors see it as the measure of success. As long as the desired outcome is unknown, they judge any improvement difficult or impossible.
The Input block lists what the system knows at the moment of decision: the amount, the merchant, the time, the card's country.
The Training block describes the history that links past situations to their known outcome. The model learns from this history. If this history is missing, the authors suggest deploying a mediocre model and waiting for it to improve. That choice then rests on the Feedback block.
The Feedback block describes what the system collects once in service. This data trains the model continuously. It can stay specific to one site or be aggregated across several. The observed outcome of each action feeds it.
The Machine Learning Canvas
Louis Dorard published the Machine Learning Canvas in 2015 to frame a machine learning project before development. Version 1.2 has ten blocks.
The canvas is centred on the value proposition. The left half groups the prediction blocks: how the system produces and uses predictions. The right half groups the blocks on learning from data. The bottom block covers the system in service. Dorard places the blocks by proximity. At the top of each half, the Decisions and Collecting Data blocks deal with how the system fits into day-to-day business operations. At the bottom of each half, the Making Predictions and Building Models blocks deal with the prediction engine and its limits on latency and throughput.
The Value Proposition block says what you are trying to do and why. It also names who will use the system or bear its effects. Dorard advises naming the end user first, then the value delivered to them.
The ML Task block sets the type of problem (classification, regression…), the input and the output to predict with its possible values. Dorard asks for the precise question the model decides: "Is this email important?" He adds a baseline: what a person or the old method does today.
The Decisions block says when and how predictions become decisions that deliver the promised value. For Dorard, a prediction is only information: it produces nothing until someone acts on it. He advises starting without full automation. A person reviews the system's recommendations before acting.
The Making Predictions block sets when the system predicts on a new input and how long it takes. It specifies the frequency, the volume and the other operating constraints.
The Offline Evaluation block chooses the methods and metrics that evaluate the predictions and their use before deployment: the test cases, the size of the test set, the cost of a false positive and that of a false negative. A false positive flags a case wrongly. A false negative lets through a case that should have been flagged. In the physiotherapy invoice example, a false negative lets an erroneous invoice be paid and costs CHF 120 on average. A false positive sends a correct invoice to review and costs CHF 15. These costs come from the AI Canvas Judgment block and weight the offline metric.
The Data Sources block lists the raw data available.
The Collecting Data block says how new examples to learn from are obtained: their inputs and their outputs. Dorard asks you to think first about obtaining the outputs. The outputs are the scarce data: the known outcome of each past case. In the invoice example, the output is the result of the review.
The Features block describes how raw data becomes the model's input variables. Dorard uses the block to fix the date on which each case is observed (for example the customer's state on the 1st of the month) and the aggregation windows ("purchases in the last thirty days").
The Building Models block sets when a model is created or updated with new training data and how long it takes.
The Live Evaluation and Monitoring block specifies the methods and metrics that evaluate the system after deployment and put a figure on the value created.
How the two canvases map onto each other
The authors do not link the two canvases.
| AI Canvas | Machine Learning Canvas | What the second specifies |
|---|---|---|
| Prediction | ML Task | The task type, the output and its values, the human baseline |
| Judgment | Decisions, Offline Evaluation | Error costs become metrics measured before deployment |
| Action | Decisions | When and how the prediction triggers the action |
| Outcome | Live Evaluation and Monitoring | The metrics of the value created after deployment |
| Input | Data Sources, Features, Making Predictions | The transformation into features and the prediction latency |
| Training | Data Sources, Collecting Data, Building Models | How often the model is updated |
| Feedback | Collecting Data, Live Evaluation and Monitoring | The continuous collection of outputs |
| No block | Value Proposition | The end user and the value they receive |
Filling in a canvas
- Choose a decision
A canvas carries one decision whose outcome depends on an uncertainty. "Improve customer service" covers dozens of decisions. "Escalate this ticket to second level or not" is one. - Write the prediction
Phrase it as a closed question or a value to estimate. If the answer changes no action, go back to step 1. - Put a figure on the judgment
Name each error and its cost in francs, in delay or in risk. The business owner supplies these figures. The data team cannot guess them. - Describe the action and the outcome
Name who acts on the prediction. Write down how you will see after the fact that the action was the right one. - Fill in the data blocks
For each data item, write the source, its owner and the date it was last checked. "We must have it somewhere" leaves the block empty. - Read the empty blocks
Each empty or uncertain block becomes a question with an owner and a deadline. The sponsor then decides to go ahead, reframe or drop the project. - Move to the Machine Learning Canvas
Once the decision holds, the data team takes the case onto the second canvas. Dorard starts with the end user, then the value proposition.
For the AI Canvas, a one- to two-hour session brings together the owner of the decision, a person who takes it every day and a member of the data team. Without the owner, the Judgment block stays empty.
Reading the empty blocks
- Empty Judgment: nobody knows how to set the model's threshold. The data team will choose it on a technical metric. The business will discover the cost of errors in production.
- Empty Outcome: the project will not be able to show that it succeeded. To fill the block, the team names at least one indicator and the date of its first measurement.
- Empty Training: the authors suggest launching a mediocre model and letting it learn. This option assumes a filled-in Feedback block.
- Empty Feedback: once in service, the model no longer sees its errors. Nobody notices that it degrades when situations change.
- Input marked "to check": the project rests on data that nobody has seen. Before the next session, a participant pulls a data extract and looks at it.
What makes the exercise fail
Starting from the model
The team arrives with a model or a dataset and looks for the decision it would serve. The Prediction block fills up first. The Judgment and Action blocks stay vague. Starting from the decision restores the order: the prediction follows from what the decision does not know.
A perfect model that earns nothing
Dorard shows that an error-free churn model can earn nothing. A customer brings in CHF 10. A retention campaign keeps one in five. Each contact costs CHF 2. Contacting a customer about to leave brings in CHF 2 on average and costs CHF 2. The gain is zero. The Judgment block exposes this calculation before development.
Feedback distorted by the action
Once in service, the model influences the data it learns from. If only flagged customers receive an offer, nobody knows any more which ones would have left without it. Dorard recommends keeping a sample of cases outside the system's action. These control cases show what happens without it.
One canvas for a whole programme
An AI programme touches dozens of decisions. A single canvas mixes their predictions and their error costs. No block can be read any more. The authors fill in one canvas per critical decision.
Blocks filled in optimistically
"CRM data" in the Training block states neither the period nor the owner nor whether the outcome is in it. A block that names only systems looks complete in the session. The first data extract shows what is missing. Each line of a data block carries a checked source or the note "to check".
AI considerations
A conversational assistant produces a first version of the canvas from a description of the use case. It suggests possible errors for the Judgment block, candidate data sources and questions for each empty block. It also points out inconsistencies between blocks: an action that depends on no prediction, an outcome that nothing measures. The workshop participants correct this version block by block.
The assistant does not fill in the Judgment block. The cost of an error is a business choice. A plausible figure suggested by a model later passes for data. The assistant also does not know which data the organisation holds. A source it names because it is plausible looks like a checked source. The description of a use case often contains volumes, costs and the names of internal systems. The analyst submits it only to an assistant that their organisation has approved.
Examples
A Swiss health insurer wants a model to sort physiotherapy invoices before payment. The filled-in canvas shows what its empty blocks signal.
The threshold in the Action block comes from the Judgment block. Reviewing an invoice costs CHF 15. Paying an erroneous invoice costs CHF 120 on average. A review detects the error and prevents that payment. The review pays off as soon as the probability of an error exceeds 12.5% (15 divided by 120).
The Feedback block is empty. No claims handler will see the invoices paid without review any more. The model will therefore never see the errors it lets through. The Training block has the same flaw: the history contains only the reviewed invoices, which a claims handler had picked. A model trained on them learns the claims handlers' sorting. The insurer adds a random review of 2% of the automatically paid invoices to the Action block. Each random review costs CHF 15. This amount is the price of the feedback data. These reviews feed the Feedback block. The history thus also receives invoices that no claims handler picked.
Visualisations
A colour marks each empty block and makes it visible at a glance. The mapping table guides the move from one canvas to the other.
Cost
| Phase | Level | Justification |
|---|---|---|
| Preparation | Low | Choose the decision, bring together three to five people, gather what is known about the data. |
| Execution | Medium | One session of one to two hours for the AI Canvas. The Machine Learning Canvas often takes several sessions with the data team. |
| Documentation | Low | One page per decision. The team updates it when an empty block is filled. |
Tooling
An A3 sheet or a whiteboard is enough for the AI Canvas. A team spread across several sites uses a shared document or an online whiteboard.
Dorard publishes the ten-block Machine Learning Canvas on ownml.co as PDF, Word, ODT and Google Doc. The download requires signing up to his newsletter.
Sources
Both canvases are authored methods. No professional body of knowledge describes them.
- Agrawal, A., Gans, J. and Goldfarb, A. (2018), A Simple Tool to Start Making Decisions with the Help of AI, Harvard Business Review, 17 April 2018: the AI Canvas and its authors' definition of its blocks.
- Dorard, L., Machine Learning Canvas, OWNML: the author's presentation of the canvas, version 1.2 and the downloadable templates.
- Dorard, L. (2016), From Data to AI with the Machine Learning Canvas (Part I), Medium: the ten blocks, their questions and their layout.
- Dorard, L., From Data to AI with the Machine Learning Canvas (Part II), LinkedIn: data collection and features.
- Dorard, L., From Data to AI with the Machine Learning Canvas (Part III), LinkedIn: the value proposition, the ML task, decisions reviewed by a person and the control sample.
- Dorard, L., From Data to AI with the Machine Learning Canvas (Part IV), LinkedIn: making predictions, offline evaluation and the churn model that earns nothing.