Smoke test and concierge MVP
The smoke test and the concierge MVP are two forms of minimum viable product that test an offer before the team builds it. The smoke test presents the offer through an ad and a page that shows its price. It counts the people who sign up for it or pre-order it. The concierge MVP delivers the future product's service by hand to a few paying customers. The first measures interest in the offer. The second shows what the service demands and whether customers keep paying. Eric Ries names the concierge MVP in The Lean Startup and counts the smoke test among the forms of MVP.
Goal
The smoke test and the concierge MVP test a hypothesis about a product through what real customers do, before the team builds it. The smoke test asks whether the target customers act on the offer: they click, leave their address or pre-order. Ries points out that it measures only their interest in trying the product. The concierge MVP asks whether the team can deliver the service and whether customers pay for it at each delivery. Ries treats the concierge as a learning activity that tests the hypotheses of the growth model.
Both techniques replace what people say with what they do. A team that builds on what customers say in interviews discovers their behaviour once the product is launched. The smoke test supports the decision to build, to change the offer or to give it up. The concierge MVP supports the decision to keep or change the growth model. Its log also points to the moment to automate and to the first task to automate.
The smoke test delivers the quantified hypothesis, the counts at each step and the verdict. The concierge MVP delivers a log kept per customer and per delivery. The team reads it against the figures written before the first customer. To choose between these tests and the exploration interview, see the lean validation board. For the definition of the MVP, see minimum viable product.
Usage
When to use it
- An offer that fits in one ad and one page: measure interest with a smoke test before building.
- A product that is expensive to build: a page and a fixed advertising budget test the idea before the investment.
- A new feature in a live product: a link to the announced feature counts the users who ask for it.
- A service whose content is still to be discovered: the concierge MVP serves customers by hand to learn what they expect.
- A recurring-payment hypothesis: the concierge collects payment at each delivery and shows who keeps paying.
- An internal programme to launch: serve about ten colleagues by hand and count those who come back.
- A decision to build driven by an executive's opinion: a rate measured against a threshold written in advance settles it instead.
When not to use it
- A customer problem that is still an assumption: a refusal does not show whether the problem or the offer is at fault. Run exploration interviews first.
- A smoke test on a few buyers known by name: too few visitors for a rate. Present the offer to them in a meeting instead. The pitch on the lean validation board covers this variant as well as the offer page.
Description
Two tests on the same pattern
The team writes down in figures what it expects. It presents the offer to the customer by the cheapest means. It counts what the customer does, then compares the count with the figure written down. The smoke test customer sees a promise. The concierge customer receives the service.
| Smoke test | Concierge MVP | |
|---|---|---|
| Question | Do the target customers act on the offer? | Can the team deliver the service? Do customers pay for it at each delivery? |
| What the customer receives | An ad and a page that describe the offer and its price. | The service that the team delivers by hand. |
| What the team counts | The click rate on impressions and the sign-up rate on visits. | Payments, referrals and time spent per customer. |
| Stopping rule | The budget or the duration set in advance runs out. | The team no longer has time to take on one more customer. |
| Limit of the result | A threshold met proves initial interest and nothing about use. | A profitable concierge can invalidate the growth model. |
Running a smoke test
Ries describes the smoke test as an old direct-marketing technique: the customer can pre-order a product that has not been built yet. The steps come from Ries's 2009 interview with Babak Nivi and from Alberto Savoia.
1. Write the hypothesis in figures
Before building the page, the team writes the hypothesis in the form Savoia calls XYZ: "At least X% of Y will Z." Y describes the target market. Z describes the expected action. X sets the share of the market that must act for the idea to be viable. For Savoia, these values are starting estimates. The team derives them from the minimum market size the idea needs. The figure comes from the team's business plan.
When the team puts figures on the hypothesis, its disagreements come to light. In Savoia's example, two students on the same team discover that they do not have the same price in mind.
| Vague wording | XYZ wording |
|---|---|
| Restaurant owners | At least 8% of independent cafés and restaurants in French-speaking Switzerland (X and Y) |
| would be interested | sign up for a trial (Z) |
| in a scheduling tool | of the service that draws up the staff schedule every week |
| at a reasonable price | at CHF 15 a week |
2. Build the entry point
The entry point presents the offer, its price and a single action. On the web, it is an ad linked to a landing page. In 2009, Ries describes this set-up with a purchase offer on the page. In a live product, the entry point is a link or a button to the announced feature. Savoia calls any such entry point a Fake Door: an ad linked to a page, a link or a button. He credits the name to Jess Lee (Polyvore). The team pretends that the product exists and watches who clicks.
The chosen action sets what the test proves. A click costs the visitor one second. A visitor who leaves an e-mail address commits more. One who pays for a pre-order commits more still. The more the action costs the visitor, the closer the count comes to a purchase decision.
3. Tell the visitor
The page that follows the action tells the visitor that the product does not exist yet. If the page has taken payment for a pre-order, the next page also announces the refund. Savoia asks the team to tell the person at once and to thank them with a gift close to what they thought they were buying. He notes that the technique has become too common and is often misused. He wants a test that benefits the team, the customer and the market.
4. Bring in the traffic
The team buys ads targeted at market Y. It sets the budget and the duration in advance. In Savoia's examples, the ad costs a few dozen dollars and runs for a month or until the budget is spent. The visitors to the page are a sample of market Y. The rate measured on them estimates the share X. The targeting sets the denominator. An ad shown outside the target market lowers the rate without saying anything about that market.
5. Count at each step
Savoia measures the Initial Level of Interest: the number of actions divided by the number of opportunities to act. He divides clicks by ad impressions, then actions by page visits. Ries takes the click-through rate as the first signal: if nobody clicks, the content of the next page does not matter.
The number of visits depends on the budget. Only the rate is compared with the X of the hypothesis. In a live product, Ries shows the announced feature to a small share of customers and compares their cohort with the rest. He asks the team to measure the effect on the cohort's overall behaviour. He cites a test in which few customers clicked a link to an exclusive club, while the cohort that saw the link spent more. This set-up is an A/B test read through cohort analysis.
6. Decide
The team compares the rate with the X written before the test. Below the threshold, it drops the offer or changes its price, its promise or its segment. It then tests the new hypothesis. Above the threshold, it holds evidence of interest. Ries points out that a smoke test is not enough to validate a whole growth model. Savoia adds that a Fake Door measures only a first visit: nobody knocks twice. Lasting interest needs a heavier test. The concierge MVP is one.
Running a concierge MVP
Manuel Rosso ran Food on the Table. He presented the method from the team's point of view: finding the first customers, moving contact from meetings to the phone and then to e-mail, deciding when to write code.
1. Write down what the concierge must teach
Before serving the first customer, the team writes down in figures what it expects of the growth model. Ries asks these questions about an internal volunteering programme at HP: how many of the first participants complete their assignment? If ten do, how many will come back? How many will recruit a colleague?
2. Find the early adopters
Ries looks for early adopters rather than the average customer: the customers who feel the need most acutely. They forgive defects more readily and give their opinion willingly. Rosso found his first twenty users among the people he met during the discovery phase. Well-connected people recommended other candidates to him. The team's personal network completed the list. Rosso accepted a range of profiles.
3. Sell at the end of the interview
The team runs the interview as market research, then attempts a sale. It describes the benefits, states the price and offers to sign the person up. Most people refuse: according to Ries, they are not early adopters and do not sign up for a service they have not seen. The refusal is part of the test. The person who accepts becomes the first customer and pays.
4. Serve by hand and collect payment at each delivery
For each customer, the team does by hand what the software will do later. It prepares the service, hands it over, asks for feedback and applies it to the next delivery. Ries treats payment at each delivery as the most important point. A free service says nothing about what the customer would pay.
Rosso reduces the team's direct involvement from one delivery to the next. The first exchange takes place in person: the team learns how the customer solves the problem alone and describes the solution to them as it would appear on screen. The second takes place by phone and the result goes out by e-mail. After that, the customer does as much as possible alone. At Food on the Table, Google Apps documents stood in for web pages.
5. Keep a log per customer
Ries asks the team to measure what customers do: they pay, come back or recommend the service. The team records each delivery in a log: the customer, the date, the time spent per task, the payment, the feedback received and the correction made.
6. Add customers one at a time
Ries reports that the Food on the Table team added customers one at a time within the same scope: the same supermarket. Each customer served made the next one easier to take on. The team got to know the supermarket's products and its customers.
7. Automate when time runs short
Ries says when to automate: the team invests in software when it is too busy to take on one more customer. It then automates one task at a time. Each version gives it back some time and lets it serve a few more customers. Ries does not say which task goes first. The log settles it: among the tasks that repeat at each delivery, the one that costs the most hours goes first. The exchanges in which the team learns from the customer stay manual.
Rosso ties automation to learning. The team automates when repeated tasks slow learning down. It writes code only if the code makes learning cheaper. The team moves on to the next learning stage when it can predict what the early adopters will say.
8. Read the result against the growth model
Ries notes that a concierge often invalidates the proposed growth model, even when it is profitable. Without a written growth model, many companies settle for a small profitable business when a pivot would lead to stronger growth. The team therefore reads the log against the figures written at step 1.
Concierge and Wizard of Oz
Ries also describes the Wizard of Oz test. The customer believes they are using the real product. Behind the scenes, people do the work. Ries considers this test as labour-intensive as the concierge. It tells the team whether customers will use the product once its technical problems are solved. In the concierge that Ries describes, the customer knows that a person is serving them. Savoia calls the Wizard of Oz the Mechanical Turk.
AI considerations
To write the hypothesis, a language model reviews the XYZ wording and flags the vague words that remain: "interested", "reasonable", "some customers". It proposes several values of X and asks the team where each one comes from. The team keeps the figure that the business plan justifies.
For the smoke test, it drafts variants of the ad and the page to compare in an A/B test. It writes the spreadsheet formulas that compute the rate at each step from the advertising platform's exports. The analyst checks one rate by hand. The answers that a model generates for fictional customers remain guesses. The test collects real actions. The e-mail addresses collected identify people. The analyst removes them from the exports before passing the exports to an AI service hosted by a third party.
For the concierge, it groups the feedback recorded in the log by theme and spots the tasks that recur at each delivery. These tasks are the candidates for automation. A task handed to the model is already an automation. The team records it as such in the log. Without that entry, the log no longer points to the task to automate. The sales interview and the delivery of the service stay with the team: it learns from the customer during these exchanges.
Examples
A Lausanne team is preparing a service that draws up the weekly staff schedule for independent cafés and restaurants. It runs a smoke test, then a concierge MVP. The figures are an illustration: neither Ries nor Savoia gives a threshold.
| Hypothesis written before the test | At least 8% of independent cafés and restaurants in French-speaking Switzerland sign up for a trial at CHF 15 a week. |
|---|---|
| Threshold | The service covers its costs at 480 customers. The team estimates the market at 6'000 independent cafés and restaurants: 480 / 6'000 = 8%. |
| Ad | Targeted at managers of independent cafés and restaurants in French-speaking Switzerland. Budget of CHF 300 over three weeks. |
| Ad impressions | 18'400 |
| Clicks to the page | 410 of 18'400 impressions (2.2%) |
| Trial sign-ups (business name and e-mail address) | 41 of 410 visits (10%) |
| Page after sign-up | "This service does not exist yet. The first schedules will be prepared by hand from April. As a thank-you, here is a weekly schedule template to print." |
| Verdict | 10% against a threshold of 8%: interest established. Lasting interest remains to be tested. |
The team moves on to the concierge. Before the first customer, it writes down what it expects: of the first six businesses, at least five are still paying in week 6 and one recommends the service. It first serves two of the businesses that signed up. It devotes at most nine hours a week to the service.
| Week | Paying customers | Payments received | Service hours | Hours asking staff for their absences | Fact recorded |
|---|---|---|---|---|---|
| 1 | 2 | CHF 30 | 3 | 1.5 | "The PDF does not print on A4": layout corrected |
| 2 | 3 | CHF 45 | 4.5 | 2.25 | Add the Vaud school holidays |
| 3 | 4 | CHF 60 | 6 | 3 | Receive the schedules on Thursday |
| 4 | 5 | CHF 75 | 7.5 | 3.75 | A customer recommends the service to a fellow owner |
| 5 | 6 | CHF 90 | 9 | 4.5 | Nine-hour ceiling reached |
| 6 | 6 | CHF 90 | 9 | 4.5 | Seventh business turned away for lack of time |
In week 6, all six businesses are still paying and one of them has recommended the service. The figures written before the first customer are met: the team keeps its growth model. The same week, it turns away a seventh business. Ries places the move to automation at this point. Asking staff for their absences by phone takes half the hours. A form that the staff fill in replaces these calls first.
Visualisations
The diagram in the Description section aligns the two techniques on the same steps and marks the concierge's stopping rule in orange. The smoke test record and the concierge log show the deliverables of the example.
Cost
| Phase | Smoke test | Concierge MVP |
|---|---|---|
| Preparation | Low: write the hypothesis, set up the page and the ad in a few hours. | Medium: interviews until the first customer agrees to pay. |
| Execution | Low: a fixed advertising budget and a few weeks of running ads. | High: the team spends time on each customer and each delivery. |
| Documentation | Low: the hypothesis, the counts and the verdict fit on one page. | Medium: the team writes one log line per delivery. |
Tooling
For the smoke test, a page builder (Webflow, Carrd) carries the offer and a form (Typeform, Google Forms) collects the action. Advertising platforms (Google Ads, LinkedIn Ads, Meta) target the intended market and count impressions and clicks. A web analytics tool (Matomo, Google Analytics) counts visits and actions on the page. In a live product, a feature-flag system shows the announced link to a chosen cohort.
The concierge makes do with office tools. A spreadsheet keeps the log. The phone, e-mail and a messaging app carry the exchanges with each customer. Shared documents replace the screens of the future product. A QR-bill or TWINT collects payment for each delivery.
Sources
The procedure for the smoke test comes from an interview with Ries and from Alberto Savoia. The procedure for the concierge comes from Ries and from Manuel Rosso, who ran one.
- Eric Ries, The Lean Startup, Crown Business, 2011: chapter 4 "Experiment" for early adopters, the HP programme and the name of the concierge MVP; chapter 6 "Test" for the forms of MVP, how to run the concierge and the Wizard of Oz test; chapter 7 "Measure" for the smoke test.
- Eric Ries, What is the minimum viable product?, interview with Babak Nivi, Venture Hacks, 23 March 2009: the page that promises the product, the ads that bring traffic to it, the click-through rate and A/B tests read by cohort.
- Alberto Savoia, Pretotype It, tenth-anniversary edition, 2022: the Fake Door and how to run it ethically; the Initial Level of Interest; the Mechanical Turk; the XYZ hypothesis, in the bonus chapter taken from The Right It.
- Manuel Rosso, Concierge MVP: Learning from early adopters at Food on the Table, presentation, SlideShare: recruiting the first users, moving contact from meetings to the phone and then to e-mail, when to write code and the signal to move on to the next stage.

