Your Training Partner
Techniques Toolbox
The smoke test and the concierge MVP on the same steps: quantify the hypothesis, present the offer, count, decide. The smoke test writes an XYZ hypothesis, shows an ad and an offer page with its price, divides sign-ups by visits and compares the rate with the threshold. A missed threshold leads back to a new hypothesis. A threshold met calls for a heavier test. The concierge is one. The concierge MVP writes down the expected payments and referrals, sells to each customer, serves them by hand and keeps a log of payments and hours. Figures that fall short lead back to the growth model. When the team is too busy to take on one more customer, it automates one task and takes on the next customer.

Smoke test and concierge MVP

The smoke test and the concierge MVP are two forms of minimum viable product that test an offer before the team builds it. The smoke test presents the offer through an ad and a page that shows its price. It counts the people who sign up for it or pre-order it. The concierge MVP delivers the future product's service by hand to a few paying customers. The first measures interest in the offer. The second shows what the service demands and whether customers keep paying. Eric Ries names the concierge MVP in The Lean Startup and counts the smoke test among the forms of MVP.

Goal

The smoke test and the concierge MVP test a hypothesis about a product through what real customers do, before the team builds it. The smoke test asks whether the target customers act on the offer: they click, leave their address or pre-order. Ries points out that it measures only their interest in trying the product. The concierge MVP asks whether the team can deliver the service and whether customers pay for it at each delivery. Ries treats the concierge as a learning activity that tests the hypotheses of the growth model.

Both techniques replace what people say with what they do. A team that builds on what customers say in interviews discovers their behaviour once the product is launched. The smoke test supports the decision to build, to change the offer or to give it up. The concierge MVP supports the decision to keep or change the growth model. Its log also points to the moment to automate and to the first task to automate.

The smoke test delivers the quantified hypothesis, the counts at each step and the verdict. The concierge MVP delivers a log kept per customer and per delivery. The team reads it against the figures written before the first customer. To choose between these tests and the exploration interview, see the lean validation board. For the definition of the MVP, see minimum viable product.

Usage

When to use it

  • An offer that fits in one ad and one page: measure interest with a smoke test before building.
  • A product that is expensive to build: a page and a fixed advertising budget test the idea before the investment.
  • A new feature in a live product: a link to the announced feature counts the users who ask for it.
  • A service whose content is still to be discovered: the concierge MVP serves customers by hand to learn what they expect.
  • A recurring-payment hypothesis: the concierge collects payment at each delivery and shows who keeps paying.
  • An internal programme to launch: serve about ten colleagues by hand and count those who come back.
  • A decision to build driven by an executive's opinion: a rate measured against a threshold written in advance settles it instead.

When not to use it

  • A customer problem that is still an assumption: a refusal does not show whether the problem or the offer is at fault. Run exploration interviews first.
  • A smoke test on a few buyers known by name: too few visitors for a rate. Present the offer to them in a meeting instead. The pitch on the lean validation board covers this variant as well as the offer page.

Description

Two tests on the same pattern

The team writes down in figures what it expects. It presents the offer to the customer by the cheapest means. It counts what the customer does, then compares the count with the figure written down. The smoke test customer sees a promise. The concierge customer receives the service.

Quantify the hypothesis
Present the offer
Count
Decide
Smoke test
Quantify the hypothesisAt least X% of Y will Z
Present the offerAd and offer page with price
CountSign-ups divided by visits
DecideThreshold missed: drop or change the offer
DecideThreshold met: interest established
New hypothesis
Heavier test
Concierge MVP
Quantify the hypothesisExpected payments and referrals
Present the offerSale, then service delivered by hand
CountLog of payments and hours
DecideFigures not met: change the growth model
DecideTeam too busy for one more customer: automate one task
New model
Next customer
The smoke test and the concierge MVP follow the same steps. The smoke test compares the sign-up rate with the threshold written in advance. A missed threshold leads back to a new hypothesis. A threshold met calls for a heavier test. The concierge is one. The concierge serves each customer by hand and keeps a log. Figures that fall short lead back to the growth model. When the team cannot take on one more customer, it automates one task and takes on the next customer.
Smoke testConcierge MVP
QuestionDo the target customers act on the offer?Can the team deliver the service? Do customers pay for it at each delivery?
What the customer receivesAn ad and a page that describe the offer and its price.The service that the team delivers by hand.
What the team countsThe click rate on impressions and the sign-up rate on visits.Payments, referrals and time spent per customer.
Stopping ruleThe budget or the duration set in advance runs out.The team no longer has time to take on one more customer.
Limit of the resultA threshold met proves initial interest and nothing about use.A profitable concierge can invalidate the growth model.
The smoke test and the concierge MVP compared: the question, what the customer receives, what the team counts, the stopping rule and the limit of the result.

Running a smoke test

Ries describes the smoke test as an old direct-marketing technique: the customer can pre-order a product that has not been built yet. The steps come from Ries's 2009 interview with Babak Nivi and from Alberto Savoia.

1. Write the hypothesis in figures

Before building the page, the team writes the hypothesis in the form Savoia calls XYZ: "At least X% of Y will Z." Y describes the target market. Z describes the expected action. X sets the share of the market that must act for the idea to be viable. For Savoia, these values are starting estimates. The team derives them from the minimum market size the idea needs. The figure comes from the team's business plan.

When the team puts figures on the hypothesis, its disagreements come to light. In Savoia's example, two students on the same team discover that they do not have the same price in mind.

Vague wordingXYZ wording
Restaurant ownersAt least 8% of independent cafés and restaurants in French-speaking Switzerland (X and Y)
would be interestedsign up for a trial (Z)
in a scheduling toolof the service that draws up the staff schedule every week
at a reasonable priceat CHF 15 a week
A vague hypothesis rewritten in the XYZ format. Each vague word receives a value that a count can disprove. Invented example.

2. Build the entry point

The entry point presents the offer, its price and a single action. On the web, it is an ad linked to a landing page. In 2009, Ries describes this set-up with a purchase offer on the page. In a live product, the entry point is a link or a button to the announced feature. Savoia calls any such entry point a Fake Door: an ad linked to a page, a link or a button. He credits the name to Jess Lee (Polyvore). The team pretends that the product exists and watches who clicks.

The chosen action sets what the test proves. A click costs the visitor one second. A visitor who leaves an e-mail address commits more. One who pays for a pre-order commits more still. The more the action costs the visitor, the closer the count comes to a purchase decision.

3. Tell the visitor

The page that follows the action tells the visitor that the product does not exist yet. If the page has taken payment for a pre-order, the next page also announces the refund. Savoia asks the team to tell the person at once and to thank them with a gift close to what they thought they were buying. He notes that the technique has become too common and is often misused. He wants a test that benefits the team, the customer and the market.

4. Bring in the traffic

The team buys ads targeted at market Y. It sets the budget and the duration in advance. In Savoia's examples, the ad costs a few dozen dollars and runs for a month or until the budget is spent. The visitors to the page are a sample of market Y. The rate measured on them estimates the share X. The targeting sets the denominator. An ad shown outside the target market lowers the rate without saying anything about that market.

5. Count at each step

Savoia measures the Initial Level of Interest: the number of actions divided by the number of opportunities to act. He divides clicks by ad impressions, then actions by page visits. Ries takes the click-through rate as the first signal: if nobody clicks, the content of the next page does not matter.

The number of visits depends on the budget. Only the rate is compared with the X of the hypothesis. In a live product, Ries shows the announced feature to a small share of customers and compares their cohort with the rest. He asks the team to measure the effect on the cohort's overall behaviour. He cites a test in which few customers clicked a link to an exclusive club, while the cohort that saw the link spent more. This set-up is an A/B test read through cohort analysis.

6. Decide

The team compares the rate with the X written before the test. Below the threshold, it drops the offer or changes its price, its promise or its segment. It then tests the new hypothesis. Above the threshold, it holds evidence of interest. Ries points out that a smoke test is not enough to validate a whole growth model. Savoia adds that a Fake Door measures only a first visit: nobody knocks twice. Lasting interest needs a heavier test. The concierge MVP is one.

Running a concierge MVP

Manuel Rosso ran Food on the Table. He presented the method from the team's point of view: finding the first customers, moving contact from meetings to the phone and then to e-mail, deciding when to write code.

1. Write down what the concierge must teach

Before serving the first customer, the team writes down in figures what it expects of the growth model. Ries asks these questions about an internal volunteering programme at HP: how many of the first participants complete their assignment? If ten do, how many will come back? How many will recruit a colleague?

2. Find the early adopters

Ries looks for early adopters rather than the average customer: the customers who feel the need most acutely. They forgive defects more readily and give their opinion willingly. Rosso found his first twenty users among the people he met during the discovery phase. Well-connected people recommended other candidates to him. The team's personal network completed the list. Rosso accepted a range of profiles.

3. Sell at the end of the interview

The team runs the interview as market research, then attempts a sale. It describes the benefits, states the price and offers to sign the person up. Most people refuse: according to Ries, they are not early adopters and do not sign up for a service they have not seen. The refusal is part of the test. The person who accepts becomes the first customer and pays.

4. Serve by hand and collect payment at each delivery

For each customer, the team does by hand what the software will do later. It prepares the service, hands it over, asks for feedback and applies it to the next delivery. Ries treats payment at each delivery as the most important point. A free service says nothing about what the customer would pay.

Rosso reduces the team's direct involvement from one delivery to the next. The first exchange takes place in person: the team learns how the customer solves the problem alone and describes the solution to them as it would appear on screen. The second takes place by phone and the result goes out by e-mail. After that, the customer does as much as possible alone. At Food on the Table, Google Apps documents stood in for web pages.

5. Keep a log per customer

Ries asks the team to measure what customers do: they pay, come back or recommend the service. The team records each delivery in a log: the customer, the date, the time spent per task, the payment, the feedback received and the correction made.

6. Add customers one at a time

Ries reports that the Food on the Table team added customers one at a time within the same scope: the same supermarket. Each customer served made the next one easier to take on. The team got to know the supermarket's products and its customers.

7. Automate when time runs short

Ries says when to automate: the team invests in software when it is too busy to take on one more customer. It then automates one task at a time. Each version gives it back some time and lets it serve a few more customers. Ries does not say which task goes first. The log settles it: among the tasks that repeat at each delivery, the one that costs the most hours goes first. The exchanges in which the team learns from the customer stay manual.

Rosso ties automation to learning. The team automates when repeated tasks slow learning down. It writes code only if the code makes learning cheaper. The team moves on to the next learning stage when it can predict what the early adopters will say.

8. Read the result against the growth model

Ries notes that a concierge often invalidates the proposed growth model, even when it is profitable. Without a written growth model, many companies settle for a small profitable business when a pivot would lead to stronger growth. The team therefore reads the log against the figures written at step 1.

Concierge and Wizard of Oz

Ries also describes the Wizard of Oz test. The customer believes they are using the real product. Behind the scenes, people do the work. Ries considers this test as labour-intensive as the concierge. It tells the team whether customers will use the product once its technical problems are solved. In the concierge that Ries describes, the customer knows that a person is serving them. Savoia calls the Wizard of Oz the Mechanical Turk.

AI considerations

To write the hypothesis, a language model reviews the XYZ wording and flags the vague words that remain: "interested", "reasonable", "some customers". It proposes several values of X and asks the team where each one comes from. The team keeps the figure that the business plan justifies.

For the smoke test, it drafts variants of the ad and the page to compare in an A/B test. It writes the spreadsheet formulas that compute the rate at each step from the advertising platform's exports. The analyst checks one rate by hand. The answers that a model generates for fictional customers remain guesses. The test collects real actions. The e-mail addresses collected identify people. The analyst removes them from the exports before passing the exports to an AI service hosted by a third party.

For the concierge, it groups the feedback recorded in the log by theme and spots the tasks that recur at each delivery. These tasks are the candidates for automation. A task handed to the model is already an automation. The team records it as such in the log. Without that entry, the log no longer points to the task to automate. The sales interview and the delivery of the service stay with the team: it learns from the customer during these exchanges.

Examples

A Lausanne team is preparing a service that draws up the weekly staff schedule for independent cafés and restaurants. It runs a smoke test, then a concierge MVP. The figures are an illustration: neither Ries nor Savoia gives a threshold.

Hypothesis written before the testAt least 8% of independent cafés and restaurants in French-speaking Switzerland sign up for a trial at CHF 15 a week.
ThresholdThe service covers its costs at 480 customers. The team estimates the market at 6'000 independent cafés and restaurants: 480 / 6'000 = 8%.
AdTargeted at managers of independent cafés and restaurants in French-speaking Switzerland. Budget of CHF 300 over three weeks.
Ad impressions18'400
Clicks to the page410 of 18'400 impressions (2.2%)
Trial sign-ups (business name and e-mail address)41 of 410 visits (10%)
Page after sign-up"This service does not exist yet. The first schedules will be prepared by hand from April. As a thank-you, here is a weekly schedule template to print."
Verdict10% against a threshold of 8%: interest established. Lasting interest remains to be tested.
The smoke test record: the hypothesis written before the test, the counts at each step and the verdict. Invented example.

The team moves on to the concierge. Before the first customer, it writes down what it expects: of the first six businesses, at least five are still paying in week 6 and one recommends the service. It first serves two of the businesses that signed up. It devotes at most nine hours a week to the service.

WeekPaying customersPayments receivedService hoursHours asking staff for their absencesFact recorded
12CHF 3031.5"The PDF does not print on A4": layout corrected
23CHF 454.52.25Add the Vaud school holidays
34CHF 6063Receive the schedules on Thursday
45CHF 757.53.75A customer recommends the service to a fellow owner
56CHF 9094.5Nine-hour ceiling reached
66CHF 9094.5Seventh business turned away for lack of time
The concierge log summarised by week: paying customers, payments, service hours and the hours spent asking staff for their absences. Invented example.

In week 6, all six businesses are still paying and one of them has recommended the service. The figures written before the first customer are met: the team keeps its growth model. The same week, it turns away a seventh business. Ries places the move to automation at this point. Asking staff for their absences by phone takes half the hours. A form that the staff fill in replaces these calls first.

Visualisations

The diagram in the Description section aligns the two techniques on the same steps and marks the concierge's stopping rule in orange. The smoke test record and the concierge log show the deliverables of the example.

Cost

PhaseSmoke testConcierge MVP
PreparationLow: write the hypothesis, set up the page and the ad in a few hours.Medium: interviews until the first customer agrees to pay.
ExecutionLow: a fixed advertising budget and a few weeks of running ads.High: the team spends time on each customer and each delivery.
DocumentationLow: the hypothesis, the counts and the verdict fit on one page.Medium: the team writes one log line per delivery.

Tooling

For the smoke test, a page builder (Webflow, Carrd) carries the offer and a form (Typeform, Google Forms) collects the action. Advertising platforms (Google Ads, LinkedIn Ads, Meta) target the intended market and count impressions and clicks. A web analytics tool (Matomo, Google Analytics) counts visits and actions on the page. In a live product, a feature-flag system shows the announced link to a chosen cohort.

The concierge makes do with office tools. A spreadsheet keeps the log. The phone, e-mail and a messaging app carry the exchanges with each customer. Shared documents replace the screens of the future product. A QR-bill or TWINT collects payment for each delivery.

Sources

The procedure for the smoke test comes from an interview with Ries and from Alberto Savoia. The procedure for the concierge comes from Ries and from Manuel Rosso, who ran one.

  • Eric Ries, The Lean Startup, Crown Business, 2011: chapter 4 "Experiment" for early adopters, the HP programme and the name of the concierge MVP; chapter 6 "Test" for the forms of MVP, how to run the concierge and the Wizard of Oz test; chapter 7 "Measure" for the smoke test.
  • Eric Ries, What is the minimum viable product?, interview with Babak Nivi, Venture Hacks, 23 March 2009: the page that promises the product, the ads that bring traffic to it, the click-through rate and A/B tests read by cohort.
  • Alberto Savoia, Pretotype It, tenth-anniversary edition, 2022: the Fake Door and how to run it ethically; the Initial Level of Interest; the Mechanical Turk; the XYZ hypothesis, in the bonus chapter taken from The Right It.
  • Manuel Rosso, Concierge MVP: Learning from early adopters at Food on the Table, presentation, SlideShare: recruiting the first users, moving contact from meetings to the phone and then to e-mail, when to write code and the signal to move on to the next stage.
SMED
All techniques
Snaking the Drain