Design of experiments
A design of experiments (DOE) is a programme of runs fixed before the first run. The team chooses the factors of a process and gives each one a low level and a high level. It then carries out the runs in the planned combinations. Comparing the results shows which factors move the target measure and which act together. R. A. Fisher set out the method in 1935: a trial that varies several factors in a fixed pattern yields information on each of them. Changing one factor at a time takes more runs for the same precision and shows no interaction.
Goal
A process owner wants to know which settings move a measured result and by how much. The owner also wants to know which settings act together. There is an interaction between two factors when the effect of one depends on the level of the other.
NIST defines design of experiments as an efficient procedure for planning experiments whose data lead to valid and objective conclusions. The design states which factors the team varies, at which levels, in which combinations and in which order.
The design prepares a decision: the setting to adopt, the factors to watch, the ones the team does not need to control. The deliverable is the completed design matrix with the table of effects and the interaction plot.
Usage
When to use it
- A process with several adjustable parameters: find the combination of settings that reaches a quality or yield target.
- One-factor-at-a-time trials that led nowhere: look for the interaction those trials cannot see.
- The Improve phase of a DMAIC project: measure the effect of the selected causes and of their combinations in only a few runs.
- Candidate causes from a root cause analysis: test the ones that can be adjusted.
- Many candidate factors: a screening design keeps the few factors that matter.
- Expensive runs: each run of a factorial design yields information on every factor.
- A process sensitive to outside conditions: look for a setting that gives the right result despite the uncontrolled factors.
- A choice between suppliers or materials: compare the options under several production conditions.
When not to use it
- Factors that cannot be set: with observational data only, prefer hypothesis formulation and testing.
- An unstable process: its drifts mask the effects. Stabilise it first with a control chart.
Description
Factors, levels and response
Design of experiments treats the process as a black box. The factors go in and the response comes out.
- Material batch
- Operator
- Day
- Ambient temperature
- Humidity
A factor is an input that the experimenter sets: a temperature, a pressure, a cycle time, a supplier. It can be continuous or take only a few distinct values. A level is the value a factor takes during a run. A two-level design gives each factor a low level (coded -1) and a high level (coded +1).
The response is the result measured at the output. A run operates the process at one combination of levels and measures the response.
Uncontrolled factors also act on the response. The team does not set them. Some are known and identifiable: the material batch, the operator, the day. These are the nuisance factors. The design neutralises them with blocks. Others go unrecorded: the ambient temperature, the humidity. Randomisation spreads their effect.
The main effect of a factor is the difference between the mean response of the runs at its high level and the mean response of the runs at its low level. An interaction is calculated the same way from the product of the signs of two factors. Fisher notes that it is symmetrical: if B modifies the effect of A, A modifies the effect of B.
One factor at a time: more runs, no interaction
The intuitive method starts from the current setting. It changes one factor, keeps the best result and moves on to the next factor. This practice comes from the teaching of elementary physics. It works for demonstrating a known law. A researcher does not know which factors matter or whether they act independently. Fisher treats changes to an apparatus, a machine or an industrial process as always liable to interact.
The factorial design is more economical. In Fisher's example, four ingredients of a mixture each take a small dose and a large dose. The 16 combinations are tested six times each: 96 runs. For each ingredient, 48 runs at the large dose are compared with 48 runs at the small dose. This comparison has the same precision as if all 96 runs had been spent on that one ingredient. Fisher concludes that the factorial design reaches the same precision with a quarter of the observations.
The factorial design also measures the interactions. Even with 96 runs per ingredient, separate experiments give no idea of them. If A improves the mixture only in the presence of B, only a trial of A with B and without B reveals it. The factorial design makes this comparison for every pair of factors.
Fisher adds that a conclusion drawn from runs in which the other factors varied holds over a wider range than a conclusion drawn under constant conditions.
A moulder sets the mould temperature (40 or 60 °C) and the holding pressure (500 or 700 bar) for a clip. The response is the pull-off force of the clip. The one-factor-at-a-time method starts at 40 °C and 500 bar, where the mean force over the two cooling times is 48 N. Raising the mould temperature to 60 °C gives 46 N: the team keeps 40 °C. Raising the pressure to 700 bar then gives 44 N: the team keeps 500 bar. It concludes that no setting improves the clip. It never tried 60 °C with 700 bar. That setting gives 62 N.
Randomisation, replication and blocks
Randomisation sets the order of the runs by a random draw. The standard order numbers the runs by the signs of the factors. In that order the sign of A changes fastest. In the standard order of the clip, the cooling time stays at its low level for runs 1 to 4 and at its high level for runs 5 to 8. If the mould heats up over the day, this drift passes for an effect of the cooling time. The random draw spreads such drifts over all the factors.
Replication runs the complete design more than once. It estimates the experimental error, which the team needs to judge whether an effect is real. Each combination then gets a mean and a measure of spread. The team checks that the variability stays the same across the whole tested region. It also spots the settings that give erratic results.
A block groups runs made under homogeneous conditions: one material batch, one shift, one day. The nuisance factor stays constant within the block. The analysis compares the factor levels within each block. The difference between blocks no longer biases the estimated effects. Fisher proposes to split his 96 runs into six series that are more homogeneous than the whole.
Choosing the design
The choice depends on the objective and on the number of factors. A comparative objective concerns one factor known to matter: is supplier A as good as supplier B? A screening objective separates the few factors that matter from the many factors that matter little. A response surface objective estimates interactions and curvature. The team uses it to find the best setting, to correct a weak point or to make the process robust against outside influences.
| Number of factors | Comparative | Screening | Response surface |
|---|---|---|---|
| 1 | Completely randomised one-factor design | Not applicable | Not applicable |
| 2 to 4 | Randomised block design | Full or fractional factorial design | Central composite or Box-Behnken design |
| 5 or more | Randomised block design | Fractional factorial or Plackett-Burman design | Screening first to reduce the number of factors |
With a single factor at two levels, the design reduces to a randomised comparison. On a web page or a message it is the A/B test.
A two-level full factorial design tries every combination of the levels. With k factors it takes 2k runs: 4 for two factors, 8 for three, 16 for four, 64 for six. With five or more factors, the team prefers a fractional design or a Plackett-Burman design, which screens with fewer runs.
The team chooses a design somewhat shorter than the budget allows. The reserve covers repeats of failed runs and the centre points. A centre point sets each factor halfway between its two levels. It reveals a curvature that two levels alone cannot see. It assumes quantitative factors.
Fractional designs and confounding
A fractional factorial design runs a chosen fraction of the full design: half, a quarter, an eighth. The half-fraction 23-1 keeps 4 of the 8 runs of a three-factor design.
In this half-fraction, the column of factor C repeats the product of columns A and B. The effect calculated for C is the sum of the effect of C and the AB interaction. These two effects are confounded (also called aliased). Each main effect of the half-fraction is likewise confounded with a two-factor interaction. The interaction belongs to the process. Confounding belongs to the design.
AB = C
The fractional design rests on an assumption that NIST calls sparsity of effects: the confounded interactions are small compared with the main effects. If they are negligible, the estimate of the main effects holds. In runs 2, 3, 5 and 8 of the clip, the effect calculated for the cooling time is 11 N: 1 N from that effect and 10 N from the temperature-pressure interaction.
The resolution of a fractional design states which effects it confounds.
| Resolution | Confounded effects |
|---|---|
| III | Main effects are confounded with two-factor interactions. |
| IV | No main effect is confounded with a two-factor interaction. At worst, a main effect is confounded with three-factor interactions. Some two-factor interactions are confounded with each other. |
| V | No main effect or two-factor interaction is confounded with any other main effect or two-factor interaction. Two-factor interactions are confounded with three-factor interactions. |
A high-order interaction is less likely to be significant than a two-factor interaction. A resolution IV design therefore confounds less severely than a resolution III design. The software that generates the design displays its resolution and its alias list. The team reads that list before the first run.
Running the experiment
- Set the objective: the decision to make, the response to measure and its measurement method.
- Choose the factors and their levels: two to four factors for a full design, more for screening.
- Choose the design: from the objective and the number of factors.
- Carry out the design: in the randomised order. Record everything that happens.
- Check the data: confirm that they meet the assumptions of the experiment.
- Analyse and interpret: read the main effects, then the interactions.
- Use the results: apply the adopted setting or prepare the next runs.
The team measures the response on a continuous scale where possible. It checks the measuring instrument first. A measurement that varies more than the effect sought masks that effect.
The low and high levels lie far enough apart to produce a visible effect. They stay within the range the process tolerates without damage. The process owner approves that range. The team also checks that every planned combination is feasible: two levels that are possible separately are not always possible together.
During the runs the team avoids any unplanned change such as an operator replaced halfway. It watches for process drift and keeps time and material in reserve for the unexpected. It keeps all the raw data, never only the averages. It returns the equipment to its initial state at the end. The team obtains the agreement of every stakeholder before the first run, since the design ties up a machine or a line.
The team runs a sequence of small experiments rather than one large one. A screening design keeps the factors that matter. A response surface design then sets those factors as close to the target as possible.
Reading the results
The team looks at the data before any model: the histogram of the responses, the responses in run order, the centre points. A trend in run order signals a drift during the design.
The main effects plot shows the mean response of each factor at its two levels. The interaction plot shows the mean response of one factor at its two levels with one line per level of the other factor. Parallel lines indicate no interaction. Lines that diverge or cross indicate an interaction. When the interaction is strong, the average main effect misleads: the team reads the effect of one factor at each level of the other.
A design without replication does not estimate the experimental error directly. The normal or half-normal plot of effects then ranks the estimated effects. Negligible effects behave like noise and fall on a straight line. Large effects stand away from it. Next, the team examines the residuals. A residual is the difference between an observed response and the response the model predicts.
NIST calls for at least three confirmation runs at the best setting, even if that setting was part of the design. The team carries them out under conditions as close as possible to those of the design. Their responses must stay close to the prediction.
The confirmed setting then enters the process. Process capability measures whether it holds the tolerances. A control chart shows whether it drifts.
Pitfalls
Reading the main effect alone. In the clip example, the average effect of temperature is 8 N. Yet at 500 bar a higher temperature costs 2 N. It helps only at 700 bar.
Keeping the standard order. Running the trials in the order of the design matrix ties each factor to the time of the run. A process drift then passes for an effect.
Reading a fractional design as a full design. An effect confounded with a strong interaction takes on the value of that interaction. The alias list shows which effects the design does not separate.
Concluding outside the tested region. The design describes the process between the low and high levels. Beyond them it says nothing.
Putting too many factors in one design. The team keeps the experiment as simple as possible. A screening followed by a finer design costs less than one design that tries to settle everything at once.
Keeping only the averages. Only the raw data and the run log can explain an abnormal result.
AI considerations
A language model reads the process documents, the FMEA and the incident reports. It draws up a list of candidate factors with the setting ranges those documents cite. The team sorts that list. The language model also drafts the protocol: the design matrix, the record sheet, the instructions for the operators.
The language model writes the R or Python script that generates the design, randomises the order, calculates the effects and draws the plots. The calculation happens in the script. A language model that computes averages over eight rows by itself sometimes gets a sign or a row wrong. The team checks the script on a design whose effects it knows.
The language model does not set the levels. A level that is too high can damage the tooling or exceed a safety limit. The process owner sets the range. Nor does the language model decide that an effect is real: the half-normal plot and the confirmation runs decide. Manufacturing parameters and test results are often trade secrets. They go to an external AI service only with the agreement of the data owner.
Examples
A subcontractor in the Jura injection-moulds a housing with a clip. The team tests three factors at two levels in a full factorial design of 8 runs. The response is the pull-off force of the clip.
| Run | Run order | A: mould temperature | B: holding pressure | C: cooling time | Force (N) |
|---|---|---|---|---|---|
| 1 | 6 | - (40 °C) | - (500 bar) | - (10 s) | 47 |
| 2 | 2 | + (60 °C) | - (500 bar) | - (10 s) | 46 |
| 3 | 4 | - (40 °C) | + (700 bar) | - (10 s) | 44 |
| 4 | 8 | + (60 °C) | + (700 bar) | - (10 s) | 61 |
| 5 | 1 | - (40 °C) | - (500 bar) | + (20 s) | 49 |
| 6 | 7 | + (60 °C) | - (500 bar) | + (20 s) | 46 |
| 7 | 5 | - (40 °C) | + (700 bar) | + (20 s) | 44 |
| 8 | 3 | + (60 °C) | + (700 bar) | + (20 s) | 63 |
| Effect | Mean of runs at + | Mean of runs at - | Effect (N) |
|---|---|---|---|
| A: temperature | 54 | 46 | 8 |
| B: pressure | 53 | 47 | 6 |
| C: cooling | 50.5 | 49.5 | 1 |
| AB | 55 | 45 | 10 |
| AC | 50 | 50 | 0 |
| BC | 50 | 50 | 0 |
| ABC | 50.5 | 49.5 | 1 |
The AB interaction is the largest effect. The team adopts 60 °C and 700 bar. The estimated effect of the cooling time (1 N) is of the same order as the AC, BC and ABC effects: the team sets it at 10 s, which shortens the cycle. The mean of the eight runs is 50 N. Each effect adds half its value to this mean when its sign is + and subtracts that half when its sign is -. The sign of AB is the product of the signs of A and B. At 60 °C and 700 bar the signs of A, B and AB are all +. The three-effect model therefore predicts 50 + (8 + 6 + 10)/2 = 62 N at 60 °C and 700 bar. Three confirmation runs at this setting must stay close to 62 N.
Visualisations
The black-box diagram places the factors, the response and the uncontrolled factors. The square of settings shows the corner that the one-factor-at-a-time method never visits. The cube of the half-fraction shows the four runs kept and the confounding of C with AB. The interaction plot of the clip shows two lines that cross. The choice of design, the resolutions, the design matrix and the table of effects are tables.
Cost
| Phase | Level | Rationale |
|---|---|---|
| Preparation | Medium | Set the objective, factors, levels and design with the process owner. Check the measuring instrument. |
| Execution | High | Each run ties up the machine and consumes material. A design of 8 runs with 3 centre points and 3 confirmation runs comes to 14 runs. |
| Documentation | Low | The software produces the table of effects and the plots. The run log and the raw data are kept during the runs. |
Tooling
A spreadsheet is enough for a full design of 4 or 8 runs. It carries one column of signs per factor and per interaction with an average formula at the foot of each column. Sorting on a column filled by the RAND function randomises the run order.
Beyond that, statistical software generates the design, displays the resolution and the aliases of a fractional design, then draws the effects and interaction plots: Minitab, JMP, Design-Expert. In R, the FrF2 package builds two-level fractional designs. In Python, the statsmodels library fits the model and calculates the residuals.
On the shop floor, a record sheet on paper or on a tablet notes for each run the time, the operator, the material batch and any incident.
Sources
- R. A. Fisher, The Design of Experiments, 2nd ed., Oliver and Boyd, Edinburgh, 1937 (1st ed. 1935), ch. VI, §37 "The Single Factor", §38 "A Simple Factorial Scheme" and §39 "The Basis of Inductive Inference": the critique of the one-factor-at-a-time method, the example of the four ingredients in 96 runs, the definition of interaction, the three advantages of the factorial design.
- NIST/SEMATECH, e-Handbook of Statistical Methods, §5.1 Introduction: the definition, the black-box model, the uses, the seven steps, the practical precautions, the sequence of small experiments.
- NIST/SEMATECH, e-Handbook of Statistical Methods, §5.3.3 How do you select an experimental design?: the table for choosing a design, blocks, full factorial designs, the polishing example with replication, randomisation and centre points.
- NIST/SEMATECH, e-Handbook of Statistical Methods, §5.3.3.4.3 Confounding (also called aliasing): the half-fraction 23-1, the confounding of effects, sparsity of effects.
- NIST/SEMATECH, e-Handbook of Statistical Methods, §5.3.3.4.4 Fractional factorial design specifications and design resolution: resolutions III, IV and V.
- NIST/SEMATECH, e-Handbook of Statistical Methods, §5.4 Analysis of DOE data: the effects and interaction plots, the normal plot of effects, residuals, confirmation runs.
- G. E. P. Box, J. S. Hunter, W. G. Hunter, Statistics for Experimenters: Design, Innovation, and Discovery, 2nd ed., Wiley, Hoboken, 2005: a textbook of experimental statistics.

