Exploratory Data Analysis
Exploratory data analysis is the examination of a dataset with no prior hypothesis, to find out what it contains before drawing anything from it. The analyst checks the integrity of the extract, describes each variable, measures their dependencies, handles missing and outlying values, then plots charts until it is clear what the data allows. The IIBA guide places it upstream of hypothesis testing, machine learning and statistical inference: what it produces is a judgement on the data and on the research question the data can carry.
Goal
Exploratory data analysis questions a dataset before any analytical decision and answers three questions: what data is available, what state it is in and which research question it can carry. The IIBA guide sets the intent: to get the most out of the data by examining it, analysing it and summarising it, so as to gain familiarity with it before applying more formal approaches.
The term comes from John Tukey, whose 1977 book founded the discipline by setting two kinds of data analysis against each other. Confirmatory analysis starts from a hypothesis and sets out to settle it. Exploratory analysis starts from the data and lets it yield regularities, anomalies and hypotheses worth testing; Tukey describes it as detective work, done before the court sits. The NIST handbook gives the operational formulation: a philosophy of how a dataset is dissected, what is looked for in it and how it is looked at and interpreted.
The deliverable is a findings note. It carries the inventory of variables and their characteristics, the handling of missing and outlying values with its justification, the dependencies between variables, the preliminary hypotheses and the reformulation of the research question. That note binds the work that follows: it says what can be modelled, what has to be collected again and what this extract will not answer. Exploration establishes that a sample is biased; it does not correct it.
Exploration produces a preliminary hypothesis, which hypothesis formulation and testing then tries out on data held back for that purpose. It is also the precondition for data mining, which looks for the pattern meant to become a deployed model, whereas exploration stops at the judgement on the data.
Usage
When to use it
- An unknown dataset: the first extract received from a source system; nobody knows its real content, nor the share of duplicates, empty fields and impossible values.
- Before any modelling: establishing what the data supports and narrowing a still broad research question down to what is measurable with what exists.
- Data migration or takeover: profiling what is there governs the transformation rules.
- A new source to integrate: checking that it covers the expected population before adding it to the base.
- A contested finding: working through the extract again from the raw data to establish what the figures allow anyone to say.
When not to use it
- A hypothesis stated on data already known: move to hypothesis formulation and testing.
- A finding intended for a management board: exploratory plots are unreadable out of context; use business visualizations.
Description
The nine-step scheme
The IIBA guide offers a typical, iterative sequence, which it says structures the practice without fixing the order.
- Check consistency and integrity on loading: dimensions of the set, number of records, column labels and types, volume of missing values. Separate the training set from the validation set at this point.
- Describe each variable: type and shape of the distribution, central tendency (mean, median, mode), variance, skewness, kurtosis.
- Measure interdependence and collinearity: correlations, variance inflation factor, ANOVA, t-test, F-test, chi-squared test.
- Handle the missing values and the outliers: replacement by the mean, the median or the mode, Bayesian or algorithmic imputation, application of a business rule.
- Draw charts at the steps where they help and take the first findings from them.
- Build the derived variables: aggregates, ratios, groupings of categories, calculated indicators. This step is called feature engineering. Each derived variable goes back to step 2, which describes it like the others.
- Formulate and try out a simple hypothesis, to check that a finding stands up.
- Reformulate the research question from what the exploration has shown. A reformulation restarts the full round from step 1.
- Report the first findings, backed by plots, with the corroboration obtained elsewhere.
The scheme adjusts to the nature of the data, which the guide states on two cases. A set of images calls for dimension reduction and vectorisation before a descriptive statistic means anything. A text corpus calls for stop-word removal and part-of-speech tagging. Steps 1 and 7 to 9 hold in every case; the content of steps 2 to 6 changes.
Every decision is recorded when it is taken: the row discarded and the reason, the threshold chosen, the derived variable and its formula, the version of the extract. This log makes the exploration reproducible. Reconstructed three weeks later, it becomes a reconstruction from memory, whose distance from what was done can no longer be measured.
Plots are the analyst's instrument
The guide makes charts the main vehicle of understanding: the shape of a distribution or an outlier is read at a glance, whereas a table of statistics has to be worked through. The choice of plot depends on the number of variables, on whether they are categorical or continuous and on the step of the scheme.
| Family | Plots | What is read from them | Steps |
|---|---|---|---|
| Univariate | Histogram, probability distribution curve, run chart of the values in collection order | Shape and spread of one variable's distribution, frequencies, drift over the course of collection | 1, 2 |
| Bivariate | Bar chart, scatter plot, box plot, correlation heat map | Trend between two variables, outliers, interdependence and collinearity | 3, 4 |
| Special purpose | Scatter plot matrix, contour plots, density curves, radar chart, lag plot and autocorrelation plot, Weibull, logarithmic and lognormal plots | More than two variables at once, the dominant variable among many, the structure of a time series, a distribution spread over several orders of magnitude | 3, 5, 6 |
The guide separates the plots made to understand a phenomenon from those made to report a finding. An exploratory chart readily carries fifteen series, raw axes and no formatting, because its single reader knows what they are looking for in it. Putting that same image in front of a committee is the business of business visualizations and data storytelling, which start again from one message and a named recipient.
Reporting and corroborating the findings
The guide asks that findings be validated by other business analysis techniques before they are shared. Business rules analysis explains an outlier by an entry rule or a missing control. Process analysis explains an empty field by a step nobody performs. Elicitation with the people who operate the system settles whether it is a data anomaly or the intended behaviour.
The report carries the scheme followed, the hypotheses adopted, the decision points and the provenance of the data, as much as the finding itself. A finding delivered without its trail can be neither contested nor replayed.
The pitfalls
Retreat into the quantitative
The guide counts this drift among the limitations of the technique: the analyst becomes absorbed in the data at hand and loses sight of the question asked. The symptom is an exploration that drags on and produces correlations nobody knows what to do with. The research question is written at the head of the exploration log and re-read at each iteration. A working session with someone from the business closes each round: the guide makes the technique depend on domain knowledge as much as on statistics.
The hypothesis born of the exploration and checked on the same data
A dataset observed long enough ends up showing regularities that chance alone explains. A hypothesis found during the exploration and then tested on the same extract confirms itself. This is the separation Tukey draws between the two kinds of analysis: exploration suggests, confirmation settles on data the exploration has not seen. The guide's scheme applies it by separating the training set from the validation set at the very first step, before any examination: the second is opened to try out what the first suggested.
The outlier taken for an error
A value out of range is a data entry error or the most informative point in the set. A claim of CHF 480'000 in a portfolio whose median is CHF 2'400 may be the very event one is trying to predict. How to handle it depends on the value's origin, found in the source system or recorded as untraceable. Distance from the mean flags the value; it does not explain it.
The imputation that manufactures information
Replacing missing values with the mean tightens the variance and creates an artificial mass at the centre of the distribution. Applied to the fifteen per cent of rows in the example below, the procedure distorts every statistic computed afterwards. Look first for the business rule that explains the absence, since an empty field almost always has a cause in the process, then impute only the remainder. Every imputation is marked with an indicator variable, so that the work that follows knows which rows carry a manufactured value.
Assumptions that do not survive a change of environment
The guide notes that models built on the assumptions of an exploration scale badly when the business environment and the objectives change markedly, which forces the exercise to be redone. The counter is to date the assumptions and to name, for each one, the business condition it depends on. A change of tariff or the opening of a sales channel then reads as an expired assumption rather than as an unexplained drift of the model.
AI considerations
The first use is producing the exploration code. From the data dictionary and an extract, an assistant writes in minutes the descriptive statistics pass, the correlation matrix and the first series of plots, repetitive work where a typo costs more than the thinking does. The second is interpreting an unusual shape: a bimodal distribution, a fat tail or a misaligned seasonality gets a list of candidate explanations, which the analyst then checks in the source system. The third is drafting the findings note from the exploration log, whose structure is fixed and whose content is already decided.
Three limits frame these uses. The model knows nothing of the process that produces the data: asked about the cause of an empty field, it supplies a plausible explanation where the technique calls for one established with whoever operates the system. Automated exploration tools, which deliver a report of several hundred charts on a simple load, bring the drift described above back in volume: quantitative material in abundance, with no research question to sort it. Finally, exploration works on production data, often personal data. Sending it to a hosted service is a decision under the revised Federal Act on Data Protection (revFADP), SR 235.1, taken before the assistant is called on. An anonymised extract or processing on infrastructure under the organisation's own control are the two common ways out.
Examples
A Geneva online retailer holds 18'400 orders over twelve months and wants to know whether it can model return risk. The findings note below is what the exploration hands back.
| Variable | What the exploration shows | Decision |
|---|---|---|
| order_amount (continuous) | Mean CHF 214, median CHF 138, right-skewed distribution. 23 orders above CHF 5'000 | Work on the logarithm of the amount. The 23 rows are reseller orders, out of scope |
| item_count (continuous) | Correlation of 0.81 with order_amount | Collinearity: only one of the two variables will enter a linear model |
| canton (categorical, 26 categories) | 71% of orders in GE and VD, nine cantons below 20 orders | Group into four categories: GE, VD, the rest of French-speaking Switzerland, German- and Italian-speaking Switzerland combined |
| delivery_days (continuous) | 2'760 missing values (15.0%), all of them on orders collected in store | Business rule: collection in store, no delivery time. No statistical imputation |
| returned (binary) | 1'601 returns, 8.7% of orders | Candidate target variable. Imbalanced classes, to be handled at modelling time |
| channel × returned (cross-tabulation) | App 858 returns on 5'200 orders (16.5%), website 634 on 9'900 (6.4%), in-store kiosk 109 on 3'300 (3.3%) | Preliminary hypothesis, to be tested formally |
The note concludes nothing about return risk. It establishes that a model is possible on this extract, on three written conditions: one amount variable out of the two, a grouping of the cantons, a treatment for the class imbalance. The gap between channels is the only lead the exploration brought up, and confirming it means testing it on the validation set put aside at step one, which is the business of hypothesis formulation and testing. The delivery row shows the order of operations: the 2'760 missing values are the trace of a delivery method, and imputation by the median would have manufactured fifteen per cent of delivery times that never existed.
Visualisations
The position of exploratory analysis in the analytics chain is the one object in this fiche whose sense rests on spatial relations: one input, the sourced data; a single output, the findings note; two branches leaving it, towards hypothesis testing and towards data mining; a return to the source when the extract does not suffice. A sentence lists them without showing that both continuations hang on the same deliverable, a note that judges the data and stops there. It is drawn.
The plot taxonomy and the findings note are grids of rows and columns. Printing them as images would cost the first its steps column, which the reader scans to find the step they are at. The second would lose what makes it a deliverable: text that is selectable, searchable and readable on a phone. Drawing the plots themselves would be a demonstration of statistical graphics, the subject of another fiche. The nine-step scheme is a numbered procedure: the two points where it loops back are named at the step that triggers them; a drawing of nine boxes would restate the nine labels to add two arrows.
The position figure serves a second time, set beside an exploration in progress: it asks where the work stands. A team already talking about variables to keep, a training set and model performance has chosen a branch before writing the findings note, so before judging the data. One question checks that in a review: which written document says what cannot be done with this extract.
Cost
| Phase | Level | Rationale |
|---|---|---|
| Preparation | Medium | Obtaining a representative extract, the access rights and the data dictionary goes through the owner of the source and through data protection. That wait often exceeds the duration of the analysis. |
| Execution | High | The run through the scheme is iterative and is measured in days per dataset. Among the limitations the guide records is that the technique demands command of a scientific software package and of a programming language; that skill governs the pace. |
| Documentation | Medium | The log is kept as the work proceeds and the findings note is drafted in half a day. What costs is the discipline: a decision not recorded the same day is not recovered. |
Tooling
A spreadsheet is enough for a few thousand rows and a dozen columns: pivot table, histogram, sorting by a column to see the extremes. It reaches its limit on volume and on reproducibility, since an exploration conducted with the mouse cannot be replayed.
Beyond that, the practice runs in a programming language: R and Python are where it is done, with their libraries for table manipulation, statistical computation and plotting; the guide cites them among the skills the technique demands. The entry cost is in the skill more than in the licence, since the tooling is open source.
The notebook (Jupyter, Quarto, R Markdown) is the medium of the exploration log: the code, the result and the commentary sit in one versioned file, which makes step 9 almost free.
Data profiling tools produce the portrait of an extract in a single command: types, missing values, distributions, pairwise correlations. They absorb step 1 and part of step 2. They ask no question, so their report stays a starting point.
The data dictionary and the catalogue give the meaning, the unit and the provenance of the columns, without which the exploration describes numbers whose meaning nobody knows. Dashboard tools are a poor fit for this step, since they assume a view already chosen, whereas exploration consists of trying thirty of them.
Sources
- IIBA, Guide to Business Data Analytics, §3.11 Exploratory Data Analysis: the definition, the nine-step scheme, the plot taxonomy, the reporting of findings, the strengths and the limitations.
- John W. Tukey, Exploratory Data Analysis, Addison-Wesley, 1977: the founding work, the opposition between exploratory and confirmatory analysis, the box plot.
- NIST/SEMATECH, e-Handbook of Statistical Methods, chapter 1: Exploratory Data Analysis: the operational definition and the objectives of exploration.
- Federal Act on Data Protection (revised FADP), SR 235.1: the Swiss frame that applies when personal data is sent to a hosted service.