Toolkit · Template 3 of 5 · v1.0
Experiment Ledger
A running record of small experiments, written so you cannot quietly decide afterwards what counted as a win.
For anyone running small marketing or product experiments who wants to avoid declaring a winner from thin data, and wants a record of what was tried, including the dead ends.
- Completion rate: the share of views that reach the end of a video.
Copy the templateWorked exampleWhy these fieldsAll five templates
How to use it
- Write the hypothesis and the decision rule before the experiment starts. Never fit the rule to the result afterwards.
- Change one thing between the two conditions. If two things changed, you cannot say which one mattered.
- Log every experiment, including the ones that “failed”. A dead end is data.
- Record raw counts for each condition in the Result field, then the rates they give. Do not write adjectives like “big” or “great”.
- One result is a data point, not a proven method. Say so in the Caveats field.
Copy the blank template
Plain text. Paste it into a document or notes app. Nothing on this page collects what you type. Free to copy, adapt and share.
Experiment Ledger (template v1.0) Experiment ID: [A unique reference, for example EXP-001.] Hypothesis: [What you believe and why, written before the run.] What changes: [The single thing that differs between conditions.] Control: [The baseline you compare against, and its usual rate.] Sample size: [How many views, tries or people each condition got.] Window: [Start and end dates.] Decision rule (locked): [The exact threshold that triggers a decision, and whether it is a relative gain or a number of percentage points. Written before results exist.] Result: [Raw counts for each condition, and the rate each one gives.] Decision: [What you actually decided.] Caveats: [Anything that weakens trust in the result.] Next action: [What happens because of this entry.]
CSV header (optional)
id,hypothesis, what_changes, control, sample_size, window, decision_rule, result, decision, caveats, next_action
Worked example
Example: one entry in a ledger. All names and numbers are invented and illustrative.
| Field | Example entry |
|---|---|
| Experiment ID | EXP-014 |
| Hypothesis | For a budgeting-app demo, an opening line that names a cost will hold viewers longer than one that asks a question. |
| What changes | The opening line only. Everything else is identical, and both versions ran over the same 3 days. |
| Control | The question-style opening, already in use. Its usual rate is 70 percent. |
| Sample size | 4,000 views each (invented number). |
| Window | 2000-01-03 to 2000-01-05. |
| Decision rule (locked) | Switch to the cost opening only if its completion rate is more than 20 percent higher than control, as a relative gain and not percentage points (above 84 percent, when control is 70 percent), measured at 72 hours. |
| Result | Control: 2,800 of 4,000 views completed (70 percent). Cost opening: 3,136 of 4,000 (78.4 percent). Cost opening: 12 percent higher completion than control, relative (8.4 percentage points). Below the 20 percent threshold. |
| Decision | Keep the control. Do not switch on this evidence. |
| Caveats | One test, one platform, invented numbers. |
| Next action | Repeat the test on a second platform before ruling the cost opening out. |
Keep the ledger as a spreadsheet or CSV, one row per experiment, with the template version noted in the file name. The one-line CSV header above is ready to paste. Any cell that contains a comma or a line break must be wrapped in double quotes, as spreadsheet apps do on export. When you add a column later, leave old rows blank for it rather than guessing.
Why these fields are here
The plain fields explain themselves. These are the ones people skip, and the reason each one is there.
- What changes
- If two things differ you cannot say which one mattered, and every result becomes arguable.
- Control
- A gain only means something against a baseline, and the baseline has to run over the same days as the change.
- Decision rule (locked)
- A rule written after the numbers are in will always find something to call a win.
- Result
- Adjectives hide the size of a gap. Counts let a later reader check the arithmetic and judge the sample.