Key terms

Glossary of key experimentation terms

Term	Definition
Assignment event	Another name for enrollment event.
Audience	A group of users targeted for the experiment. Amplitude usually splits this audience evenly into "control" and "variant" groups.
Baseline conversion rate	The current rate of your primary success metric before this experiment.
Bonferroni correction	A statistical technique that counteracts the multiple comparisons problem (also known as multiplicity or the look-elsewhere effect).
Confidence interval	A range of plausible values that contains the parameter of interest. The parameter to estimate is the difference in means between the treatment and control. For example, if the confidence level is 95 and the same experiment runs 100 times, the confidence interval in each run contains the true parameter at least 95 times.
Confidence / significance level	The probability of a false positive. For example, at a 95% confidence level, there's a 5% chance of detecting a change to your success metric when no actual change occurred.
Enrollment event	The event that converts users or customers into registered participants or members.
Exposure event	The event that indicates when a user has seen a change based on an experiment.
Guardrail metric	A metric you want to protect from regression while you increase your success metrics. For example, if you drive users to a free trial of your business product, trials of your consumer product can act as a counter metric. If business trials go up, consumer trials go down. Confirm the net effect is positive.
CUPED	Controlled-experiment using pre-existing data (CUPED) is an optional statistical technique that reduces variance in experimentation.
Hypothesis	An assumption about the methods that can solve or ease the problem statement, and the reasoning behind those methods.
p-value	The probability of observing data as extreme as what you saw or more, assuming no difference between treatment and control.
Payload	Variables attached to a variant. Use payloads to change flags and experiments remotely without a code change.
Primary success metric	The main metric you want to move by running this experiment. Ideally drives both customer and business success.
Problem statement	An explanation of the internal business or user problem you're trying to solve.
Primary metric	A quantitative measure that evaluates your experiment against your goals. The primary metric determines whether your hypothesis is accepted or rejected and whether your experiment has succeeded or failed.
Run time	The duration your experiment takes to run, based on the sample size needed per variant and your traffic levels.
Rollout percentage	The percentage or number of targeted users that receive this variant.
Sample size	The number of users or amount of traffic you need in each experimental variant to reliably detect statistical significance.
Secondary success metric	An additional metric you expect to move with this experiment.
Sequential testing	A statistical analysis where the sample size isn't fixed in advance. Sequential testing lets you conduct an A/B test, review your results, and conclude the test without inflating false positives.
Statistical power	The probability that you detect a change to your success metric when a change exists to detect.
T-test	A statistical analysis that compares means between two populations of data to decide if the difference is statistically significant.
Target lift / minimum detectable effect (MDE)	The percentage change you expect to drive on your primary success metric as a result of this experiment.
Type 1 error	Incorrectly classifying that a statistically significant difference exists between treatment and control when none exists.
Type 2 error	Incorrectly classifying that no difference exists between treatment and control when a difference exists.

Was this helpful?