FAQ Why can’t you use confidence intervals to analyse and make inferences from your experiment?

Why can’t you use confidence intervals to analyse and make inferences from your experiment in AS 3.11?

For the standard AS 3.11 teachers should be strongly discouraged from using CIs on the following grounds:

1. Cannot assess the strength of the evidence with a CI difference.

2. Randomisation test is about significance testing and quantifying the strength of evidence for the existence of any claimed effects or differences and is aligned with the logic of performing a randomised experiment. CIs try to estimate the size of any claimed effects or differences and Normal-theory CIs come from sampling theory.

3. Avoids duplication as AS 3.10 assesses CIs.

4. The use of the randomisation test is based on the key principle that the test should mimic the data production process, that is random assignment to two groups, an important principle in experimental design.  Using CIs violates this principle as the CI theory taught assumes that a random sample is taken from the population which is not the case with almost all experiments. There are two types of randomisation, random sampling and random assignment leading to two types of inference sample-to-population inference and experiment-to-causation inference respectively and therefore for beginners it is important to keep these two ideas separate.  Hence the randomisation test is used for experiments and CIs are used for samples from populations.

Please read this link for further information and background: Why do randomisation tests in the experiments standard?