Conceptual pathways: Experiments and the Randomisation Test

EXPERIMENTS

1. Why experiment?

Minimal issues to cover: experiments versus observational studies and deficiencies of the later for concluding that effects are causal. “In what key ways does an experiment differ from an observational study?”

2. How can we experiment?

Minimal issues to cover: What does a simple randomised experiment look like? Why randomise? …

EXPERIMENT-TO-CAUSATION INFERENCE:
                   the Randomization Test for drawing a conclusion from experimental data

3. What is the problem? (that the randomisation test is intended to solve)

Random allocation to groups, acting alone, can produce apparent group differences that are surprisingly large
.                       (VIT’s “Randomisation variation” module is aimed at establishing this)
.           a. So the group differences I see in my data may not result from real group differences
.           b. If randomisation acting alone can quite often produce apparent group differences larger than I see in my data,
.                then I haven’t proved anything (and this is completely obvious!)

[Note: Random allocation to groups is equivalent to randomly putting group labels on individuals]

4. Partial solution to the problem (A minimal triaging strategy):

If my observed group difference is not out in the tails of the distribution of group differences produced by random relabeling then I haven’t proved anything       (follows from 3b.)
My study (on its own) is inconclusive.

If it is out in the tails then I am on firmer ground in concluding that this is probably not just an artifact produced by randomization variation and that (provided the experiment has been executed correctly) the experimental intervention caused a true change.

Notes:

What is the purpose of simulation and dynamic graphics?

To unpack the meaning of the phrase “out in the tails of the distribution of group differences produced by random relabeling” and its components as concretely as possible.
And connect “out in the extreme tails” to small tail proportions

Do the tools used by statisticians use dynamic graphics?
No, they just instantly supply tail proportions.

Why do we?
As a part of a learning process to give meaning to tail probabilities/“p-values”

Can students learn this by just using software?
No. They need in teacher-led learning activities that make them concentrate on what is happening and why, and they need to get to the point where they can explain all of this to others. Until then they don’t understand it.