Premium problem42. Type I Error After Filtering the Sample

Hard Locked

A cautionary simulation about selecting on the outcome.

For each of trials repetitions:

  • draw n values from N(0,1)\mathcal{N}(0, 1), so the null hypothesis μ=0\mu = 0 is true
  • compute the sample mean, then discard every value below it
  • run a one-sided z-test on what is left, testing μ>0\mu > 0 with known σ=1\sigma = 1
  • count it as a rejection if the p-value is below alpha

Return the fraction of repetitions that rejected. Under a correct procedure this would sit near alpha; filtering on the sample's own mean is what breaks it.

Seed np.random.seed(seed) inside solve, immediately before drawing.

Input

n = 20
trials = 400
alpha = 0.05
seed = 0

Output

0.815

Premium problem

This one's part of Premium. Unlock the full Probability track plus every other premium problem on the site.

Implement solve(...)