Synthetic panel 60x800, iid returns, top_k=10, hold=10. Null = within-date score permutation.
1. Size (zero-alpha panel, noise signals)
K=150 noise signals, N=100 perms: fraction p<0.05 = 4.0% (target ~5%), mean p = 0.50, KS-vs-Uniform p = 0.884 (large = consistent with uniform).
2. Monte-Carlo stability of p vs permutation count
| perms | mean p (8 seeds) | std p | theoretical sqrt(p(1-p)/N) |
|---|---|---|---|
| 40 | 0.606 | 0.097 | 0.077 |
| 100 | 0.610 | 0.048 | 0.049 |
| 400 | 0.597 | 0.034 | 0.025 |
3. Power (persistent alpha, trailing-return signal)
| alpha_sd | mean p | frac p<0.05 (12 seeds) |
|---|---|---|
| 0.0000 | 0.310 | 17% |
| 0.0003 | 0.200 | 25% |
| 0.0006 | 0.117 | 42% |
| 0.0012 | 0.008 | 92% |
Reading
- A well-calibrated gate rejects ~5% of noise signals and its p-values are ~Uniform. If the size is far above 5%, the gate is too permissive.
- The K=150 size test is the authoritative calibration; the alpha=0 row of the power table (17% at n=12) is within sampling noise of 5% (P(>=2/12 false positives by chance) ~ 12%).
- p is a Monte-Carlo estimate: with N perms its std is ~sqrt(p(1-p)/N). At N=40 the std near p=0.5 is ~0.08, so a 40-perm p is only good to ~+/-0.08; use N>=200 for reported p-values (this is why H10 read 0.000 at N=40 and 0.020 at N=200).
- Power rises as the injected alpha grows, so the gate is not degenerate (it can detect a real signal).