{
  "method": "Exploratory within-run analysis. All 2,000 images matched by ID and SHA-256 across five models; all attempts retained, including retries and the refusal. Two-sided within-image label-swap permutation tests of marginal median and p95 differences (19,999 Monte Carlo swaps with +1 correction), plus exact two-sided binomial sign tests of P(A faster than B)=0.5. Holm adjustment jointly over all 30 tests (10 pairs x 3 outcomes). Pointwise 95% percentile bootstrap CIs from 19,999 paired image resamples stratified by the 30 scene classes. Exact pointwise binomial 95% CIs for faster fractions.",
  "plan": "Outcomes, all-pair scope, correction family, resample count and seed fixed before running these tests. Historical descriptive medians/p95 were already known; this is not a preregistered experiment.",
  "limits": "One trial per image; five provider processes launched together with up to six requests per model. Requests were not randomized/interleaved across independent time blocks. Per-request timestamps are not in the local exports. Timing begins after payload preparation and image download, includes request/retry backoff and response parsing, and ends before final logging. Shared network and service load, temporal dependence, provider differences and retries prevent causal attribution to model inference speed or claims across deployments. Image-level tests assume independent sampling units; block sensitivity uses Jev completion order only as a proxy for temporal grouping, not verified simultaneous blocks.",
  "seed": 20261029,
  "resamples": 19999,
  "numpy_version": "2.5.4",
  "scipy_version": "1.18.1",
  "source_sha256": {
    "runs/aid-definitions-v1-jev/jev.jsonl": "865e98e0e21b326daae4848b0c40e6d2f32d2a0a1693f6159515cd5657998aea",
    "runs/aid-definitions-v1-d1-ultra/d1-ultra.jsonl": "e4cd2115993232e9ded53f1debafc26427930fff591c9b0759e84d54372bcc5d",
    "runs/aid-definitions-v1-luna/luna.jsonl": "d376bf9b1b2619d650801de69c34ae7522bc18fd0a1b10192ef64800fb13ce91",
    "runs/aid-definitions-v1-d1/d1.jsonl": "f5d84b84fdff5a72cee8e277a058c5d70ab60bead1d19cfaf9100b27c304764c",
    "runs/aid-definitions-v1-clef/clef.jsonl": "52a50be24d6ec4972b17c50d4a49f0166cc7d79c680875098c509793602e9f58"
  },
  "models": {
    "jev": {
      "n": 2000,
      "median_seconds": 4.58733327050868,
      "p95_seconds": 16.946375262500077,
      "max_seconds": 95.89878812499228,
      "median_ci95": [
        4.401592304171208,
        4.7387065420043655
      ],
      "p95_ci95": [
        15.818080333003309,
        18.778034625021974
      ],
      "retried": 1,
      "invalid": 0
    },
    "d1-ultra": {
      "n": 2000,
      "median_seconds": 4.651496479506022,
      "p95_seconds": 17.160273814915854,
      "max_seconds": 51.061882999987574,
      "median_ci95": [
        4.539923166012159,
        4.805131020999397
      ],
      "p95_ci95": [
        16.0223671542044,
        18.252512677409683
      ],
      "retried": 0,
      "invalid": 0
    },
    "luna": {
      "n": 2000,
      "median_seconds": 4.6347458954987815,
      "p95_seconds": 17.505681806891516,
      "max_seconds": 100.05367412499618,
      "median_ci95": [
        4.49424902049941,
        4.773952687493875
      ],
      "p95_ci95": [
        16.05136143304844,
        18.896460138144903
      ],
      "retried": 19,
      "invalid": 1
    },
    "d1": {
      "n": 2000,
      "median_seconds": 4.730486104497686,
      "p95_seconds": 16.062344108043174,
      "max_seconds": 62.80001716699917,
      "median_ci95": [
        4.6207916459970875,
        4.841881208500126
      ],
      "p95_ci95": [
        14.676108872615439,
        18.03405881875078
      ],
      "retried": 0,
      "invalid": 0
    },
    "clef": {
      "n": 2000,
      "median_seconds": 5.183020624506753,
      "p95_seconds": 17.321287347614994,
      "max_seconds": 95.08638024999527,
      "median_ci95": [
        5.043389438380109,
        5.32897299999604
      ],
      "p95_ci95": [
        16.146269439251046,
        19.079039872596326
      ],
      "retried": 2,
      "invalid": 0
    }
  },
  "pairs": [
    {
      "a": "jev",
      "b": "d1-ultra",
      "n": 2000,
      "median_delta_seconds_a_minus_b": -0.06416320899734274,
      "median_delta_ci95": [
        -0.2802333392253786,
        0.11212462101320848
      ],
      "p95_delta_seconds_a_minus_b": -0.2138985524157775,
      "p95_delta_ci95": [
        -1.7622227280531761,
        1.8343060609522965
      ],
      "a_faster_count": 1038,
      "ties": 0,
      "a_faster_fraction": 0.519,
      "a_faster_fraction_ci95": [
        0.49683847151807253,
        0.541105748836802
      ],
      "median_p_raw": 0.4689,
      "p95_p_raw": 0.8407,
      "sign_p_raw": 0.09350774585201944,
      "single_attempt_valid_only": {
        "n": 1999,
        "a_faster_fraction": 0.5192596298149075,
        "median_delta_seconds": -0.06531570898368955,
        "p95_delta_seconds": -0.22364803812815381
      },
      "median_p_holm_30": 1.0,
      "p95_p_holm_30": 1.0,
      "sign_p_holm_30": 1.0
    },
    {
      "a": "jev",
      "b": "luna",
      "n": 2000,
      "median_delta_seconds_a_minus_b": -0.047412624990101904,
      "median_delta_ci95": [
        -0.258474457943521,
        0.13240224718319946
      ],
      "p95_delta_seconds_a_minus_b": -0.5593065443914398,
      "p95_delta_ci95": [
        -2.253785890357838,
        1.6677347576254393
      ],
      "a_faster_count": 1047,
      "ties": 0,
      "a_faster_fraction": 0.5235,
      "a_faster_fraction_ci95": [
        0.5013402447326581,
        0.545590764264273
      ],
      "median_p_raw": 0.6424,
      "p95_p_raw": 0.6389,
      "sign_p_raw": 0.03754099405048784,
      "single_attempt_valid_only": {
        "n": 1979,
        "a_faster_fraction": 0.5194542698332492,
        "median_delta_seconds": -0.025058666011318564,
        "p95_delta_seconds": -0.16096138330466303
      },
      "median_p_holm_30": 1.0,
      "p95_p_holm_30": 1.0,
      "sign_p_holm_30": 0.7883608750602445
    },
    {
      "a": "jev",
      "b": "d1",
      "n": 2000,
      "median_delta_seconds_a_minus_b": -0.14315283398900647,
      "median_delta_ci95": [
        -0.3422787634197448,
        0.03513541559805162
      ],
      "p95_delta_seconds_a_minus_b": 0.8840311544569026,
      "p95_delta_ci95": [
        -1.1044694575089655,
        3.074942752392957
      ],
      "a_faster_count": 1057,
      "ties": 0,
      "a_faster_fraction": 0.5285,
      "a_faster_fraction_ci95": [
        0.5063443010068337,
        0.5505720284104576
      ],
      "median_p_raw": 0.1183,
      "p95_p_raw": 0.3079,
      "sign_p_raw": 0.011493531080335328,
      "single_attempt_valid_only": {
        "n": 1999,
        "a_faster_fraction": 0.5287643821910956,
        "median_delta_seconds": -0.14537058398127556,
        "p95_delta_seconds": 0.869593791593811
      },
      "median_p_holm_30": 1.0,
      "p95_p_holm_30": 1.0,
      "sign_p_holm_30": 0.2528576837673772
    },
    {
      "a": "jev",
      "b": "clef",
      "n": 2000,
      "median_delta_seconds_a_minus_b": -0.5956873539980734,
      "median_delta_ci95": [
        -0.8182582362969697,
        -0.40313668126473207
      ],
      "p95_delta_seconds_a_minus_b": -0.3749120851149179,
      "p95_delta_ci95": [
        -2.4658935119421246,
        1.859838184069871
      ],
      "a_faster_count": 1182,
      "ties": 0,
      "a_faster_fraction": 0.591,
      "a_faster_fraction_ci95": [
        0.569082464406087,
        0.612650316531485
      ],
      "median_p_raw": 0.0001,
      "p95_p_raw": 0.7022,
      "sign_p_raw": 3.9810947825424245e-16,
      "single_attempt_valid_only": {
        "n": 1997,
        "a_faster_fraction": 0.5908863294942414,
        "median_delta_seconds": -0.5952083329902962,
        "p95_delta_seconds": -0.1015158502093918
      },
      "median_p_holm_30": 0.0026000000000000003,
      "p95_p_holm_30": 1.0,
      "sign_p_holm_30": 1.1943284347627273e-14
    },
    {
      "a": "d1-ultra",
      "b": "luna",
      "n": 2000,
      "median_delta_seconds_a_minus_b": 0.016750584007240832,
      "median_delta_ci95": [
        -0.1441669600113528,
        0.19142644462699535
      ],
      "p95_delta_seconds_a_minus_b": -0.3454079919756623,
      "p95_delta_ci95": [
        -2.0000597194527043,
        1.325893110458857
      ],
      "a_faster_count": 1023,
      "ties": 0,
      "a_faster_fraction": 0.5115,
      "a_faster_fraction_ci95": [
        0.48933946316989196,
        0.5336267756935661
      ],
      "median_p_raw": 0.8377,
      "p95_p_raw": 0.7105,
      "sign_p_raw": 0.31430442147996623,
      "single_attempt_valid_only": {
        "n": 1980,
        "a_faster_fraction": 0.5070707070707071,
        "median_delta_seconds": 0.04935266700340435,
        "p95_delta_seconds": 0.058187798260775025
      },
      "median_p_holm_30": 1.0,
      "p95_p_holm_30": 1.0,
      "sign_p_holm_30": 1.0
    },
    {
      "a": "d1-ultra",
      "b": "d1",
      "n": 2000,
      "median_delta_seconds_a_minus_b": -0.07898962499166373,
      "median_delta_ci95": [
        -0.21188995357151724,
        0.07713489772722806
      ],
      "p95_delta_seconds_a_minus_b": 1.09792970687268,
      "p95_delta_ci95": [
        -0.9323106816256839,
        2.828986507601726
      ],
      "a_faster_count": 1012,
      "ties": 0,
      "a_faster_fraction": 0.506,
      "a_faster_fraction_ci95": [
        0.4838433228231128,
        0.5281390627207848
      ],
      "median_p_raw": 0.3042,
      "p95_p_raw": 0.2011,
      "sign_p_raw": 0.6070563417975082,
      "single_attempt_valid_only": {
        "n": 2000,
        "a_faster_fraction": 0.506,
        "median_delta_seconds": -0.07898962499166373,
        "p95_delta_seconds": 1.09792970687268
      },
      "median_p_holm_30": 1.0,
      "p95_p_holm_30": 1.0,
      "sign_p_holm_30": 1.0
    },
    {
      "a": "d1-ultra",
      "b": "clef",
      "n": 2000,
      "median_delta_seconds_a_minus_b": -0.5315241450007306,
      "median_delta_ci95": [
        -0.7080845198222959,
        -0.35058812705538017
      ],
      "p95_delta_seconds_a_minus_b": -0.16101353269914043,
      "p95_delta_ci95": [
        -2.2689638048054563,
        1.5649121882806485
      ],
      "a_faster_count": 1168,
      "ties": 0,
      "a_faster_fraction": 0.584,
      "a_faster_fraction_ci95": [
        0.5620382857272758,
        0.6057150598362397
      ],
      "median_p_raw": 0.0001,
      "p95_p_raw": 0.9087,
      "sign_p_raw": 5.994327499060906e-14,
      "single_attempt_valid_only": {
        "n": 1998,
        "a_faster_fraction": 0.5835835835835835,
        "median_delta_seconds": -0.5270131240104092,
        "p95_delta_seconds": 0.1248399920179537
      },
      "median_p_holm_30": 0.0026000000000000003,
      "p95_p_holm_30": 1.0,
      "sign_p_holm_30": 1.738354974727663e-12
    },
    {
      "a": "luna",
      "b": "d1",
      "n": 2000,
      "median_delta_seconds_a_minus_b": -0.09574020899890456,
      "median_delta_ci95": [
        -0.25215487501700407,
        0.06633365668094478
      ],
      "p95_delta_seconds_a_minus_b": 1.4433376988483424,
      "p95_delta_ci95": [
        -0.701810962101574,
        3.23440208264547
      ],
      "a_faster_count": 996,
      "ties": 0,
      "a_faster_fraction": 0.498,
      "a_faster_fraction_ci95": [
        0.47585366505654625,
        0.5201522064232914
      ],
      "median_p_raw": 0.2669,
      "p95_p_raw": 0.1601,
      "sign_p_raw": 0.875624423433703,
      "single_attempt_valid_only": {
        "n": 1980,
        "a_faster_fraction": 0.502020202020202,
        "median_delta_seconds": -0.12705072949756868,
        "p95_delta_seconds": 1.039741908611905
      },
      "median_p_holm_30": 1.0,
      "p95_p_holm_30": 1.0,
      "sign_p_holm_30": 1.0
    },
    {
      "a": "luna",
      "b": "clef",
      "n": 2000,
      "median_delta_seconds_a_minus_b": -0.5482747290079715,
      "median_delta_ci95": [
        -0.7327806624911318,
        -0.3681790273949446
      ],
      "p95_delta_seconds_a_minus_b": 0.18439445927652187,
      "p95_delta_ci95": [
        -1.9946825501591237,
        2.0487242678764743
      ],
      "a_faster_count": 1156,
      "ties": 0,
      "a_faster_fraction": 0.578,
      "a_faster_fraction_ci95": [
        0.5560039826521357,
        0.5997669879013205
      ],
      "median_p_raw": 0.0001,
      "p95_p_raw": 0.8532,
      "sign_p_raw": 3.2144273664068376e-12,
      "single_attempt_valid_only": {
        "n": 1978,
        "a_faster_fraction": 0.5819009100101112,
        "median_delta_seconds": -0.5920725830073934,
        "p95_delta_seconds": 0.004062679452200513
      },
      "median_p_holm_30": 0.0026000000000000003,
      "p95_p_holm_30": 1.0,
      "sign_p_holm_30": 8.678953889298461e-11
    },
    {
      "a": "d1",
      "b": "clef",
      "n": 2000,
      "median_delta_seconds_a_minus_b": -0.4525345200090669,
      "median_delta_ci95": [
        -0.6242016837219125,
        -0.2922195545579599
      ],
      "p95_delta_seconds_a_minus_b": -1.2589432395718205,
      "p95_delta_ci95": [
        -3.44612144149615,
        0.9170877643775721
      ],
      "a_faster_count": 1161,
      "ties": 0,
      "a_faster_fraction": 0.5805,
      "a_faster_fraction_ci95": [
        0.5585178771850929,
        0.6022457498183729
      ],
      "median_p_raw": 0.0001,
      "p95_p_raw": 0.163,
      "sign_p_raw": 6.336915598622331e-13,
      "single_attempt_valid_only": {
        "n": 1998,
        "a_faster_fraction": 0.58008008008008,
        "median_delta_seconds": -0.45090106151474174,
        "p95_delta_seconds": -0.9637139605532958
      },
      "median_p_holm_30": 0.0026000000000000003,
      "p95_p_holm_30": 1.0,
      "sign_p_holm_30": 1.7743363676142526e-11
    }
  ],
  "block_sensitivity": {
    "20": [
      {
        "a": "jev",
        "b": "d1-ultra",
        "a_faster_fraction_ci95": [
          0.48700000000000004,
          0.5495
        ],
        "a_faster_fraction_ci99_5": [
          0.47399749999999996,
          0.5619999999999999
        ],
        "median_delta_ci95": [
          -0.33210092116787565,
          0.15013484587834666
        ],
        "median_delta_ci99_75": [
          -0.4576502349373914,
          0.27307351790816536
        ],
        "p95_delta_ci95": [
          -1.9953541634698844,
          4.09452961310053
        ],
        "p95_delta_ci99_75": [
          -2.929965264382628,
          9.20126350753055
        ]
      },
      {
        "a": "jev",
        "b": "luna",
        "a_faster_fraction_ci95": [
          0.485,
          0.5619999999999999
        ],
        "a_faster_fraction_ci99_5": [
          0.46949749999999996,
          0.5774999999999999
        ],
        "median_delta_ci95": [
          -0.3813311475467344,
          0.25736796211058366
        ],
        "median_delta_ci99_75": [
          -0.5437067090146592,
          0.4142288392953976
        ],
        "p95_delta_ci95": [
          -2.300539508867513,
          3.1920127536381133
        ],
        "p95_delta_ci99_75": [
          -3.3771602037984167,
          6.594596212007692
        ]
      },
      {
        "a": "jev",
        "b": "d1",
        "a_faster_fraction_ci95": [
          0.4925,
          0.564
        ],
        "a_faster_fraction_ci99_5": [
          0.47650000000000003,
          0.5785025000000006
        ],
        "median_delta_ci95": [
          -0.403182723934151,
          0.10379174091358419
        ],
        "median_delta_ci99_75": [
          -0.5328318747553203,
          0.24543252169656696
        ],
        "p95_delta_ci95": [
          -1.0581816814790965,
          3.660875912378467
        ],
        "p95_delta_ci99_75": [
          -2.3685799362450237,
          7.374721719232456
        ]
      },
      {
        "a": "jev",
        "b": "clef",
        "a_faster_fraction_ci95": [
          0.545,
          0.6350000000000001
        ],
        "a_faster_fraction_ci99_5": [
          0.5235,
          0.6535
        ],
        "median_delta_ci95": [
          -1.0033186458065757,
          -0.22948535862960856
        ],
        "median_delta_ci99_75": [
          -1.1908016303689875,
          0.011114803129640572
        ],
        "p95_delta_ci95": [
          -3.064274495234015,
          2.844580846888628
        ],
        "p95_delta_ci99_75": [
          -4.955371084409165,
          5.949434056019015
        ]
      },
      {
        "a": "d1-ultra",
        "b": "luna",
        "a_faster_fraction_ci95": [
          0.47100000000000003,
          0.5515000000000001
        ],
        "a_faster_fraction_ci99_5": [
          0.4539974999999999,
          0.5705025000000006
        ],
        "median_delta_ci95": [
          -0.27163383691076887,
          0.3414255814423086
        ],
        "median_delta_ci99_75": [
          -0.43025721253154187,
          0.5055722746906425
        ],
        "p95_delta_ci95": [
          -2.7427069851255474,
          1.7708187077187791
        ],
        "p95_delta_ci99_75": [
          -5.05884834635903,
          2.814961409781134
        ]
      },
      {
        "a": "d1-ultra",
        "b": "d1",
        "a_faster_fraction_ci95": [
          0.484,
          0.5285
        ],
        "a_faster_fraction_ci99_5": [
          0.4750000000000001,
          0.5379999999999999
        ],
        "median_delta_ci95": [
          -0.21269941756108893,
          0.09182378441182663
        ],
        "median_delta_ci99_75": [
          -0.2854322390824018,
          0.16724624966518614
        ],
        "p95_delta_ci95": [
          -1.4060546618759766,
          2.6623941734515393
        ],
        "p95_delta_ci99_75": [
          -2.3919488072467012,
          3.46228491739125
        ]
      },
      {
        "a": "d1-ultra",
        "b": "clef",
        "a_faster_fraction_ci95": [
          0.5385,
          0.628
        ],
        "a_faster_fraction_ci99_5": [
          0.5184975,
          0.6465
        ],
        "median_delta_ci95": [
          -0.8867739025277842,
          -0.14784214326864478
        ],
        "median_delta_ci99_75": [
          -1.055470514707704,
          0.03531684479487972
        ],
        "p95_delta_ci95": [
          -3.8931360099493695,
          2.034559400915167
        ],
        "p95_delta_ci99_75": [
          -5.507372129849289,
          2.974145195059979
        ]
      },
      {
        "a": "luna",
        "b": "d1",
        "a_faster_fraction_ci95": [
          0.45699999999999996,
          0.539
        ],
        "a_faster_fraction_ci99_5": [
          0.4389974999999999,
          0.5560025000000005
        ],
        "median_delta_ci95": [
          -0.38676754930929746,
          0.210015705625119
        ],
        "median_delta_ci99_75": [
          -0.5572327277733712,
          0.37485800430091887
        ],
        "p95_delta_ci95": [
          -1.1856480154161564,
          3.2912132190122114
        ],
        "p95_delta_ci99_75": [
          -2.765111903226372,
          4.301960448126892
        ]
      },
      {
        "a": "luna",
        "b": "clef",
        "a_faster_fraction_ci95": [
          0.5365,
          0.618
        ],
        "a_faster_fraction_ci99_5": [
          0.5185000000000001,
          0.6359999999999999
        ],
        "median_delta_ci95": [
          -0.885133430150745,
          -0.2174657669158478
        ],
        "median_delta_ci99_75": [
          -1.0599344454825768,
          -0.02473227214813772
        ],
        "p95_delta_ci95": [
          -3.0069795668988033,
          2.2348509594699832
        ],
        "p95_delta_ci99_75": [
          -4.345032671140142,
          3.5010055378771714
        ]
      },
      {
        "a": "d1",
        "b": "clef",
        "a_faster_fraction_ci95": [
          0.5315000000000001,
          0.628
        ],
        "a_faster_fraction_ci99_5": [
          0.5099974999999999,
          0.6485
        ],
        "median_delta_ci95": [
          -0.811771978080651,
          -0.11552068330202023
        ],
        "median_delta_ci99_75": [
          -0.9822402099785904,
          0.09066501199868084
        ],
        "p95_delta_ci95": [
          -4.079199416999472,
          1.2533091626939126
        ],
        "p95_delta_ci99_75": [
          -5.832597266397611,
          2.632841707779653
        ]
      }
    ],
    "50": [
      {
        "a": "jev",
        "b": "d1-ultra",
        "a_faster_fraction_ci95": [
          0.48950000000000005,
          0.5485
        ],
        "a_faster_fraction_ci99_5": [
          0.47700000000000004,
          0.5599999999999999
        ],
        "median_delta_ci95": [
          -0.3753731790166057,
          0.15643819758042785
        ],
        "median_delta_ci99_75": [
          -0.5358716816455853,
          0.2812251391197431
        ],
        "p95_delta_ci95": [
          -2.1248754662083282,
          6.22717171974515
        ],
        "p95_delta_ci99_75": [
          -3.1424630083042575,
          9.459400787828013
        ]
      },
      {
        "a": "jev",
        "b": "luna",
        "a_faster_fraction_ci95": [
          0.4755,
          0.569
        ],
        "a_faster_fraction_ci99_5": [
          0.45299749999999994,
          0.5885025000000005
        ],
        "median_delta_ci95": [
          -0.41801346622087293,
          0.33832911695790235
        ],
        "median_delta_ci99_75": [
          -0.6014817663131907,
          0.5948404757550887
        ],
        "p95_delta_ci95": [
          -2.267944938830104,
          3.345760103420325
        ],
        "p95_delta_ci99_75": [
          -5.1698559294834086,
          6.2194179159996565
        ]
      },
      {
        "a": "jev",
        "b": "d1",
        "a_faster_fraction_ci95": [
          0.49399999999999994,
          0.5625
        ],
        "a_faster_fraction_ci99_5": [
          0.47850000000000004,
          0.578
        ],
        "median_delta_ci95": [
          -0.4430225784373761,
          0.1084528533014236
        ],
        "median_delta_ci99_75": [
          -0.6197434933298791,
          0.2586609373455614
        ],
        "p95_delta_ci95": [
          -1.0858036125006052,
          5.455123516811859
        ],
        "p95_delta_ci99_75": [
          -2.1562977996611252,
          8.584392629052049
        ]
      },
      {
        "a": "jev",
        "b": "clef",
        "a_faster_fraction_ci95": [
          0.5315,
          0.6445249999999997
        ],
        "a_faster_fraction_ci99_5": [
          0.5035000000000001,
          0.6655
        ],
        "median_delta_ci95": [
          -1.0734417595522245,
          -0.08942973759913019
        ],
        "median_delta_ci99_75": [
          -1.2888012092828285,
          0.1833486322363062
        ],
        "p95_delta_ci95": [
          -3.022830207948573,
          3.610207703342168
        ],
        "p95_delta_ci99_75": [
          -4.9231788702565575,
          6.566197511759547
        ]
      },
      {
        "a": "d1-ultra",
        "b": "luna",
        "a_faster_fraction_ci95": [
          0.457,
          0.5660000000000001
        ],
        "a_faster_fraction_ci99_5": [
          0.4349974999999999,
          0.5885
        ],
        "median_delta_ci95": [
          -0.37536285354872234,
          0.45238457055966114
        ],
        "median_delta_ci99_75": [
          -0.5645852006359928,
          0.7059969578433499
        ],
        "p95_delta_ci95": [
          -4.792628898891498,
          1.9997013621863111
        ],
        "p95_delta_ci99_75": [
          -7.134475393897142,
          3.089722958830413
        ]
      },
      {
        "a": "d1-ultra",
        "b": "d1",
        "a_faster_fraction_ci95": [
          0.48650000000000004,
          0.5265000000000001
        ],
        "a_faster_fraction_ci99_5": [
          0.4779999999999999,
          0.5355000000000001
        ],
        "median_delta_ci95": [
          -0.2125986250139249,
          0.10317491597888874
        ],
        "median_delta_ci99_75": [
          -0.2817775738508499,
          0.19286351116290576
        ],
        "p95_delta_ci95": [
          -1.4297437392990169,
          2.4444351679048033
        ],
        "p95_delta_ci99_75": [
          -2.4798375482115524,
          3.180341665409801
        ]
      },
      {
        "a": "d1-ultra",
        "b": "clef",
        "a_faster_fraction_ci95": [
          0.5225,
          0.641
        ],
        "a_faster_fraction_ci99_5": [
          0.4939974999999999,
          0.6645
        ],
        "median_delta_ci95": [
          -0.971251183343702,
          0.005495364662783572
        ],
        "median_delta_ci99_75": [
          -1.1584223624576042,
          0.29869071585301005
        ],
        "p95_delta_ci95": [
          -4.104755272901957,
          2.2201020300461063
        ],
        "p95_delta_ci99_75": [
          -5.918162041598386,
          3.463363218156362
        ]
      },
      {
        "a": "luna",
        "b": "d1",
        "a_faster_fraction_ci95": [
          0.44400000000000006,
          0.552
        ],
        "a_faster_fraction_ci99_5": [
          0.41850000000000004,
          0.575
        ],
        "median_delta_ci95": [
          -0.491324728255131,
          0.29694480030157133
        ],
        "median_delta_ci99_75": [
          -0.7275672602167469,
          0.5034315104028066
        ],
        "p95_delta_ci95": [
          -0.5805448743616424,
          3.776533957035287
        ],
        "p95_delta_ci99_75": [
          -1.6080830512024133,
          5.766000405890704
        ]
      },
      {
        "a": "luna",
        "b": "clef",
        "a_faster_fraction_ci95": [
          0.5235,
          0.6285000000000001
        ],
        "a_faster_fraction_ci99_5": [
          0.49650000000000005,
          0.6475
        ],
        "median_delta_ci95": [
          -0.9306007089042396,
          -0.14684799514579994
        ],
        "median_delta_ci99_75": [
          -1.135238905517981,
          0.10761476043764998
        ],
        "p95_delta_ci95": [
          -2.179816914225234,
          1.8378239156285414
        ],
        "p95_delta_ci99_75": [
          -3.323825861628984,
          3.44947007450734
        ]
      },
      {
        "a": "d1",
        "b": "clef",
        "a_faster_fraction_ci95": [
          0.5155000000000001,
          0.643
        ],
        "a_faster_fraction_ci99_5": [
          0.4879975,
          0.6685000000000001
        ],
        "median_delta_ci95": [
          -0.9033629554134677,
          0.03286446739366504
        ],
        "median_delta_ci99_75": [
          -1.0991542571481476,
          0.3001598993646569
        ],
        "p95_delta_ci95": [
          -3.891078728318828,
          1.062019625649551
        ],
        "p95_delta_ci99_75": [
          -5.4273695409890435,
          2.4010044374350943
        ]
      }
    ],
    "100": [
      {
        "a": "jev",
        "b": "d1-ultra",
        "a_faster_fraction_ci95": [
          0.49749999999999994,
          0.5395
        ],
        "a_faster_fraction_ci99_5": [
          0.487,
          0.549
        ],
        "median_delta_ci95": [
          -0.284215563966427,
          0.06418363128759656
        ],
        "median_delta_ci99_75": [
          -0.3935579105504439,
          0.14780456150270999
        ],
        "p95_delta_ci95": [
          -1.701276071897369,
          6.722919657820087
        ],
        "p95_delta_ci99_75": [
          -2.0322017483612833,
          7.828028708987404
        ]
      },
      {
        "a": "jev",
        "b": "luna",
        "a_faster_fraction_ci95": [
          0.4655,
          0.5755
        ],
        "a_faster_fraction_ci99_5": [
          0.4370000000000001,
          0.5955
        ],
        "median_delta_ci95": [
          -0.4575940130787785,
          0.4174618958124482
        ],
        "median_delta_ci99_75": [
          -0.6444010299315415,
          0.7341714919265732
        ],
        "p95_delta_ci95": [
          -1.9413246687530772,
          2.317102838502617
        ],
        "p95_delta_ci99_75": [
          -2.843403996037615,
          3.9612092510215007
        ]
      },
      {
        "a": "jev",
        "b": "d1",
        "a_faster_fraction_ci95": [
          0.49850000000000005,
          0.56
        ],
        "a_faster_fraction_ci99_5": [
          0.487,
          0.5730000000000001
        ],
        "median_delta_ci95": [
          -0.3830804537843505,
          0.06531701218264048
        ],
        "median_delta_ci99_75": [
          -0.5119871708974096,
          0.19695668988693887
        ],
        "p95_delta_ci95": [
          -0.5798264451672019,
          5.47780628395995
        ],
        "p95_delta_ci99_75": [
          -1.4045288985871665,
          7.505644043034408
        ]
      },
      {
        "a": "jev",
        "b": "clef",
        "a_faster_fraction_ci95": [
          0.5125,
          0.655
        ],
        "a_faster_fraction_ci99_5": [
          0.4755000000000001,
          0.6745025000000004
        ],
        "median_delta_ci95": [
          -1.136290309677861,
          0.09148863197042323
        ],
        "median_delta_ci99_75": [
          -1.3272199995317715,
          0.6095308812276894
        ],
        "p95_delta_ci95": [
          -2.846252617583232,
          3.4804239590012003
        ],
        "p95_delta_ci99_75": [
          -4.40989547946748,
          5.652438669186659
        ]
      },
      {
        "a": "d1-ultra",
        "b": "luna",
        "a_faster_fraction_ci95": [
          0.44650000000000006,
          0.5765
        ],
        "a_faster_fraction_ci99_5": [
          0.4185,
          0.608
        ],
        "median_delta_ci95": [
          -0.4035152905045834,
          0.5443136433714243
        ],
        "median_delta_ci99_75": [
          -0.6269868118148588,
          0.8525967996340649
        ],
        "p95_delta_ci95": [
          -5.995986901545182,
          2.0818258361204784
        ],
        "p95_delta_ci99_75": [
          -7.075493043556515,
          2.958178570463838
        ]
      },
      {
        "a": "d1-ultra",
        "b": "d1",
        "a_faster_fraction_ci95": [
          0.48650000000000004,
          0.5265
        ],
        "a_faster_fraction_ci99_5": [
          0.4795,
          0.5349999999999999
        ],
        "median_delta_ci95": [
          -0.2214622756560857,
          0.11702945529686985
        ],
        "median_delta_ci99_75": [
          -0.30259954892168023,
          0.21302660974048465
        ],
        "p95_delta_ci95": [
          -1.3472109907437577,
          2.2966799336602044
        ],
        "p95_delta_ci99_75": [
          -2.053391333989566,
          2.8318001271872757
        ]
      },
      {
        "a": "d1-ultra",
        "b": "clef",
        "a_faster_fraction_ci95": [
          0.5025000000000002,
          0.6545
        ],
        "a_faster_fraction_ci99_5": [
          0.4620000000000001,
          0.6809999999999999
        ],
        "median_delta_ci95": [
          -1.0444749193986353,
          0.1955801149320905
        ],
        "median_delta_ci99_75": [
          -1.2102051013886739,
          0.7209699514046554
        ],
        "p95_delta_ci95": [
          -4.1023883089562085,
          2.4450053053762537
        ],
        "p95_delta_ci99_75": [
          -5.392598149078578,
          3.520135515885059
        ]
      },
      {
        "a": "luna",
        "b": "d1",
        "a_faster_fraction_ci95": [
          0.433,
          0.5614999999999999
        ],
        "a_faster_fraction_ci99_5": [
          0.4019975,
          0.589
        ],
        "median_delta_ci95": [
          -0.5433654261942138,
          0.32754960385136656
        ],
        "median_delta_ci99_75": [
          -0.8428309387456102,
          0.5726440640101284
        ],
        "p95_delta_ci95": [
          -0.7385718259333093,
          4.884911291999742
        ],
        "p95_delta_ci99_75": [
          -1.624258537235608,
          5.803566904628014
        ]
      },
      {
        "a": "luna",
        "b": "clef",
        "a_faster_fraction_ci95": [
          0.506975,
          0.6365000000000001
        ],
        "a_faster_fraction_ci99_5": [
          0.4739999999999999,
          0.6565025000000004
        ],
        "median_delta_ci95": [
          -0.954555766104022,
          -0.03781154422322385
        ],
        "median_delta_ci99_75": [
          -1.1378162264342608,
          0.32069523542592016
        ],
        "p95_delta_ci95": [
          -1.7757470904146628,
          2.4546428330324166
        ],
        "p95_delta_ci99_75": [
          -2.7305400784471154,
          3.683354008279745
        ]
      },
      {
        "a": "d1",
        "b": "clef",
        "a_faster_fraction_ci95": [
          0.496,
          0.6555249999999998
        ],
        "a_faster_fraction_ci99_5": [
          0.4575,
          0.683
        ],
        "median_delta_ci95": [
          -0.9564960726114805,
          0.21040080106031378
        ],
        "median_delta_ci99_75": [
          -1.1545091935850724,
          0.7115422518156967
        ],
        "p95_delta_ci95": [
          -4.1695410524712395,
          1.415753296143448
        ],
        "p95_delta_ci99_75": [
          -5.784128000028431,
          2.863598557511088
        ]
      }
    ]
  },
  "method_sources": [
    "https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.permutation_test.html",
    "https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.binomtest.html",
    "https://www.statsmodels.org/stable/generated/statsmodels.stats.multitest.multipletests.html"
  ],
  "block_sensitivity_method": "Resample contiguous blocks of 20, 50, or 100 paired images in Jev completion order, without class stratification. For each block size, 99.75% percentile intervals use a Bonferroni level for 20 median/p95 contrasts; 99.5% sign-fraction intervals use a level for 10 pairs. Also report pointwise 95% intervals. This is a sensitivity analysis using completion order as a temporal proxy; it cannot reconstruct simultaneous request time blocks. Bootstrap intervals are approximate.",
  "interpretation": {
    "median": "Clef had a larger median than all four other models; each image-level Monte Carlo permutation comparison has Holm-adjusted p=0.0026 in the joint 30-test family. This value reflects the Monte Carlo resolution; none of the 19,999 swaps in the smaller tail exceeded the observed difference. No median difference among the other four was significant.",
    "p95": "None of the ten 95th-percentile comparisons was significant; all Holm-adjusted p-values are 1.0.",
    "faster_fraction": "Every other model was faster than Clef on 57.8\u201359.1% of matched images, with significant image-level sign tests. No other pair was significant.",
    "dependence": "The Clef comparisons are not robust to all block sizes after conservative multiplicity adjustment. All four median-difference intervals include zero at 99.75% confidence with blocks of 50 or 100. Treat the image-level significance as conditional, exploratory evidence within this run, not a stable deployment or inference-speed ranking."
  }
}