Jev, Clef, and Liquid AI for aerial image classification
Izzy Hurley15 minHave you ever stared out the window of a plane and tried to call out what you see? A patchwork of farmland, a gridlocked highway, or a winding river… your house? What is often a simple game for children or bored flyers is also a decades-old computer vision problem. From above, a park can resemble a forest, and a commercial district can look like a neighborhood. Interpreting aerial imagery also has military applications, including targeting, which makes understanding these systems’ capabilities and limitations especially important.
Decision models select from a defined set of answers, often returning probabilities alongside their choices. Given an image, they can choose “forest” or “farmland” rather than write a description, acting more like a traditional image classifier but without needing to be trained specifically for that task.
We wanted to explore how far decision models’ capabilities extend to this specialized visual task. A team processing a large collection of aerial images needs to consider cost and how much human review the predictions require. A time-sensitive application also needs answers quickly. Strong average accuracy helps, but we also need to know which scenes cause trouble, whether the probabilities could help identify mistakes, and what happens when images get distorted.
So, we tested Jev, Clef, GPT-6 Luna, Liquid AI’s D1, and D1 Ultra decision models on 2,000 labeled aerial images. All five models handled some scenes very easily and struggled with others. D1 Ultra delivered nearly the same overall accuracy as Jev at substantially lower cost. Jev’s confidence tracked its accuracy more closely overall, while D1 Ultra tended to underestimate itself. For both Liquid AI models, the high-confidence answers were almost always correct, but they were underconfident in many others.
Methodology
Jev and Clef appeared in our earlier eval of decision models for risk and harm classification, and Jev's behavior as a judge is covered in When Jev holds up as a judge. This eval looks into the same class of models but focuses on their multimodal capacity specifically on images taken from the sky.
Models tested
| Model | Provider |
|---|---|
| D1 | Liquid AI |
| D1 Ultra | Liquid AI |
| Jev 1.14 | TypeSafe |
| Clef | Cloudflare |
| GPT-6 Luna (Decisions API) | OpenAI |
Every model received the same 2,000 images from the AID dataset,[5] the same instructions including a short description of each of the 30 scene labels,[3] and the list of labels. For example, the Airport label description reads “Airport: An aviation facility with runways, taxiways, aprons, terminals, or parked aircraft.” We scored top-1 accuracy by agreement with the original dataset’s human labels.[6]
The label descriptions were refined after inspecting errors on this same 2,000-image test set, so the reported accuracies are optimistic for unseen data and need independent validation.
We stored the AID eval subset as a dataset in Braintrust and tracked latency, token usage, and API cost alongside each model’s predictions and scores.
A custom view generated by Loop shows each image alongside its telemetry and the probabilities of top labels.
Additionally, we ran experiments to measure how well the accuracy results held up when images were distorted. We applied blur, changes in contrast, and sideways shifts and compared each model’s accuracy with its results on the unmodified images.
Accuracy and API cost
Accuracy ranged from about 84% to 86%, while the most expensive model cost nearly 15 times as much as the cheapest.
| Model | Accuracy | API cost per 2,000 images |
|---|---|---|
| Jev | 86.15% | $0.2095 |
| D1 Ultra | 85.85% | $0.1209 |
| Luna | 85.55% | $0.2787 |
| D1 | 84.30% | $0.0576 |
| Clef | 83.85% | $0.8558 |
D1 Ultra came within 0.3 percentage points of Jev at 42% lower cost. D1 was the cheapest model. These API costs do not include human review.[1]
Most of these gaps are small enough that chance could explain them. Only Jev's lead over Clef was large enough to rule chance out as a likely explanation. The results do not show that the other models perform identically.[2] Luna refused one image, which counts as a wrong answer.[6]
D1, D1 Ultra, and Jev make up the Pareto frontier. No other model beats any of them on cost and accuracy at the same time. Luna and Clef were not competitive for their cost proposition, but Luna may still warrant consideration for workloads dominated by dense residential or playground imagery, where it achieved the highest observed accuracy.
Accuracy varies by category
Some scene types were easy for every model and a few were hard for every model. All five models missed this image, whose reference label is Church. Across the 48 church images, Jev correctly classified 11, compared with 86.15% accuracy overall.
| Model | Predicted label | Confidence in predicted label |
|---|---|---|
| Jev | Center | 68.69% |
| D1 Ultra | Center | 34.75% |
| Luna | Square | 27.00% |
| D1 | Center | 24.76% |
| Clef | Center | 22.13% |
Jev's overall calibration looks good, but on church images its confidence ran far ahead of its accuracy. On the 48 reference-labeled church images, it averaged 56.9% confidence and was right 22.9% of the time. D1 Ultra averaged 57.5% confidence and 56.3% accuracy on those same images. Checking confidence by scene category reveals weaknesses that the overall calibration score misses.
- All five models correctly classified every airport, stadium, and viaduct image. Thirteen categories reached at least 90% accuracy for every model.
- Church, meadow, commercial, and square stayed below 70% for every model. These categories warrant closer review before anyone relies on their labels.
- Luna correctly classified about 82% of dense residential images, compared with 43–61% for the other models. The chart does not show uncertainty around these rates.
Common mistakes included confusing dense and medium residential areas, and labeling centers as stadiums.
All five models missed the same 96 images. A preliminary review suggested that many were ambiguous or had questionable reference labels, including a highway interchange labeled as commercial. These judgments need independent review.[8]
These judgments have not been independently validated by a human reviewer and do not establish a dataset annotation-error rate.
Confidence and human review
How well each model's confidence matched its accuracy
Confidence is useful when it matches how often a model is right. Among predictions assigned about 90% confidence, a well-calibrated model should be correct about 90% of the time.
Jev's confidence matched its accuracy most closely overall. D1 Ultra tended to underestimate its accuracy. Its confidence averaged 78%, while it was right about 86% of the time.[4]
Mean confidence 86.1%, accuracy 86.15%
Mean confidence 78.0%, accuracy 85.85%
Mean confidence 89.9%, accuracy 85.55%
Mean confidence 79.6%, accuracy 84.30%
Mean confidence 73.0%, accuracy 83.85%
The table shows this difference among predictions with about 90% confidence. Jev was right about as often as its confidence suggested. D1 Ultra was right more often. Each model assigned this confidence to a different set of images.
| Model | Predictions with 85% to 95% confidence | Average confidence | Accuracy |
|---|---|---|---|
| D1 Ultra | 477 | 90.65% | 97.48% |
| Jev | 382 | 90.14% | 89.01% |
Keeping only the answers the model is sure about
Imagine a workflow where a model labels aerial images. Its most confident predictions are accepted automatically, while the rest go to a human reviewer. The confidence cutoff determines how much work you can automate and how many images still need review.
Among predictions with at least 90% confidence, D1 Ultra was 99.1% accurate. Only 39% of images met that cutoff. Jev retained more images with lower accuracy.
For D1 Ultra, that kept 781 of 2,000 images, with seven incorrect predictions, and left 1,219 needing review or another processing step.
Raising the confidence cutoff generally leaves fewer images to accept automatically. Each model keeps a different set of images, and the rest still need review. Test any cutoff on new images before using it in a deployment.
Additional experiments
Photos from recent flights
To try this ourselves, we sent all five models photos taken by our Braintrust teammates on recent flights. The gallery below shows their predictions side by side.
These personal photos are separate from the scored AID dataset.
Photo by Martin BergmanStorytellerResponse timing
Clef had the slowest typical response at about 5.2 seconds, compared with 4.6–4.7 seconds for the others. Further checks suggested that shared service load could explain the difference. This run did not establish a consistent speed ranking.[9]
The replay below shows one test image per scene category. Select Replay responses and each model's recorded answer appears at the time its API call took, with a running clock and the reference label visible so speed and correctness can be read together.
How blur, contrast, and shifts affected accuracy
Water droplets on a camera lens, clouds, movement in strong winds, and changing light can make aerial images harder to interpret. How well do the models handle images when the view is less clear or the framing changes?
To explore this, we applied three controlled changes to 300 images: blur, changes in contrast, and sideways shifts. We used the same prompt and label descriptions as the main eval. Each model classified every altered image and a fresh unmodified version. The sample included equal numbers from each scene category, so its overall accuracy can differ from the full test set.[10]
Shifting images left reduced Clef's accuracy from 84% to 77%. This was the only change that remained statistically significant after accounting for all the planned comparisons. Stronger blur reduced every model's observed accuracy, but those differences were too uncertain to establish an effect. The results do not rule out smaller effects from the other changes.
Use the explorer below to select an image and adjust its blur, contrast, or position. It shows the models' recorded predictions for each tested setting.
Follow-up experiment with local D1 models
For deployments that need to process images locally, we also tested two downloadable D1 model versions, known as checkpoints. D1-3B and D1-omni-600M processed the same 2,000 images with the same prompt and label descriptions as the hosted eval. Both ran one image at a time on an Apple M5 laptop with 32 GB of memory.
These model versions and their image processing differ from the hosted models. Local timing covers preparing the image and computing the answer after the model has loaded and warmed up. API timing also includes network and service delays, so these measurements do not isolate a speed advantage for local inference.[7]
The median is the time within which half the images finished; p95 is the time within which 95% finished.
| Local model | Images | Median per image | p95 per image | Accuracy |
|---|---|---|---|---|
| D1-3B, 8-bit | 2,000 | 634 ms | 1,201 ms | 83.3% |
| D1-omni-600M, float16 | 2,000 | 293 ms | 387 ms | 25.1% |
Both models returned valid predictions for every image. The smaller model finished faster but was substantially less accurate on this task.
Considerations for deployment
Start with the scene types your application needs to recognize and how many images your team can review. D1 and D1 Ultra are worth testing when API cost is a priority. Jev is worth testing when you need confidence scores that closely track accuracy. Compare them on new examples from your own workload before choosing.
Include difficult categories such as church, meadow, commercial, and square in that check. Review mistakes and ambiguous labels, then test how changes in your descriptions and criteria change results. Make sure your dataset includes adversarial and confusing examples that reflect real-world challenges. If you use confidence to decide which predictions need review, check its reliability within each category as well as overall.
You can run an image classification eval like this one, attach the images to each row, record probabilities alongside accuracy and cost, and check calibration by category before trusting a confidence threshold. Sign up for free to run it on your own models, or book a demo to walk through your setup.
- 1.Costs use Liquid AI's reported API charges for D1 and D1 Ultra and input-token price estimates for the other providers. They exclude Braintrust charges and any retry charges that providers did not report.
- 2.Accuracy comparisons use paired McNemar tests with Holm correction across the ten model pairs. Only Jev's lead over Clef remained significant after correction. Non-significant differences do not establish that two models perform equivalently.
- 3.Mai et al. (2024), section 3.4 found that vague AID class names hurt zero-shot classification and that clarifying the Center and Commercial classes improved accuracy. All models received the same short label descriptions, using published distinctions where available and documented working interpretations elsewhere. The label descriptions were refined after inspecting errors on this same 2,000-image test set, so the reported accuracies are optimistic for unseen data and need independent validation. AID is publicly available, and training-data overlap with any of the evaluated models cannot be ruled out.
- 4.Calibration uses the probability assigned to the chosen label rather than any separate confidence field in the API response. Confidence and accuracy are compared in ten equally spaced bins, weighted by image count. Responses that returned no probabilities are excluded from the calibration analysis. In the calibration chart, each marker is one confidence bin and the vertical line through it is the 95% Wilson interval for that bin's accuracy. Markers with a white fill are bins with fewer than 20 images, which here means between 1 and 11, so their positions are too noisy to read closely.
- 5.AID is a dataset of 10,000 labeled Google Earth images across 30 scene categories at 600 × 600 pixels. The evaluation split is the 2,000-image test set from a fixed community split of 6,000 training, 2,000 validation, and 2,000 test images, which differs from the benchmark protocol in the original AID paper. Provider-side image preprocessing can differ. The window-seat photos in the gallery were resized so the longest side is 600 pixels before being sent.
- 6.Scoring is top-1 accuracy across all attempts. API failures and invalid outputs count as incorrect, and the valid-output rate is reported separately. Accuracy and coverage at a 90% confidence threshold are reported alongside the calibration analysis.
- 7.Local runs used the same frozen 559-word prompt containing the label descriptions as the hosted experiment on an Apple M5 laptop with 32 GB of memory. D1-omni-600M used float16 MLX weights, and D1-3B used an 8-bit MLX checkpoint. Each runner processed one image at a time with batch size one. Timing starts with encoded image bytes in memory and includes decoding, native preprocessing, model computation, probability validation and normalization, and GPU synchronization. Model loading and five warmup calls per session are excluded. D1-3B completed in two sessions, with the final 78 images resumed using unchanged weights, runtime, and prompt. The initial session overlapped with other local runs; the resumed session started on AC power and ended on battery power. Latency summarizes these observed conditions. Local checkpoints and image processors differ from the hosted models.
- 8.The follow-up review covered all 96 shared failures using 380-pixel thumbnails, with reference labels and model predictions visible. All five models chose the same wrong label on 71 of these images. The preliminary review judged the models' labels more plausible than the reference on 18 images, found 67 ambiguous, and identified 11 apparent model errors. These judgments have not been independently validated by a human reviewer and do not establish a dataset annotation-error rate.
- 9.Latency comparisons match all 2,000 images across models, including retries and the refusal. We ran two-sided paired permutation tests for median and 95th-percentile differences with 19,999 label swaps, plus exact sign tests of which model was faster per image, and applied Holm correction jointly across all 30 comparisons. Each Clef median comparison had a Monte Carlo adjusted p-value of 0.0026; no other median or 95th-percentile comparison was significant. Paired bootstrap intervals preserve scene-category counts. Sensitivity checks resampling consecutive groups of 20, 50, and 100 requests in Jev completion order produced wider intervals; all four Clef median comparisons included zero with groups of 50 or 100 at the multiplicity-adjusted confidence level. Completion order is only a proxy for time because the local exports lack request timestamps. These exploratory results come from one run with up to six concurrent requests per model and include network, service, parsing, and retry overhead; they do not isolate model inference speed or establish a ranking across deployments. Full latency analysis.
- 10.The controlled-definitions-v1 experiment used the same frozen 559-word prompt and label descriptions as the main eval. All five models made fresh calls on 300 images, 10 per category, under seven conditions: an unmodified PNG control, Gaussian blur at radii of 1 and 2 pixels, contrast factors of 0.5 and 1.5, and horizontal shifts of 60 pixels left and right with reflected fill. Each model returned 2,100 responses; Luna's one refusal counts as incorrect. Accuracy comparisons use paired tests with Holm correction across 90 planned comparisons. Clef's left-shift effect had an adjusted p-value of 0.0093 and a pointwise 95% bootstrap interval of −10.3 to −3.7 percentage points. Calibration uses only source images with valid responses in all seven conditions, with ten equal-width confidence bins. Small changes include ordinary model variability; shifts also crop and reflect image content. The balanced sample's accuracy is not directly comparable to the full 2,000-image results. Full distortion analysis, intervals, and run verification.
Subscribe to the Department of Evals
A newsletter for unfiltered thoughts on eval methodology, analysis, and failures
Subscribe