Earth observation has no shortage of models or datasets, but it has a shortage of comparable results. New architectures are routinely evaluated under inconsistent splits, loss functions and training regimes, and when the same models are re-run under a single controlled protocol, much of the reported progress does not survive. This is a problem for the field generally, and a blocking one for AutoML: meta-learning, performance modelling and automated pipeline design all depend on an empirical record that can be trusted and queried.
    In this talk I will argue that benchmarking and open science infrastructure are two halves of the same problem, and describe our work on both. On the benchmarking side, I will present results from the AiTLAS Benchmark Arena (500+ models across 22 classification datasets) and a recent change detection benchmark (10 architectures across 10 heterogeneous datasets), where well-optimized Siamese U-Nets remain competitive with transformers and state-space models once computational cost is accounted for. On the infrastructure side, I will describe FAIR-EO and the FAIR-EO Hub: an ontology-based schema that describes datasets, methods, runs and evaluation results as linked data, making benchmark outcomes machine-actionable rather than merely human-readable.