Introducing the Arena Score

We built TS-Arena as the first live benchmark platform for time series forecasting. Running it over the first months has taught us a lot (not least that a live benchmark is a lot of maintenance work). We want to share what we got wrong and what we learned. The biggest change we introduce based on our learnings is the new Arena Score instead of ELO.

ELO? Not anymore

Live operation exposed at least four problems in our implementation of ELO:

  • A rating depended on how many rounds a model had played. Every model started at 1000 and moved a little per round. A new model stayed near the middle for a while.
  • A model's confidence interval could be narrow after only a few rounds. In the bootstrap, we reshuffled the rounds to remove ELO's path dependence. That in itself was right, but we should have resampled the rounds with replacement, not only reshuffled them, which in consequence could lead to too narrow intervals.
  • We ranked models from their first round on to speed up displaying new models. Looking back, that was too optimistic. After a single round, even a correct bootstrap shows no uncertainty. Now a model gets a rank only after 4 weeks.
  • In ELO each round was a win or a loss, so just a binary outcome. But in forecasting know actually also by how much a model won/loss. We can measure forecast quality much more precisely than, for example, the performance of LLMs. Just declaring a clear winner of every match felt like throwing away useful information. Winning by 0.1 % counted as much as winning by 50 %.

The Arena Score

For every series in every round, we compare each pair of models that forecast it. The outcome is a share of the pair's total error, err_B / (err_A + err_B): MAE 2 against MAE 3 scores 0.6 to 0.4. A Bradley–Terry model fitted to all these comparisons gives each model its score. The point ranking uses the MAE. The probabilistic ranking uses the quantile loss. Both cover every table and the full history. They are recomputed four times a day.

Simply taking plain Bradley–Terry would result again in a winner and a loser per match, so we adapted it. As the lead, Haya Halimeh played a key role in developing the concept. Many thanks to her! We thought about different approaches and changed the following:

  • graded outcomes
  • ratios of errors
  • every forecast counts once
  • no prior that pulls new models toward the middle
  • confidence intervals that treat forecasts close in time as linked

Advantages:

  • Difficult series do not drag down the score.
  • The scale of a series does not matter.
  • One failed forecast counts at most as one clear loss.
  • For other forecasts we respect the relative advantage of a model against the other
  • Changes in forecast difficulty over time (e.g. summer vs. winter) do not matter. We will keep an eye on this, though.
  • Models that join at different times are supported natively.

Drawbacks:

  • We may still not capture the difficulty of each forecast well.
  • It still takes some time before a new model gets a rank.
  • Let us know if you see others!

How to read the Arena Score

We decided to not scale the Arena Score into the same scale as the ELO score in order to differentiate both approaches better.

  • Higher is better. 0 is the naive forecast: the last observed value, repeated over the horizon. It takes part in every comparison internally. A negative score means a larger error than the naive forecast.
  • The score is 100 · ln(naive error / model error). Example: a model with 42 % lower error than the naive forecast scores 100 · ln(1 / 0.58) ≈ 54. A gap between two models reads the same way. For small gaps, 1 point is about 1 % lower error.
  • The 95 % confidence interval comes from a bootstrap. We redraw the history in blocks of 28 consecutive days and refit 200 times. Forecasts close in time are linked: same weather, overlapping horizons and so on. Redrawing single days would count them as independent and make the intervals too narrow. We lengthened the blocks in our TS-Arena history until the intervals stopped widening much, in our case 28 days.
  • The blocks are drawn from the whole history. So short records get a minimum width, taken from established models. With it, the intervals hold at about 95 % from the first week.
  • The rank is the place by score. Below it is the range of places the intervals allow. "1–7" means the model is somewhere between 1st and 7th.
  • A model with fewer than 4 weeks of results is provisional. It has a score and an interval, but no rank yet. A test on our data showed that 4 weeks give the first robust places. We will review this rule as more domains and series are evaluated. Your feedback is welcome.
  • To be ranked, a model must have forecast at least half of the series in each challenge's rounds since it joined. This replaces the July rule. It now applies to every table.

Formulas

For every pair of models A and B that forecast the same series in the same round:

outcome   y_AB = err_B / (err_A + err_B)
weight    w    = 1 / (k − 1)        k = models that forecast this series in this round

err is the MAE for the point ranking and the SQL for the probabilistic ranking. The Bradley–Terry fit gives each model a strength θ (σ is the logistic function). For every model A, its actual outcomes must match the outcomes its strength predicts:

fit       Σ w · y_AB  =  Σ w · σ(θ_A − θ_B)        for every model A

Both sums run over every comparison of A with another model B, in all series and rounds. All θ are solved together. This is the maximum of the weighted Bradley–Terry likelihood. The naive forecast gets its own θ in the same fit. The score is the gap to it:

score_A = 100 · (θ_A − θ_naive)

y_AB equals σ(ln err_B − ln err_A). A gap in θ is therefore a log error ratio.

The confidence interval is the wider of two ranges: a bootstrap that resamples 28-day blocks, and a minimum width that shrinks as a model plays more weeks. A model's best place counts only the models whose interval lies entirely above its own. Its worst place also counts every model whose interval overlaps. The details are in the code: arena_score.py.

MASE and per series comparison

Another change is regarding the additional metrics we report. Our initial MASE was scaled incorrectly. We divided by the error of a naive forecast on the evaluation window. Following Hyndman's best practice, as fev-bench and GIFT-Eval do, we now divide MASE and SQL by the in-sample error of a seasonal naive forecast on the context. In the end, only the denominator changes, but we are now consistent with them.

The bigger problem was that ELO worked on aggregated rounds. It compared models on their mean MASE per round, which we did to save compute. Each result therefore depended on which series we had put into which challenge, and how each was scaled. Unintentionally, our challenge definitions shaped the ELO. To reduce the influence of our design choices on the final score, the Arena Score works per series. Every forecast counts once, whatever challenge it belongs to. What still matters, of course, is which series enter TS-Arena, but we are aware of this and want to add more domains and more series. Sub-results are worth a look too, e.g. which model performs best on market prices. If you are interested in other or more granular evaluations, please let us know.

We also expected in the beginning that MASE and SQL would change over time. Some series might be easier to predict in summer than in winter or vice versa. That is also why we did not rely on an overall MASE alone but focused on ELO. Our data now shows a clear seasonal pattern in the errors. The field's typical MASE moves with the season. An average over a model's whole history says as much about when it joined as about how good it is. To account for this, the MASE and SQL columns next to the Arena Score show the geometric mean over the last 30 days. They are shown for models scored on at least 80 % of that period's series. This strenghtens again the argument that time series cross validation is important for TSFMs (we actually employ one in real-time right now).

Quantiles

We now fill in missing deciles by interpolation before scoring a forecast. Before, SQL averaged only the deciles sent. A 0.1/0.5/0.9 set looked about 20 % better than the full set. Past forecasts with a partial set have been rescored. Details are in the participation guide.

Pre-registration and the submission format are unchanged.

To be ranked, forecast at least half the series of every challenge. A new model needs four weeks. If you send quantiles, send at least all nine deciles.

We welcome your feedback and are happy to discuss. Open an issue at github.com/DAG-UPB/ts-arena or contact us.

Back to news