Backtesting and benchmarks

motac now treats forecasting as a probabilistic task, not only a deterministic mean-field rollout.

Rolling-origin protocol

Backtests are run with rolling train/test folds:

1. fit on y[:, :train_end]

2. forecast horizon steps ahead with Monte Carlo paths

3. score on held-out y[:, train_end:train_end+horizon]

4. repeat with a configurable rolling step

This is implemented via:

• motac.eval.backtest_fit_forecast_nll

• motac.eval.run_backtest_report

Probabilistic forecasts

The model forecast path now supports sampling future count trajectories:

• forecast_count_paths_horizon returns sampled paths and latent intensity paths.

• forecast_probabilistic_horizon returns paths plus mean/quantile summaries.

Key outputs:

• forecast mean counts

• quantile envelopes (default q=(0.05, 0.5, 0.95))

• fold-level and aggregate metrics

Metrics

Backtests report:

• Negative log-likelihood (NLL)

• RMSE

• MAE

• empirical interval coverage

Baselines

Reports include simple benchmark baselines:

• last_value

• seasonal_naive

• moving_average

These provide context for model gains and avoid single-model performance claims.

Reproducible artifacts

run_backtest_report writes:

• report.json

• fold metric figure

• baseline comparison figure

The report structure is designed for machine-readable comparison across Chicago and ACLED benchmark runs.