Benchmarking skforecast¶
Benchmarking is an essential part of the development process of skforecast. It provides a transparent view of how the performance of the library evolves across versions and helps users make informed decisions when choosing the right forecaster or configuration for their use case.
In this section, we present benchmark results that measure the execution time of key methods (fit, predict, etc.) and their performance evolution across versions, allowing the detection of improvements or regressions.
Methodology¶
The benchmarking results presented here are generated using a custom benchmarking script located in the benchmarks/ directory of the repository. This script executes a series of performance tests for the main forecasting classes and their methods, recording both execution time and variability across multiple runs.
To ensure consistency and reproducibility, all benchmarks are automatically executed as part of the continuous integration (CI) pipeline using GitHub Actions. This guarantees that each new release of the library is tested under the same environment and dependency configuration, making performance results directly comparable across versions.
To reduce the measurement noise introduced by short-running methods on shared, virtualized CI runners, the fast methods use a small number of warmup iterations (discarded from the timing) and the single maximum observation is trimmed before computing the reported statistics (only when at least 10 timings are available). Only the maximum is removed because measurement noise on shared runners is one-sided (it can only make a call slower), so the fastest observation is kept as the cleanest sample. Note that these robustness steps were introduced at a specific version, so the reported run_time_std decreases by construction from that point onward; this is a change in how the statistic is computed, not a performance change in skforecast. Timings also depend on the pinned dependency stack (for example, numpy), so changes in those dependencies can shift the baseline independently of skforecast.
Users who wish to reproduce the benchmarks locally can execute the same script by running:
# From skforecast root directory
python benchmarks/run_benchmarks.py
Results are stored in a structured format in benchmarks/benchmarks.joblib.
⚠ Warning
The plot_benchmark_results function is located at the end of the notebook. Go to plot function.
In [5]:
Copied!
# Libraries
# ==============================================================================
import platform
import psutil
import numpy as np
import pandas as pd
import joblib
import sklearn
import skforecast
# Display all columns in pandas
pd.set_option("display.max_columns", None)
# Libraries
# ==============================================================================
import platform
import psutil
import numpy as np
import pandas as pd
import joblib
import sklearn
import skforecast
# Display all columns in pandas
pd.set_option("display.max_columns", None)
In [6]:
Copied!
# Environment information
# ==============================================================================
print(f"Python version : {platform.python_version()}")
print(f"skforecast version : {skforecast.__version__}")
print(f"numpy version : {np.__version__}")
print(f"pandas version : {pd.__version__}")
print(f"scikit-learn version : {sklearn.__version__}")
print(f"Computer network name : {platform.node()}")
print(f"Processor type : {platform.processor()}")
print(f"Platform type : {platform.platform()}")
print(f"Number of physical cores : {psutil.cpu_count(logical=False)}")
print(f"Number of logical cores : {psutil.cpu_count(logical=True)}")
print(f"Memory total : {round(psutil.virtual_memory().total / 1e9, 2)} GB")
# Environment information
# ==============================================================================
print(f"Python version : {platform.python_version()}")
print(f"skforecast version : {skforecast.__version__}")
print(f"numpy version : {np.__version__}")
print(f"pandas version : {pd.__version__}")
print(f"scikit-learn version : {sklearn.__version__}")
print(f"Computer network name : {platform.node()}")
print(f"Processor type : {platform.processor()}")
print(f"Platform type : {platform.platform()}")
print(f"Number of physical cores : {psutil.cpu_count(logical=False)}")
print(f"Number of logical cores : {psutil.cpu_count(logical=True)}")
print(f"Memory total : {round(psutil.virtual_memory().total / 1e9, 2)} GB")
Python version : 3.14.3 skforecast version : 0.25.0 numpy version : 2.5.3 pandas version : 2.3.3 scikit-learn version : 1.9.1 Computer network name : ES-G19C3FYX9Y Processor type : arm Platform type : macOS-26.6.2-arm64-arm-64bit-Mach-O Number of physical cores : 10 Number of logical cores : 10 Memory total : 17.18 GB
In [7]:
Copied!
import warnings
warnings.filterwarnings(
"ignore",
category=FutureWarning,
message="'force_all_finite' was renamed to 'ensure_all_finite'"
)
import warnings
warnings.filterwarnings(
"ignore",
category=FutureWarning,
message="'force_all_finite' was renamed to 'ensure_all_finite'"
)
Global results¶
In [8]:
Copied!
# Load benchmark results
# ==============================================================================
results_benchmark_all = joblib.load("../../benchmarks/benchmark.joblib")
versions_to_exclude = ["0.15.1"]
results_benchmark_all = results_benchmark_all.query("~skforecast_version.isin(@versions_to_exclude)")
print("Shape:", results_benchmark_all.shape)
results_benchmark_all.tail(2)
# Load benchmark results
# ==============================================================================
results_benchmark_all = joblib.load("../../benchmarks/benchmark.joblib")
versions_to_exclude = ["0.15.1"]
results_benchmark_all = results_benchmark_all.query("~skforecast_version.isin(@versions_to_exclude)")
print("Shape:", results_benchmark_all.shape)
results_benchmark_all.tail(2)
Shape: (4527, 23)
Out[8]:
| run_id | forecaster_name | estimator_name | function_name | function_hash | method_name | run_time_avg | run_time_median | run_time_p95 | run_time_std | n_repeats | datetime | python_version | skforecast_version | numpy_version | pandas_version | sklearn_version | lightgbm_version | platform | processor | cpu_count | memory_gb | cpu_model | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 4685 | 20260911_d99de6 | ForecasterDirectMultiVariate | DummyRegressor | ForecasterDirectMultiVariate_backtesting_no_exog | ddb15f106bc905b52fb0b2b6560fdbac | backtesting_no_exog | 0.116042 | 0.115658 | 0.116921 | 0.000694 | 5 | 2026-09-11 11:55:40.843659 | 3.12.14 | 0.25.0 | 2.4.6 | 2.3.3 | 1.9.0 | 4.6.0 | Linux-6.17.0-1022-azure-x86_64-with-glibc2.39 | x86_64 | 4 | 16.77 | AMD EPYC 7763 64-Core Processor |
| 4686 | 20260911_d99de6 | ForecasterDirectMultiVariate | DummyRegressor | ForecasterDirectMultiVariate_backtesting_confo... | 97312a4aabed30dfcee4ceef017042e5 | backtesting_conformal | 0.138733 | 0.138820 | 0.139904 | 0.000828 | 5 | 2026-09-11 11:55:41.445229 | 3.12.14 | 0.25.0 | 2.4.6 | 2.3.3 | 1.9.0 | 4.6.0 | Linux-6.17.0-1022-azure-x86_64-with-glibc2.39 | x86_64 | 4 | 16.77 | AMD EPYC 7763 64-Core Processor |
In [9]:
Copied!
# Filter only a specific type of CPU
# ==============================================================================
# cpu_types = [
# 'AMD EPYC 9V45 96-Core Processor'
# ]
# results_benchmark_all = results_benchmark_all.query("cpu_model.isin(@cpu_types)")
# Filter only a specific type of CPU
# ==============================================================================
# cpu_types = [
# 'AMD EPYC 9V45 96-Core Processor'
# ]
# results_benchmark_all = results_benchmark_all.query("cpu_model.isin(@cpu_types)")
Regression summary: latest vs previous version¶
Before drilling into the per-forecaster plots below, this table gives a single overview of which functions got significantly slower (or faster) in the most recently recorded version compared to the version right before it. By default, only changes of at least 5% are shown, and each version's time is the median across all its CI runs (individual runs can differ noticeably due to host-to-host noise on shared runners). Use the per-forecaster sections further down to inspect a flagged function across its full version history.
In [10]:
Copied!
# Regression summary function
# ==============================================================================
from packaging.version import Version, InvalidVersion
def summarize_version_regressions(
df: pd.DataFrame,
baseline_version: str | None = None,
target_version: str | None = None,
threshold: float = 0.05,
only_significant: bool = True,
):
"""
Compare the median execution time per function between two skforecast
versions and highlight the largest relative changes.
For each version, the median of `run_time_avg` across all `run_id`s is
used as the representative time. A single CI run can be noticeably slower
or faster purely from host-to-host noise on shared runners, so comparing
single rows instead of this per-version median would produce false
regressions.
Parameters
----------
df : pandas DataFrame
Benchmark results, as loaded from `benchmarks/benchmark.joblib`.
baseline_version : str, default None
Version to compare against. If `None`, the version immediately
preceding `target_version` (by parsed semantic version) is used.
target_version : str, default None
Version being evaluated. If `None`, the newest version present in
`df` is used.
threshold : float, default 0.05
Relative change (e.g. 0.05 = 5%) above which a function is
considered a regression, and below `-threshold` an improvement.
only_significant : bool, default True
If `True`, only rows with `abs(pct_change) >= threshold` are kept.
Returns
-------
pandas.io.formats.style.Styler
Table sorted by `pct_change` (largest regressions first), colored
with a diverging scale centered at 0 (red = slower, green = faster).
"""
def try_version(v):
try:
return Version(str(v).lstrip('v'))
except InvalidVersion:
return Version("0")
versions_sorted = sorted(df['skforecast_version'].unique(), key=try_version)
if target_version is None:
target_version = versions_sorted[-1]
if baseline_version is None:
idx = versions_sorted.index(target_version)
if idx == 0:
raise ValueError(
f"'{target_version}' is the oldest version in `df`; "
"there is no earlier version to compare against."
)
baseline_version = versions_sorted[idx - 1]
medians = (
df[df['skforecast_version'].isin([baseline_version, target_version])]
.groupby(
['function_name', 'forecaster_name', 'method_name', 'skforecast_version'],
observed=True
)
.agg(time=('run_time_avg', 'median'), n_runs=('run_id', 'nunique'))
.reset_index()
)
baseline = medians[medians['skforecast_version'] == baseline_version]
target = medians[medians['skforecast_version'] == target_version]
merged = baseline.merge(
target,
on=['function_name', 'forecaster_name', 'method_name'],
how='outer',
suffixes=('_baseline', '_target'),
indicator=True,
)
only_baseline = merged.loc[merged['_merge'] == 'left_only', 'function_name'].tolist()
only_target = merged.loc[merged['_merge'] == 'right_only', 'function_name'].tolist()
if only_baseline:
print(f"Only in {baseline_version} (removed/renamed in {target_version}): {only_baseline}")
if only_target:
print(f"New in {target_version} (absent in {baseline_version}): {only_target}")
common = merged[merged['_merge'] == 'both'].copy()
common['pct_change'] = (common['time_target'] - common['time_baseline']) / common['time_baseline']
common = common.sort_values('pct_change', ascending=False)
n_regressed = int((common['pct_change'] >= threshold).sum())
n_improved = int((common['pct_change'] <= -threshold).sum())
print(
f"Compared {len(common)} functions: {baseline_version} -> {target_version}.\n"
f"Note: Negative pct_change = improvement (faster), Positive = regression (slower).\n"
f"Threshold: {threshold:.0%}.\n"
f"{n_regressed} regressed, {n_improved} improved."
)
if only_significant:
common = common[common['pct_change'].abs() >= threshold]
display_df = pd.DataFrame({
'forecaster_name': common['forecaster_name'],
'method_name': common['method_name'],
'baseline_version': baseline_version,
'baseline_time': common['time_baseline'],
'target_version': target_version,
'target_time': common['time_target'],
'pct_change': common['pct_change'],
'n_runs_baseline': common['n_runs_baseline'],
'n_runs_target': common['n_runs_target'],
})
max_abs = display_df['pct_change'].abs().max() if not display_df.empty else threshold
return (
display_df.style
.background_gradient(cmap='RdYlGn_r', subset=['pct_change'], vmin=-max_abs, vmax=max_abs)
.format({
'baseline_time': '{:.6f}',
'target_time': '{:.6f}',
'pct_change': '{:+.1%}',
})
.hide(axis='index')
)
# Regression summary function
# ==============================================================================
from packaging.version import Version, InvalidVersion
def summarize_version_regressions(
df: pd.DataFrame,
baseline_version: str | None = None,
target_version: str | None = None,
threshold: float = 0.05,
only_significant: bool = True,
):
"""
Compare the median execution time per function between two skforecast
versions and highlight the largest relative changes.
For each version, the median of `run_time_avg` across all `run_id`s is
used as the representative time. A single CI run can be noticeably slower
or faster purely from host-to-host noise on shared runners, so comparing
single rows instead of this per-version median would produce false
regressions.
Parameters
----------
df : pandas DataFrame
Benchmark results, as loaded from `benchmarks/benchmark.joblib`.
baseline_version : str, default None
Version to compare against. If `None`, the version immediately
preceding `target_version` (by parsed semantic version) is used.
target_version : str, default None
Version being evaluated. If `None`, the newest version present in
`df` is used.
threshold : float, default 0.05
Relative change (e.g. 0.05 = 5%) above which a function is
considered a regression, and below `-threshold` an improvement.
only_significant : bool, default True
If `True`, only rows with `abs(pct_change) >= threshold` are kept.
Returns
-------
pandas.io.formats.style.Styler
Table sorted by `pct_change` (largest regressions first), colored
with a diverging scale centered at 0 (red = slower, green = faster).
"""
def try_version(v):
try:
return Version(str(v).lstrip('v'))
except InvalidVersion:
return Version("0")
versions_sorted = sorted(df['skforecast_version'].unique(), key=try_version)
if target_version is None:
target_version = versions_sorted[-1]
if baseline_version is None:
idx = versions_sorted.index(target_version)
if idx == 0:
raise ValueError(
f"'{target_version}' is the oldest version in `df`; "
"there is no earlier version to compare against."
)
baseline_version = versions_sorted[idx - 1]
medians = (
df[df['skforecast_version'].isin([baseline_version, target_version])]
.groupby(
['function_name', 'forecaster_name', 'method_name', 'skforecast_version'],
observed=True
)
.agg(time=('run_time_avg', 'median'), n_runs=('run_id', 'nunique'))
.reset_index()
)
baseline = medians[medians['skforecast_version'] == baseline_version]
target = medians[medians['skforecast_version'] == target_version]
merged = baseline.merge(
target,
on=['function_name', 'forecaster_name', 'method_name'],
how='outer',
suffixes=('_baseline', '_target'),
indicator=True,
)
only_baseline = merged.loc[merged['_merge'] == 'left_only', 'function_name'].tolist()
only_target = merged.loc[merged['_merge'] == 'right_only', 'function_name'].tolist()
if only_baseline:
print(f"Only in {baseline_version} (removed/renamed in {target_version}): {only_baseline}")
if only_target:
print(f"New in {target_version} (absent in {baseline_version}): {only_target}")
common = merged[merged['_merge'] == 'both'].copy()
common['pct_change'] = (common['time_target'] - common['time_baseline']) / common['time_baseline']
common = common.sort_values('pct_change', ascending=False)
n_regressed = int((common['pct_change'] >= threshold).sum())
n_improved = int((common['pct_change'] <= -threshold).sum())
print(
f"Compared {len(common)} functions: {baseline_version} -> {target_version}.\n"
f"Note: Negative pct_change = improvement (faster), Positive = regression (slower).\n"
f"Threshold: {threshold:.0%}.\n"
f"{n_regressed} regressed, {n_improved} improved."
)
if only_significant:
common = common[common['pct_change'].abs() >= threshold]
display_df = pd.DataFrame({
'forecaster_name': common['forecaster_name'],
'method_name': common['method_name'],
'baseline_version': baseline_version,
'baseline_time': common['time_baseline'],
'target_version': target_version,
'target_time': common['time_target'],
'pct_change': common['pct_change'],
'n_runs_baseline': common['n_runs_baseline'],
'n_runs_target': common['n_runs_target'],
})
max_abs = display_df['pct_change'].abs().max() if not display_df.empty else threshold
return (
display_df.style
.background_gradient(cmap='RdYlGn_r', subset=['pct_change'], vmin=-max_abs, vmax=max_abs)
.format({
'baseline_time': '{:.6f}',
'target_time': '{:.6f}',
'pct_change': '{:+.1%}',
})
.hide(axis='index')
)
In [11]:
Copied!
# Regression summary: latest vs previous version
# ==============================================================================
summarize_version_regressions(results_benchmark_all)
# Regression summary: latest vs previous version
# ==============================================================================
summarize_version_regressions(results_benchmark_all)
Compared 75 functions: 0.24.0 -> 0.25.0. Note: Negative pct_change = improvement (faster), Positive = regression (slower). Threshold: 5%. 5 regressed, 23 improved.
Out[11]:
| forecaster_name | method_name | baseline_version | baseline_time | target_version | target_time | pct_change | n_runs_baseline | n_runs_target |
|---|---|---|---|---|---|---|---|---|
| ForecasterDirectMultiVariate | backtesting | 0.24.0 | 0.109997 | 0.25.0 | 0.127983 | +16.4% | 18 | 5 |
| ForecasterDirect | backtesting | 0.24.0 | 0.159952 | 0.25.0 | 0.182420 | +14.0% | 18 | 5 |
| ForecasterRecursiveMultiSeries | predict_bootstrapping_exog_is_dict | 0.24.0 | 3.098099 | 0.25.0 | 3.296551 | +6.4% | 18 | 5 |
| ForecasterRecursiveMultiSeries | _create_train_X_y_series_is_dict_exog_is_df_wide | 0.24.0 | 1.117295 | 0.25.0 | 1.180333 | +5.6% | 18 | 5 |
| ForecasterRecursiveMultiSeries | predict_exog_is_df_long | 0.24.0 | 0.380362 | 0.25.0 | 0.400019 | +5.2% | 18 | 5 |
| ForecasterDirectMultiVariate | _create_train_X_y_no_exog | 0.24.0 | 0.008392 | 0.25.0 | 0.007963 | -5.1% | 18 | 5 |
| ForecasterStats | backtesting_interval_exog | 0.24.0 | 0.361360 | 0.25.0 | 0.341534 | -5.5% | 18 | 5 |
| ForecasterRecursive | _create_train_X_y | 0.24.0 | 0.003273 | 0.25.0 | 0.003087 | -5.7% | 18 | 5 |
| ForecasterRecursive | fit | 0.24.0 | 0.004222 | 0.25.0 | 0.003975 | -5.8% | 18 | 5 |
| ForecasterRecursive | backtesting_conformal | 0.24.0 | 0.050398 | 0.25.0 | 0.047432 | -5.9% | 18 | 5 |
| ForecasterRecursive | _create_predict_inputs | 0.24.0 | 0.001310 | 0.25.0 | 0.001232 | -5.9% | 18 | 5 |
| ForecasterStats | backtesting_exog | 0.24.0 | 0.360299 | 0.25.0 | 0.337955 | -6.2% | 18 | 5 |
| ForecasterDirectMultiVariate | fit | 0.24.0 | 0.016601 | 0.25.0 | 0.015567 | -6.2% | 18 | 5 |
| ForecasterRecursiveMultiSeries | fit_series_is_dict_no_exog | 0.24.0 | 1.160973 | 0.25.0 | 1.084838 | -6.6% | 18 | 5 |
| ForecasterRecursiveMultiSeries | _create_predict_inputs_exog_is_df_wide | 0.24.0 | 0.105613 | 0.25.0 | 0.098370 | -6.9% | 18 | 5 |
| ForecasterDirectMultiVariate | _create_train_X_y | 0.24.0 | 0.010670 | 0.25.0 | 0.009932 | -6.9% | 18 | 5 |
| ForecasterRecursiveMultiSeries | check_predict_inputs | 0.24.0 | 0.051620 | 0.25.0 | 0.047878 | -7.3% | 18 | 5 |
| ForecasterRecursive | backtesting | 0.24.0 | 0.045286 | 0.25.0 | 0.041876 | -7.5% | 18 | 5 |
| ForecasterDirect | backtesting_conformal | 0.24.0 | 0.208994 | 0.25.0 | 0.191806 | -8.2% | 18 | 5 |
| ForecasterDirectMultiVariate | _create_lags | 0.24.0 | 0.000027 | 0.25.0 | 0.000025 | -8.3% | 18 | 5 |
| ForecasterRecursiveMultiSeries | fit_series_is_dataframe_no_exog | 0.24.0 | 1.359040 | 0.25.0 | 1.242475 | -8.6% | 18 | 5 |
| ForecasterRecursiveMultiSeries | predict_exog_is_dict | 0.24.0 | 0.272479 | 0.25.0 | 0.247170 | -9.3% | 18 | 5 |
| ForecasterRecursiveMultiSeries | _create_train_X_y_series_is_df_long_no_exog | 0.24.0 | 0.884736 | 0.25.0 | 0.801124 | -9.5% | 18 | 5 |
| ForecasterRecursiveMultiSeries | _create_train_X_y_series_is_dict_no_exog | 0.24.0 | 0.691092 | 0.25.0 | 0.619777 | -10.3% | 18 | 5 |
| ForecasterRecursiveClassifier | check_predict_inputs | 0.24.0 | 0.000148 | 0.25.0 | 0.000132 | -10.9% | 18 | 5 |
| ForecasterRecursive | check_predict_inputs | 0.24.0 | 0.000144 | 0.25.0 | 0.000127 | -11.3% | 18 | 5 |
| ForecasterRecursiveMultiSeries | _create_lags | 0.24.0 | 0.000074 | 0.25.0 | 0.000062 | -16.5% | 18 | 5 |
| ForecasterRecursiveClassifier | _create_lags | 0.24.0 | 0.000019 | 0.25.0 | 0.000015 | -21.4% | 18 | 5 |
ForecasterRecursive¶
The ForecasterRecursive class is benchmarked under a fixed experimental setup to ensure fair comparison across library versions. Key conditions are:
| Condition | Value |
|---|---|
| Estimator | sklearn.dummy.DummyRegressor (to isolate forecaster overhead) |
| Dataset length | 2000 synthetic observations (generated with bench_forecaster_recursive._make_data) |
| Lags as predictors | 50 |
| Exogenous features | 3 |
| Data transformation | StandardScaler() for time series features and exogenous variables |
| Prediction horizon | 100 steps ahead |
| Backtesting split | 1200 training, remaining for testing, step size = 50, no re-fitting (see guide) |
Note: In versions < 0.14.0, ForecasterRecursive was named ForecasterAutoreg.
In [12]:
Copied!
# Plot benchmark results
# ==============================================================================
plot_benchmark_results(
results_benchmark_all,
forecaster_names = ['ForecasterRecursive', 'ForecasterAutoreg'],
estimators = ['DummyRegressor', 'LinearRegression'],
add_mean = True,
add_median = True
)
# Plot benchmark results
# ==============================================================================
plot_benchmark_results(
results_benchmark_all,
forecaster_names = ['ForecasterRecursive', 'ForecasterAutoreg'],
estimators = ['DummyRegressor', 'LinearRegression'],
add_mean = True,
add_median = True
)
ForecasterDirect¶
The ForecasterDirect class is benchmarked under a fixed experimental setup to ensure fair comparison across library versions. Key conditions are:
| Condition | Value |
|---|---|
| Estimator | sklearn.dummy.DummyRegressor (to isolate forecaster overhead) |
| Dataset length | 2000 synthetic observations (generated with bench_forecaster_direct._make_data) |
| Lags as predictors | 20 |
| Exogenous features | 3 |
| Data transformation | StandardScaler() for time series features and exogenous variables |
| Prediction horizon | 10 steps ahead (10 estimators) |
| Backtesting split | 1200 training, remaining for testing, step size = 10, no re-fitting (see guide) |
Note: In versions < 0.14.0, ForecasterDirect was named ForecasterAutoregDirect.
In [13]:
Copied!
# Plot benchmark results
# ==============================================================================
plot_benchmark_results(
results_benchmark_all,
forecaster_names = ['ForecasterDirect', 'ForecasterAutoregDirect'],
estimators = ['DummyRegressor'],
add_mean = True,
add_median = True
)
# Plot benchmark results
# ==============================================================================
plot_benchmark_results(
results_benchmark_all,
forecaster_names = ['ForecasterDirect', 'ForecasterAutoregDirect'],
estimators = ['DummyRegressor'],
add_mean = True,
add_median = True
)
ForecasterRecursiveClassifier¶
The ForecasterRecursiveClassifier class is benchmarked under a fixed experimental setup to ensure fair comparison across library versions. Key conditions are:
| Condition | Value |
|---|---|
| Estimator | sklearn.dummy.DummyClassifier (to isolate forecaster overhead) |
| Dataset length | 2000 synthetic observations (generated with bench_forecaster_recursive_classifier._make_data) |
| Lags as predictors | 50 |
| Exogenous features | 3 |
| Data transformation | StandardScaler() for exogenous variables |
| Prediction horizon | 100 steps ahead |
| Backtesting split | 1200 training, remaining for testing, step size = 50, no re-fitting (see guide) |
In [14]:
Copied!
# Plot benchmark results
# ==============================================================================
plot_benchmark_results(
results_benchmark_all,
forecaster_names = ['ForecasterRecursiveClassifier'],
estimators = ['DummyClassifier'],
add_mean = True,
add_median = True
)
# Plot benchmark results
# ==============================================================================
plot_benchmark_results(
results_benchmark_all,
forecaster_names = ['ForecasterRecursiveClassifier'],
estimators = ['DummyClassifier'],
add_mean = True,
add_median = True
)
ForecasterStats¶
The ForecasterStats class is benchmarked under a fixed experimental setup to ensure fair comparison across library versions. Key conditions are:
| Condition | Value |
|---|---|
| Estimator | skforecast.stats.Arima |
| Dataset length | 500 synthetic observations (generated with bench_forecaster_stats._make_data) |
| Exogenous features | 3 |
| Data transformation | StandardScaler() for time series features and exogenous variables |
| Prediction horizon | 24 steps ahead |
| Backtesting split | 300 training, remaining for testing, step size = 24 with re-fitting (see guide) |
In [15]:
Copied!
# Plot benchmark results
# ==============================================================================
plot_benchmark_results(
results_benchmark_all,
forecaster_names = ['ForecasterStats'],
estimators = ['skforecast.Arima'],
add_mean = True,
add_median = True
)
# Plot benchmark results
# ==============================================================================
plot_benchmark_results(
results_benchmark_all,
forecaster_names = ['ForecasterStats'],
estimators = ['skforecast.Arima'],
add_mean = True,
add_median = True
)
ForecasterRecursiveMultiSeries¶
The ForecasterRecursiveMultiSeries class is benchmarked under a fixed experimental setup to ensure fair comparison across library versions. Key conditions are:
| Condition | Value |
|---|---|
| Estimator | sklearn.dummy.DummyRegressor (to isolate forecaster overhead) |
| Number of series | 600 time series |
| Dataset length | 2000 synthetic observations per series (generated with bench_forecaster_recursive_multiseries._make_data) |
| Lags as predictors | 50 |
| Exogenous features | 3 |
| Data transformation | StandardScaler() for time series features and exogenous variables |
| Prediction horizon | 100 steps ahead |
| Backtesting split | 1200 training, remaining for testing, step size = 50, no re-fitting (see guide) |
Note: In versions < 0.14.0, ForecasterRecursiveMultiSeries was named ForecasterAutoregMultiSeries.
In [16]:
Copied!
# Plot benchmark results
# ==============================================================================
plot_benchmark_results(
results_benchmark_all,
forecaster_names = ['ForecasterRecursiveMultiSeries', 'ForecasterAutoregMultiSeries'],
estimators = ['DummyRegressor', 'LinearRegression'],
add_mean = True,
add_median = True
)
# Plot benchmark results
# ==============================================================================
plot_benchmark_results(
results_benchmark_all,
forecaster_names = ['ForecasterRecursiveMultiSeries', 'ForecasterAutoregMultiSeries'],
estimators = ['DummyRegressor', 'LinearRegression'],
add_mean = True,
add_median = True
)
ForecasterDirectMultiVariate¶
The ForecasterDirectMultiVariate class is benchmarked under a fixed experimental setup to ensure fair comparison across library versions. Key conditions are:
| Condition | Value |
|---|---|
| Estimator | sklearn.dummy.DummyRegressor (to isolate forecaster overhead) |
| Number of series | 13 time series |
| Dataset length | 1000 synthetic observations (generated with bench_forecaster_direct_multivariate._make_data) |
| Lags as predictors | 20 |
| Exogenous features | 3 |
| Data transformation | StandardScaler() for time series features and exogenous variables |
| Prediction horizon | 10 steps ahead (10 estimators) |
| Backtesting split | 900 training, remaining for testing, step size = 10, no re-fitting (see guide) |
Note: In versions < 0.14.0, ForecasterDirectMultiVariate was named ForecasterAutoregMultiVariate.
In [17]:
Copied!
# Plot benchmark results
# ==============================================================================
plot_benchmark_results(
results_benchmark_all,
forecaster_names = ['ForecasterDirectMultiVariate', 'ForecasterAutoregMultiVariate'],
estimators = ['DummyRegressor'],
add_mean = True,
add_median = True
)
# Plot benchmark results
# ==============================================================================
plot_benchmark_results(
results_benchmark_all,
forecaster_names = ['ForecasterDirectMultiVariate', 'ForecasterAutoregMultiVariate'],
estimators = ['DummyRegressor'],
add_mean = True,
add_median = True
)
Plot function¶
In [18]:
Copied!
# Plot function
# ==============================================================================
from __future__ import annotations
import numpy as np
import pandas as pd
import plotly.io as pio
import plotly.graph_objects as go
from plotly.express.colors import qualitative
from packaging.version import Version, InvalidVersion
pio.renderers.default = "notebook_connected"
def plot_benchmark_results(
df: pd.DataFrame,
forecaster_names: str | list[str],
estimators: str | list[str] | None = None,
add_median: bool = True,
add_mean: bool = True
) -> None:
"""
Plot interactive benchmark results by method and package version.
This function renders an interactive Plotly chart that visualizes execution
times for multiple methods across `skforecast` versions. Each data point
represents a run (or an aggregated run) for a given `(method, version)` and is
jittered horizontally to avoid overlap. Points are **colored by version** and
(optionally) per-version **median** and **mean** lines are overlaid for the
selected method. A dropdown allows switching the visible method.
Parameters
----------
df : pandas DataFrame
Input data to benchmark.
forecaster_names : str, list
Forecaster(s) to filter in `df` (column `forecaster_name`).
estimators : str, list, default None
Estimator(s) to filter in `df` (column `estimator_name`). If `None`,
no additional filtering by estimator is applied.
add_median : bool, default True
If `True`, draw one per-version **median** line for the selected method.
add_mean : bool, default True
If `True`, draw one per-version **mean** line for the selected method.
Returns
-------
None
Displays the plot inline.
"""
if not isinstance(forecaster_names, list):
forecaster_names = [forecaster_names]
df = df.query("forecaster_name in @forecaster_names")
if estimators is not None:
if not isinstance(estimators, list):
estimators = [estimators]
df = df.query("estimator_name in @estimators")
if df.empty:
print("No data found for the specified filters.")
return
df = df.copy()
def try_version(v):
try:
return Version(str(v).lstrip('v'))
except InvalidVersion:
return Version("0") # fallback
versions_sorted = sorted(df['skforecast_version'].unique(), key=try_version)
version_to_num = {v: i for i, v in enumerate(versions_sorted)}
df['skforecast_version'] = pd.Categorical(
df['skforecast_version'],
categories=versions_sorted,
ordered=True
)
rng = np.random.default_rng(42)
df['x_jittered'] = (
df['skforecast_version'].map(version_to_num).astype(float) +
rng.uniform(-0.05, 0.05, size=len(df))
)
# --- paleta por versión ---
version_colors = {
v: qualitative.Plotly[i % len(qualitative.Plotly)]
for i, v in enumerate(versions_sorted)
}
methods = list(df['method_name'].unique())
fig = go.Figure()
# --- un trace por (método, métrica); colores por versión en los puntos ---
method_to_traces = {m: [] for m in methods}
for i, m in enumerate(methods):
for version in versions_sorted:
sub_df = df[
(df['method_name'] == m) &
(df['skforecast_version'] == version)
]
if sub_df.empty:
continue
marker = dict(
size=10,
color=version_colors[version],
opacity=0.85,
line=dict(color="white", width=1)
)
error_y = dict(
type='data',
array=sub_df['run_time_std'].to_numpy(),
visible=not sub_df['run_time_std'].isna().all(),
color=version_colors[version],
thickness=1.5,
width=5
)
fig.add_trace(go.Scatter(
x=sub_df['x_jittered'],
y=sub_df['run_time_avg'],
mode='markers',
marker=marker,
error_y=error_y,
visible=(i == 0),
# name=f"{methods[i]} — {label}",
text = sub_df.apply(lambda row: (
f"Run ID: {row['run_id']}<br>"
f"Forecaster: {row['forecaster_name']}<br>"
f"Estimator: {row['estimator_name']}<br>"
f"Function: {row['function_name']}<br>"
f"Function_hash: {row['function_hash']}<br>"
f"Method: {row['method_name']}<br>"
f"Datetime: {row['datetime']}<br>"
f"Python version: {row['python_version']}<br>"
f"skforecast version: {row['skforecast_version']}<br>"
f"numpy version: {row['numpy_version']}<br>"
f"pandas version: {row['pandas_version']}<br>"
f"sklearn version: {row['sklearn_version']}<br>"
f"lightgbm version: {row['lightgbm_version']}<br>"
f"Platform: {row['platform']}<br>"
f"Processor: {row['processor']}<br>"
f"CPU count: {row['cpu_count']}<br>"
f"CPU model: {row['cpu_model']}<br>"
f"Memory (GB): {row['memory_gb']:.2f}<br>"
f"Run time avg: {row['run_time_avg']:.6f} s<br>"
f"Run time median: {row['run_time_median']:.6f} s<br>"
f"Run time p95: {row['run_time_p95']:.6f} s<br>"
f"Run time std: {row['run_time_std']:.6f} s<br>"
f"Nº repeats: {row['n_repeats']}"
), axis=1),
hovertemplate = '%{text}<extra></extra>'
))
method_to_traces[m].append(len(fig.data) - 1)
median_trace_id = {}
if add_median:
for i, m in enumerate(methods):
med_df = (
df[df['method_name'] == m]
.groupby('skforecast_version', observed=True)['run_time_avg']
.median()
.reset_index()
)
if med_df.empty:
continue
median_color = "#374151"
med_df['x_center'] = med_df['skforecast_version'].map(version_to_num)
fig.add_trace(go.Scatter(
x=med_df['x_center'],
y=med_df['run_time_avg'],
mode='lines+markers',
line=dict(color=median_color, width=2),
marker=dict(size=8, color=median_color),
name='Median (per version)',
visible=(i == 0)
))
median_trace_id[m] = len(fig.data) - 1
mean_trace_id = {}
if add_mean:
for i, m in enumerate(methods):
mean_df = (
df[df['method_name'] == m]
.groupby('skforecast_version', observed=True)['run_time_avg']
.mean()
.reset_index()
)
if mean_df.empty:
continue
mean_color = "#9CA3AF"
mean_df['x_center'] = mean_df['skforecast_version'].map(version_to_num)
fig.add_trace(go.Scatter(
x=mean_df['x_center'],
y=mean_df['run_time_avg'],
mode='lines+markers',
line=dict(color=mean_color, width=2, dash='dash'),
marker=dict(size=8, color=mean_color),
name='Mean (per version)',
visible=(i == 0)
))
mean_trace_id[m] = len(fig.data) - 1
def visible_mask_for(method):
n = len(fig.data)
mask = [False] * n
# puntos del método (todas sus versiones)
for idx in method_to_traces.get(method, []):
mask[idx] = True
# su mediana y media (si existen)
if add_median and method in median_trace_id:
mask[median_trace_id[method]] = True
if add_mean and method in mean_trace_id:
mask[mean_trace_id[method]] = True
return mask
buttons_methods = []
for i, m in enumerate(methods):
buttons_methods.append(dict(
label=m,
method="update",
args=[
{"visible": visible_mask_for(m)},
{"title": {"text": f"Execution time — method: `{m}`"}}
]
))
fig.update_layout(
title=dict(
text=f"Execution time — method: `{methods[0]}`"
),
xaxis=dict(
tickmode="array",
tickvals=list(version_to_num.values()),
ticktext=list(version_to_num.keys()),
title="skforecast version",
tickangle=0,
automargin=True,
),
yaxis=dict(
title="Execution time (s)",
automargin=True
),
# template="plotly_white",
margin=dict(l=70, r=20, t=100, b=60),
updatemenus=[
dict(
type="dropdown",
buttons=buttons_methods,
showactive=True,
direction="down",
x=1.00,
y=1.03,
xanchor="right",
yanchor="bottom",
pad={"r": 2, "t": 0},
),
],
legend=dict(title=""),
showlegend=False,
)
fig.show()
# Show in web (other option)
# from IPython.display import HTML
# return HTML(fig.to_html(full_html=False, include_plotlyjs="cdn"))
# Plot function
# ==============================================================================
from __future__ import annotations
import numpy as np
import pandas as pd
import plotly.io as pio
import plotly.graph_objects as go
from plotly.express.colors import qualitative
from packaging.version import Version, InvalidVersion
pio.renderers.default = "notebook_connected"
def plot_benchmark_results(
df: pd.DataFrame,
forecaster_names: str | list[str],
estimators: str | list[str] | None = None,
add_median: bool = True,
add_mean: bool = True
) -> None:
"""
Plot interactive benchmark results by method and package version.
This function renders an interactive Plotly chart that visualizes execution
times for multiple methods across `skforecast` versions. Each data point
represents a run (or an aggregated run) for a given `(method, version)` and is
jittered horizontally to avoid overlap. Points are **colored by version** and
(optionally) per-version **median** and **mean** lines are overlaid for the
selected method. A dropdown allows switching the visible method.
Parameters
----------
df : pandas DataFrame
Input data to benchmark.
forecaster_names : str, list
Forecaster(s) to filter in `df` (column `forecaster_name`).
estimators : str, list, default None
Estimator(s) to filter in `df` (column `estimator_name`). If `None`,
no additional filtering by estimator is applied.
add_median : bool, default True
If `True`, draw one per-version **median** line for the selected method.
add_mean : bool, default True
If `True`, draw one per-version **mean** line for the selected method.
Returns
-------
None
Displays the plot inline.
"""
if not isinstance(forecaster_names, list):
forecaster_names = [forecaster_names]
df = df.query("forecaster_name in @forecaster_names")
if estimators is not None:
if not isinstance(estimators, list):
estimators = [estimators]
df = df.query("estimator_name in @estimators")
if df.empty:
print("No data found for the specified filters.")
return
df = df.copy()
def try_version(v):
try:
return Version(str(v).lstrip('v'))
except InvalidVersion:
return Version("0") # fallback
versions_sorted = sorted(df['skforecast_version'].unique(), key=try_version)
version_to_num = {v: i for i, v in enumerate(versions_sorted)}
df['skforecast_version'] = pd.Categorical(
df['skforecast_version'],
categories=versions_sorted,
ordered=True
)
rng = np.random.default_rng(42)
df['x_jittered'] = (
df['skforecast_version'].map(version_to_num).astype(float) +
rng.uniform(-0.05, 0.05, size=len(df))
)
# --- paleta por versión ---
version_colors = {
v: qualitative.Plotly[i % len(qualitative.Plotly)]
for i, v in enumerate(versions_sorted)
}
methods = list(df['method_name'].unique())
fig = go.Figure()
# --- un trace por (método, métrica); colores por versión en los puntos ---
method_to_traces = {m: [] for m in methods}
for i, m in enumerate(methods):
for version in versions_sorted:
sub_df = df[
(df['method_name'] == m) &
(df['skforecast_version'] == version)
]
if sub_df.empty:
continue
marker = dict(
size=10,
color=version_colors[version],
opacity=0.85,
line=dict(color="white", width=1)
)
error_y = dict(
type='data',
array=sub_df['run_time_std'].to_numpy(),
visible=not sub_df['run_time_std'].isna().all(),
color=version_colors[version],
thickness=1.5,
width=5
)
fig.add_trace(go.Scatter(
x=sub_df['x_jittered'],
y=sub_df['run_time_avg'],
mode='markers',
marker=marker,
error_y=error_y,
visible=(i == 0),
# name=f"{methods[i]} — {label}",
text = sub_df.apply(lambda row: (
f"Run ID: {row['run_id']}<br>"
f"Forecaster: {row['forecaster_name']}<br>"
f"Estimator: {row['estimator_name']}<br>"
f"Function: {row['function_name']}<br>"
f"Function_hash: {row['function_hash']}<br>"
f"Method: {row['method_name']}<br>"
f"Datetime: {row['datetime']}<br>"
f"Python version: {row['python_version']}<br>"
f"skforecast version: {row['skforecast_version']}<br>"
f"numpy version: {row['numpy_version']}<br>"
f"pandas version: {row['pandas_version']}<br>"
f"sklearn version: {row['sklearn_version']}<br>"
f"lightgbm version: {row['lightgbm_version']}<br>"
f"Platform: {row['platform']}<br>"
f"Processor: {row['processor']}<br>"
f"CPU count: {row['cpu_count']}<br>"
f"CPU model: {row['cpu_model']}<br>"
f"Memory (GB): {row['memory_gb']:.2f}<br>"
f"Run time avg: {row['run_time_avg']:.6f} s<br>"
f"Run time median: {row['run_time_median']:.6f} s<br>"
f"Run time p95: {row['run_time_p95']:.6f} s<br>"
f"Run time std: {row['run_time_std']:.6f} s<br>"
f"Nº repeats: {row['n_repeats']}"
), axis=1),
hovertemplate = '%{text}<extra></extra>'
))
method_to_traces[m].append(len(fig.data) - 1)
median_trace_id = {}
if add_median:
for i, m in enumerate(methods):
med_df = (
df[df['method_name'] == m]
.groupby('skforecast_version', observed=True)['run_time_avg']
.median()
.reset_index()
)
if med_df.empty:
continue
median_color = "#374151"
med_df['x_center'] = med_df['skforecast_version'].map(version_to_num)
fig.add_trace(go.Scatter(
x=med_df['x_center'],
y=med_df['run_time_avg'],
mode='lines+markers',
line=dict(color=median_color, width=2),
marker=dict(size=8, color=median_color),
name='Median (per version)',
visible=(i == 0)
))
median_trace_id[m] = len(fig.data) - 1
mean_trace_id = {}
if add_mean:
for i, m in enumerate(methods):
mean_df = (
df[df['method_name'] == m]
.groupby('skforecast_version', observed=True)['run_time_avg']
.mean()
.reset_index()
)
if mean_df.empty:
continue
mean_color = "#9CA3AF"
mean_df['x_center'] = mean_df['skforecast_version'].map(version_to_num)
fig.add_trace(go.Scatter(
x=mean_df['x_center'],
y=mean_df['run_time_avg'],
mode='lines+markers',
line=dict(color=mean_color, width=2, dash='dash'),
marker=dict(size=8, color=mean_color),
name='Mean (per version)',
visible=(i == 0)
))
mean_trace_id[m] = len(fig.data) - 1
def visible_mask_for(method):
n = len(fig.data)
mask = [False] * n
# puntos del método (todas sus versiones)
for idx in method_to_traces.get(method, []):
mask[idx] = True
# su mediana y media (si existen)
if add_median and method in median_trace_id:
mask[median_trace_id[method]] = True
if add_mean and method in mean_trace_id:
mask[mean_trace_id[method]] = True
return mask
buttons_methods = []
for i, m in enumerate(methods):
buttons_methods.append(dict(
label=m,
method="update",
args=[
{"visible": visible_mask_for(m)},
{"title": {"text": f"Execution time — method: `{m}`"}}
]
))
fig.update_layout(
title=dict(
text=f"Execution time — method: `{methods[0]}`"
),
xaxis=dict(
tickmode="array",
tickvals=list(version_to_num.values()),
ticktext=list(version_to_num.keys()),
title="skforecast version",
tickangle=0,
automargin=True,
),
yaxis=dict(
title="Execution time (s)",
automargin=True
),
# template="plotly_white",
margin=dict(l=70, r=20, t=100, b=60),
updatemenus=[
dict(
type="dropdown",
buttons=buttons_methods,
showactive=True,
direction="down",
x=1.00,
y=1.03,
xanchor="right",
yanchor="bottom",
pad={"r": 2, "t": 0},
),
],
legend=dict(title=""),
showlegend=False,
)
fig.show()
# Show in web (other option)
# from IPython.display import HTML
# return HTML(fig.to_html(full_html=False, include_plotlyjs="cdn"))