Parallelization in skforecast¶

The n_jobs argument runs part of the work of skforecast in several processes, with joblib. Two operations are parallelized:

• The folds of a backtesting: backtesting_forecaster, backtesting_forecaster_multiseries and backtesting_stats, and therefore the hyperparameter searches, which run a backtesting for each candidate.

• The fit of the models of each step in ForecasterDirect and ForecasterDirectMultiVariate.

Everything else (the fit and predict of the other forecasters, the creation of the training matrices) runs in the main process, although the estimator can use threads of its own.

Libraries¶

In [1]:

# Libraries
# ==============================================================================
import platform
import time
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import joblib
import sklearn
import lightgbm
import skforecast
from joblib import cpu_count
from joblib.externals.loky import get_reusable_executor
from lightgbm import LGBMRegressor
from sklearn.linear_model import Ridge
from skforecast.datasets import fetch_dataset
from skforecast.recursive import ForecasterRecursive
from skforecast.recursive import ForecasterRecursiveMultiSeries
from skforecast.direct import ForecasterDirect
from skforecast.model_selection import TimeSeriesFold
from skforecast.model_selection import backtesting_forecaster
from skforecast.model_selection import backtesting_forecaster_multiseries
from skforecast.model_selection import grid_search_forecaster
from skforecast.model_selection._utils import select_n_jobs_backtesting

# Libraries # ============================================================================== import platform import time import numpy as np import pandas as pd import matplotlib.pyplot as plt import joblib import sklearn import lightgbm import skforecast from joblib import cpu_count from joblib.externals.loky import get_reusable_executor from lightgbm import LGBMRegressor from sklearn.linear_model import Ridge from skforecast.datasets import fetch_dataset from skforecast.recursive import ForecasterRecursive from skforecast.recursive import ForecasterRecursiveMultiSeries from skforecast.direct import ForecasterDirect from skforecast.model_selection import TimeSeriesFold from skforecast.model_selection import backtesting_forecaster from skforecast.model_selection import backtesting_forecaster_multiseries from skforecast.model_selection import grid_search_forecaster from skforecast.model_selection._utils import select_n_jobs_backtesting

In [2]:

# Versions and system
# ==============================================================================
print(f"Python version      : {platform.python_version()}")
print(f"skforecast version  : {skforecast.__version__}")
print(f"scikit-learn version: {sklearn.__version__}")
print(f"lightgbm version    : {lightgbm.__version__}")
print(f"joblib version      : {joblib.__version__}")
print(f"pandas version      : {pd.__version__}")
print(f"numpy version       : {np.__version__}")
print("")
print(f"Operating system    : {platform.platform()}")
print(f"Processor           : {platform.processor()}")
print(f"Physical cores      : {cpu_count(only_physical_cores=True)}")
print(f"Logical processors  : {cpu_count()}")

# Versions and system # ============================================================================== print(f"Python version : {platform.python_version()}") print(f"skforecast version : {skforecast.__version__}") print(f"scikit-learn version: {sklearn.__version__}") print(f"lightgbm version : {lightgbm.__version__}") print(f"joblib version : {joblib.__version__}") print(f"pandas version : {pd.__version__}") print(f"numpy version : {np.__version__}") print("") print(f"Operating system : {platform.platform()}") print(f"Processor : {platform.processor()}") print(f"Physical cores : {cpu_count(only_physical_cores=True)}") print(f"Logical processors : {cpu_count()}")

Python version      : 3.13.18
skforecast version  : 0.27.0
scikit-learn version: 1.9.1
lightgbm version    : 4.7.0
joblib version      : 1.6.0
pandas version      : 2.3.3
numpy version       : 2.5.4

Operating system    : Windows-11-10.0.26200-SP0
Processor           : Intel64 Family 6 Model 170 Stepping 4, GenuineIntel
Physical cores      : 16
Logical processors  : 22

How n_jobs='auto' works¶

'auto', the default value, chooses the number of jobs from the forecaster, the estimator and refit, without measuring the machine. In a backtesting, the first rule that applies wins:

Rule	n_jobs	Why
refit is an integer other than 1 (intermittent refit)	1	The forecaster fitted in a fold is reused in the following ones, so the folds cannot run in parallel.
refit is False (no refit), and the estimator, or the last step of a Pipeline, is not a HistGradientBoostingRegressor or a CatBoostRegressor	1	The folds only predict with the forecaster fitted once, and sending it to the workers costs more than the predictions they share. HistGradientBoostingRegressor and CatBoostRegressor predict slowly enough to pay for the workers in a search.
The forecaster is not ForecasterRecursive, ForecasterRecursiveClassifier or ForecasterRecursiveMultiSeries	1	ForecasterDirect and ForecasterDirectMultiVariate parallelize the fit of their steps instead; the other forecasters are not parallelized.
The estimator, or the last step of a Pipeline, is a linear model of scikit-learn (Ridge, LinearRegression...)	1	It fits and predicts in less time than the workers take to start. LogisticRegression is not included: it is fitted iteratively.
The estimator is a LGBMRegressor or LGBMClassifier with n_jobs != 1	1	LightGBM already uses several threads, and more processes would compete for the same cores.
Any other case	cpu_count() - 1	One processor is left for the main process.

Starting the workers costs seconds, so a single backtesting also looks at its workload: it times the initial fit, multiplies it by the number of refits and, if the result is below the threshold of the operating system (4 seconds on Windows, 2 on Linux and macOS), runs the folds in one process. Without refit the result is zero, so a single backtesting always runs in one process. Above the threshold, it starts at most one worker per fold. The hyperparameter searches skip the workload check: they start the workers once and reuse them for every candidate.

The fit of ForecasterDirect and ForecasterDirectMultiVariate applies the rules about the estimator (one job with a linear model or with the threads of LightGBM, cpu_count() - 1 otherwise) and stores the result in the n_jobs attribute of the forecaster.

An integer n_jobs is used as given, and -1 means all the processors. The exception is an intermittent refit, which always runs in one process (with a warning if n_jobs is not 1). The rules live in select_n_jobs_backtesting, limit_n_jobs_by_workload and select_n_jobs_fit_forecaster.

n_jobs='auto' in a backtesting or a search

Is refit an integer other than 1
(intermittent refit)?

yes

1 job
the folds reuse the
fitted forecaster

no

Is refit False (no refit)
and the estimator not a
HistGradientBoostingRegressor
or a CatBoostRegressor?

yes

1 job
the folds only
predict

no

Is the forecaster other than ForecasterRecursive, ForecasterRecursiveClassifier or ForecasterRecursiveMultiSeries?

yes

1 job
Direct parallelizes its fit;
the others are not parallelized

no

Is the estimator (or the last step of the Pipeline) a linear model of scikit-learn?

yes

1 job
it fits faster than the
workers start

no

Is it a LGBMRegressor or LGBMClassifier with n_jobs != 1?

yes

1 job
LightGBM already uses
several threads

no

Single backtesting only: is the time of the initial fit × the number of refits below the threshold of the operating system?

yes

1 job
starting the workers would
cost more than it saves

no, or a search

cpu_count() - 1 jobs

select_n_jobs_backtesting applies the rules of the table to a forecaster. The workload check is not included: it needs the time of the initial fit, so it runs inside the backtesting.

In [3]:

# Decision of n_jobs='auto' for several forecasters
# ==============================================================================
forecasters = {
    "ForecasterRecursive, Ridge": ForecasterRecursive(
        estimator=Ridge(), lags=24
    ),
    "ForecasterRecursive, LGBMRegressor(n_jobs=1)": ForecasterRecursive(
        estimator=LGBMRegressor(n_jobs=1, verbose=-1), lags=24
    ),
    "ForecasterRecursive, LGBMRegressor() with its default threads": ForecasterRecursive(
        estimator=LGBMRegressor(verbose=-1), lags=24
    ),
    "ForecasterRecursiveMultiSeries, LGBMRegressor(n_jobs=1)": ForecasterRecursiveMultiSeries(
        estimator=LGBMRegressor(n_jobs=1, verbose=-1), lags=24
    ),
    "ForecasterDirect, LGBMRegressor(n_jobs=1)": ForecasterDirect(
        estimator=LGBMRegressor(n_jobs=1, verbose=-1), lags=24, steps=24
    ),
}

decisions = pd.DataFrame({
    f"refit={refit}": {
        name: select_n_jobs_backtesting(forecaster, refit=refit)
        for name, forecaster in forecasters.items()
    }
    for refit in [True, 3]
})
decisions

# Decision of n_jobs='auto' for several forecasters # ============================================================================== forecasters = { "ForecasterRecursive, Ridge": ForecasterRecursive( estimator=Ridge(), lags=24 ), "ForecasterRecursive, LGBMRegressor(n_jobs=1)": ForecasterRecursive( estimator=LGBMRegressor(n_jobs=1, verbose=-1), lags=24 ), "ForecasterRecursive, LGBMRegressor() with its default threads": ForecasterRecursive( estimator=LGBMRegressor(verbose=-1), lags=24 ), "ForecasterRecursiveMultiSeries, LGBMRegressor(n_jobs=1)": ForecasterRecursiveMultiSeries( estimator=LGBMRegressor(n_jobs=1, verbose=-1), lags=24 ), "ForecasterDirect, LGBMRegressor(n_jobs=1)": ForecasterDirect( estimator=LGBMRegressor(n_jobs=1, verbose=-1), lags=24, steps=24 ), } decisions = pd.DataFrame({ f"refit={refit}": { name: select_n_jobs_backtesting(forecaster, refit=refit) for name, forecaster in forecasters.items() } for refit in [True, 3] }) decisions

Out[3]:

	refit=True	refit=3
ForecasterRecursive, Ridge	1	1
ForecasterRecursive, LGBMRegressor(n_jobs=1)	21	1
ForecasterRecursive, LGBMRegressor() with its default threads	1	1
ForecasterRecursiveMultiSeries, LGBMRegressor(n_jobs=1)	21	1
ForecasterDirect, LGBMRegressor(n_jobs=1)	1	1

With refit=True, only the recursive forecasters with a one-thread LGBMRegressor get cpu_count() - 1 jobs (21 on this machine, which has 22 logical processors). Ridge and the default threads of LightGBM give one job, and so does every case with an intermittent refit (refit=3). ForecasterDirect gets one job in the backtesting because it parallelizes its own fit, and stores that decision in its n_jobs attribute.

In [4]:

# n_jobs of the fit of ForecasterDirect
# ==============================================================================
forecaster_direct = forecasters["ForecasterDirect, LGBMRegressor(n_jobs=1)"]
print(f"ForecasterDirect fits its {len(forecaster_direct.steps)} steps with {forecaster_direct.n_jobs} jobs")

# n_jobs of the fit of ForecasterDirect # ============================================================================== forecaster_direct = forecasters["ForecasterDirect, LGBMRegressor(n_jobs=1)"] print(f"ForecasterDirect fits its {len(forecaster_direct.steps)} steps with {forecaster_direct.n_jobs} jobs")

ForecasterDirect fits its 24 steps with 21 jobs

Choosing n_jobs yourself¶

'auto' decides without measuring the machine. Some situations call for an explicit value:

• Starting the workers. Each worker is a new Python process that imports skforecast and the libraries of the estimator. The first parallel call of a session pays it: a few seconds, more on Windows than on Linux or macOS. The following calls reuse the workers, which stay alive for 5 minutes without use. If you run many backtestings in the same session, an integer n_jobs can be faster than 'auto', which decides as if each backtesting were the only one.

• Threads of the estimator. 'auto' only detects the threads of LightGBM. In the workers, joblib already limits the threads of XGBoost and HistGradientBoostingRegressor with their default settings. LightGBM, CatBoost and a random forest with n_jobs=-1 keep all their threads in every worker and compete for the same cores. Use one level of parallelism: an estimator with one thread (n_jobs=1, thread_count=1...) and the folds in parallel, or the opposite. On some Linux systems (measured in WSL2), LightGBM with as many threads as logical processors (n_jobs=-1) trains much slower than with its default value.

• Number of processors. cpu_count() counts logical processors, including hyperthreading and efficiency cores, which also speed up the work. joblib respects the affinity of the process and the LOKY_MAX_CPU_COUNT environment variable, so you can limit the processors that skforecast uses on a shared machine without changing the code.

⚠ Warning

An integer n_jobs is used as given, also when the estimator has its own threads. A LGBMRegressor with its default threads and the folds in parallel can be tens of times slower than a single process (measured on Linux with 4 processors). Set n_jobs=1 in the estimator before passing an integer n_jobs to skforecast. 'auto' avoids this case only when the LGBMRegressor is the estimator of the forecaster or the last step of a Pipeline.

Benchmark¶

The benchmark compares n_jobs=1, 'auto' and cpu_count() - 1 in the operations that run in parallel. Each case gives two times: the first call, with the workers stopped as in a new session, and the median of 3 more calls, with the workers already started (the table also shows their minimum and maximum). The times belong to the machine and the versions printed above; they change with the processor, the operating system and the size of the data.

The benchmark uses two datasets of the documentation: the hourly users of a bike sharing system, with 6 exogenous variables, and the daily sales of the 50 items of one store. The exogenous variables only give the estimator a realistic amount of work: the weather variables are used as if they were known in advance, and the accuracy of the forecasts is not evaluated.

In [5]:

# Data
# ==============================================================================
bike = fetch_dataset(name="bike_sharing", raw=False, verbose=False)
y = bike["users"]
exog = bike[["temp", "hum", "windspeed", "holiday", "hour", "weekday"]]

sales = fetch_dataset(name="store_sales", raw=False, verbose=False)
sales = sales.query("store == 1").pivot(columns="item", values="sales").asfreq("D")
series = {f"item_{item}": sales[item] for item in sales.columns}

print(f"Bike sharing: {len(y)} hourly values and {exog.shape[1]} exogenous variables")
print(f"Store sales : {len(series)} daily series of {len(sales)} values")

# Data # ============================================================================== bike = fetch_dataset(name="bike_sharing", raw=False, verbose=False) y = bike["users"] exog = bike[["temp", "hum", "windspeed", "holiday", "hour", "weekday"]] sales = fetch_dataset(name="store_sales", raw=False, verbose=False) sales = sales.query("store == 1").pivot(columns="item", values="sales").asfreq("D") series = {f"item_{item}": sales[item] for item in sales.columns} print(f"Bike sharing: {len(y)} hourly values and {exog.shape[1]} exogenous variables") print(f"Store sales : {len(series)} daily series of {len(sales)} values")

Bike sharing: 17544 hourly values and 6 exogenous variables
Store sales : 50 daily series of 1826 values

time_call stops the workers before the first call, so that it pays their start as in a new session. The workers belong to loky, the library joblib uses to manage its processes, and get_reusable_executor().shutdown() stops them.

In [6]:

# Timing function
# ==============================================================================
def time_call(fn, n_reps=3):
    """
    Time the first call to `fn`, with the workers stopped, and the median,
    minimum and maximum of `n_reps` more calls.
    """
    get_reusable_executor().shutdown(wait=True)
    start = time.perf_counter()
    fn()
    first_call = time.perf_counter() - start

    times = []
    for _ in range(n_reps):
        start = time.perf_counter()
        fn()
        times.append(time.perf_counter() - start)

    return {
        "first call": first_call,
        "median": np.median(times),
        "min": np.min(times),
        "max": np.max(times),
    }

# Timing function # ============================================================================== def time_call(fn, n_reps=3): """ Time the first call to `fn`, with the workers stopped, and the median, minimum and maximum of `n_reps` more calls. """ get_reusable_executor().shutdown(wait=True) start = time.perf_counter() fn() first_call = time.perf_counter() - start times = [] for _ in range(n_reps): start = time.perf_counter() fn() times.append(time.perf_counter() - start) return { "first call": first_call, "median": np.median(times), "min": np.min(times), "max": np.max(times), }

The benchmark has six cases: a backtesting without refit (30 folds), a backtesting with refit and little work (14 folds), the same with more work (42 folds), a multi-series backtesting (14 folds), the fit of a ForecasterDirect (24 steps) and a grid search (2 candidates, 21 folds). The forecasters use a LGBMRegressor with one thread (n_jobs=1), so that the parallelism comes only from skforecast. A seventh case, timed only with 'auto', uses the default threads of LightGBM, with which 'auto' runs the folds in one process.

In [7]:

# Cases
# ==============================================================================
lgbm_params = {"n_estimators": 100, "max_depth": 5, "random_state": 123, "verbose": -1}
forecaster = ForecasterRecursive(
                 estimator = LGBMRegressor(n_jobs=1, **lgbm_params),
                 lags      = 24
             )
forecaster_lgbm_threads = ForecasterRecursive(
                              estimator = LGBMRegressor(**lgbm_params),
                              lags      = 24
                          )
forecaster_multiseries = ForecasterRecursiveMultiSeries(
                             estimator = LGBMRegressor(n_jobs=1, **lgbm_params),
                             lags      = 14
                         )


def backtesting(forecaster, n_folds, refit, n_jobs):
    cv = TimeSeriesFold(
             steps              = 24,
             initial_train_size = len(y) - 24 * n_folds,
             refit              = refit
         )
    backtesting_forecaster(
        forecaster    = forecaster,
        y             = y,
        exog          = exog,
        cv            = cv,
        metric        = "mean_absolute_error",
        n_jobs        = n_jobs,
        show_progress = False
    )


def backtesting_multiseries(n_folds, n_jobs):
    cv = TimeSeriesFold(
             steps              = 7,
             initial_train_size = len(sales) - 7 * n_folds,
             refit              = True
         )
    backtesting_forecaster_multiseries(
        forecaster    = forecaster_multiseries,
        series        = series,
        cv            = cv,
        metric        = "mean_absolute_error",
        n_jobs        = n_jobs,
        show_progress = False
    )


def fit_direct(n_jobs):
    forecaster_direct = ForecasterDirect(
                            estimator = LGBMRegressor(n_jobs=1, **lgbm_params),
                            lags      = 24,
                            steps     = 24,
                            n_jobs    = n_jobs
                        )
    forecaster_direct.fit(y=y, exog=exog)


def grid_search(n_folds, n_jobs):
    cv = TimeSeriesFold(
             steps              = 24,
             initial_train_size = len(y) - 24 * n_folds,
             refit              = True
         )
    grid_search_forecaster(
        forecaster    = forecaster,
        y             = y,
        exog          = exog,
        cv            = cv,
        param_grid    = {"n_estimators": [50, 100]},
        metric        = "mean_absolute_error",
        return_best   = False,
        n_jobs        = n_jobs,
        verbose       = False,
        show_progress = False
    )


cases = {
    "Backtesting, no refit, 30 folds": lambda n_jobs: backtesting(forecaster, 30, False, n_jobs),
    "Backtesting, refit, 14 folds": lambda n_jobs: backtesting(forecaster, 14, True, n_jobs),
    "Backtesting, refit, 42 folds": lambda n_jobs: backtesting(forecaster, 42, True, n_jobs),
    "Backtesting multi-series, refit, 14 folds": lambda n_jobs: backtesting_multiseries(14, n_jobs),
    "ForecasterDirect fit, 24 steps": fit_direct,
    "Grid search, 2 candidates, 21 folds": lambda n_jobs: grid_search(21, n_jobs),
}

# Cases # ============================================================================== lgbm_params = {"n_estimators": 100, "max_depth": 5, "random_state": 123, "verbose": -1} forecaster = ForecasterRecursive( estimator = LGBMRegressor(n_jobs=1, **lgbm_params), lags = 24 ) forecaster_lgbm_threads = ForecasterRecursive( estimator = LGBMRegressor(**lgbm_params), lags = 24 ) forecaster_multiseries = ForecasterRecursiveMultiSeries( estimator = LGBMRegressor(n_jobs=1, **lgbm_params), lags = 14 ) def backtesting(forecaster, n_folds, refit, n_jobs): cv = TimeSeriesFold( steps = 24, initial_train_size = len(y) - 24 * n_folds, refit = refit ) backtesting_forecaster( forecaster = forecaster, y = y, exog = exog, cv = cv, metric = "mean_absolute_error", n_jobs = n_jobs, show_progress = False ) def backtesting_multiseries(n_folds, n_jobs): cv = TimeSeriesFold( steps = 7, initial_train_size = len(sales) - 7 * n_folds, refit = True ) backtesting_forecaster_multiseries( forecaster = forecaster_multiseries, series = series, cv = cv, metric = "mean_absolute_error", n_jobs = n_jobs, show_progress = False ) def fit_direct(n_jobs): forecaster_direct = ForecasterDirect( estimator = LGBMRegressor(n_jobs=1, **lgbm_params), lags = 24, steps = 24, n_jobs = n_jobs ) forecaster_direct.fit(y=y, exog=exog) def grid_search(n_folds, n_jobs): cv = TimeSeriesFold( steps = 24, initial_train_size = len(y) - 24 * n_folds, refit = True ) grid_search_forecaster( forecaster = forecaster, y = y, exog = exog, cv = cv, param_grid = {"n_estimators": [50, 100]}, metric = "mean_absolute_error", return_best = False, n_jobs = n_jobs, verbose = False, show_progress = False ) cases = { "Backtesting, no refit, 30 folds": lambda n_jobs: backtesting(forecaster, 30, False, n_jobs), "Backtesting, refit, 14 folds": lambda n_jobs: backtesting(forecaster, 14, True, n_jobs), "Backtesting, refit, 42 folds": lambda n_jobs: backtesting(forecaster, 42, True, n_jobs), "Backtesting multi-series, refit, 14 folds": lambda n_jobs: backtesting_multiseries(14, n_jobs), "ForecasterDirect fit, 24 steps": fit_direct, "Grid search, 2 candidates, 21 folds": lambda n_jobs: grid_search(21, n_jobs), }

Results¶

In [8]:

# Benchmark
# ==============================================================================
rows = []
for case, fn in cases.items():
    for n_jobs in [1, "auto", cpu_count() - 1]:
        timings = time_call(lambda: fn(n_jobs))
        rows.append({"case": case, "n_jobs": n_jobs, **timings})

timings = time_call(lambda: backtesting(forecaster_lgbm_threads, 42, True, "auto"))
rows.append({
    "case": "Backtesting, refit, 42 folds, LightGBM threads",
    "n_jobs": "auto",
    **timings
})

results = pd.DataFrame(rows).set_index(["case", "n_jobs"]).round(2)
results

# Benchmark # ============================================================================== rows = [] for case, fn in cases.items(): for n_jobs in [1, "auto", cpu_count() - 1]: timings = time_call(lambda: fn(n_jobs)) rows.append({"case": case, "n_jobs": n_jobs, **timings}) timings = time_call(lambda: backtesting(forecaster_lgbm_threads, 42, True, "auto")) rows.append({ "case": "Backtesting, refit, 42 folds, LightGBM threads", "n_jobs": "auto", **timings }) results = pd.DataFrame(rows).set_index(["case", "n_jobs"]).round(2) results

Out[8]:

		first call	median	min	max
case	n_jobs
Backtesting, no refit, 30 folds	1	0.27	0.23	0.23	0.23
	auto	0.26	0.24	0.23	0.26
	21	4.87	0.37	0.36	0.39
Backtesting, refit, 14 folds	1	2.74	2.61	2.58	2.64
	auto	2.71	2.73	2.62	2.78
	21	3.95	0.70	0.65	2.62
Backtesting, refit, 42 folds	1	8.12	7.88	7.81	7.96
	auto	5.55	1.26	1.26	1.26
	21	5.96	1.28	1.26	1.28
Backtesting multi-series, refit, 14 folds	1	6.22	5.84	5.83	6.18
	auto	4.80	1.58	1.56	1.63
	21	5.08	2.84	2.70	4.50
ForecasterDirect fit, 24 steps	1	10.74	7.26	6.57	10.71
	auto	5.63	1.23	1.20	1.28
	21	5.96	1.22	1.19	1.28
Grid search, 2 candidates, 21 folds	1	6.94	6.79	6.71	7.49
	auto	6.62	1.44	1.40	1.85
	21	7.11	2.30	2.16	2.72
Backtesting, refit, 42 folds, LightGBM threads	auto	4.41	3.85	3.83	4.02

The chart shows the first call and the median of the following calls of each case, in seconds. The LightGBM threads case is only in the table.

In [9]:

# Plot
# ==============================================================================
fig, axes = plt.subplots(2, 3, figsize=(11, 7.5), constrained_layout=True)
width = 0.36
for ax, case in zip(axes.flat, cases):
    case_results = results.loc[case]
    x = np.arange(len(case_results))
    bars_first = ax.bar(
        x - width / 2, case_results["first call"], width,
        color="#eb6834", label="First call, workers stopped"
    )
    bars_median = ax.bar(
        x + width / 2, case_results["median"], width,
        color="#2a78d6", label="Median of 3 calls, workers started"
    )
    ax.bar_label(bars_first, fmt="%.2f", fontsize=8, padding=2)
    ax.bar_label(bars_median, fmt="%.2f", fontsize=8, padding=2)
    ax.set_xticks(x, [f"n_jobs={n_jobs}" for n_jobs in case_results.index])
    ax.set_ylim(0, case_results[["first call", "median"]].max().max() * 1.25)
    ax.set_title(case, fontsize=10)
    ax.set_ylabel("seconds")
    ax.spines[["top", "right"]].set_visible(False)
axes.flat[0].legend(fontsize=8, frameon=False)
plt.show()

# Plot # ============================================================================== fig, axes = plt.subplots(2, 3, figsize=(11, 7.5), constrained_layout=True) width = 0.36 for ax, case in zip(axes.flat, cases): case_results = results.loc[case] x = np.arange(len(case_results)) bars_first = ax.bar( x - width / 2, case_results["first call"], width, color="#eb6834", label="First call, workers stopped" ) bars_median = ax.bar( x + width / 2, case_results["median"], width, color="#2a78d6", label="Median of 3 calls, workers started" ) ax.bar_label(bars_first, fmt="%.2f", fontsize=8, padding=2) ax.bar_label(bars_median, fmt="%.2f", fontsize=8, padding=2) ax.set_xticks(x, [f"n_jobs={n_jobs}" for n_jobs in case_results.index]) ax.set_ylim(0, case_results[["first call", "median"]].max().max() * 1.25) ax.set_title(case, fontsize=10) ax.set_ylabel("seconds") ax.spines[["top", "right"]].set_visible(False) axes.flat[0].legend(fontsize=8, frameon=False) plt.show()

Conclusions¶

In this run, on Windows, where starting the workers takes longer than on Linux or macOS:

• Without refit, the folds only predict, and running them in parallel does not pay, not even with the workers already started. 'auto' runs them in one process.

• With refit and little work (14 folds, less than 3 seconds in total), 'auto' also runs the folds in one process, because the first parallel call takes longer than the sequential one. With the workers already started, parallel is almost 4 times faster: if you repeat backtestings of this size in the same session, pass an integer n_jobs.

• With more work (42 folds, the multi-series backtesting, the fit of ForecasterDirect and the grid search), 'auto' runs in parallel and is as fast as setting cpu_count() - 1 by hand, or faster: between 1 and 1.9 times faster than n_jobs=1 in the first call, and between 3.7 and 6.3 times with the workers already started. In the multi-series backtesting, 'auto' starts one worker per fold (14) instead of 21. In the grid search, 'auto' and n_jobs=21 run the same configuration, so their difference (1.44 against 2.30 seconds) is variation between runs.

• LightGBM with its own threads makes 'auto' run the folds in one process. It is twice as fast as one thread. Against the folds in parallel with one-thread estimators, it is faster in the first call (4.41 against 5.55 seconds) and three times slower with the workers already started (3.85 against 1.26 seconds). Parallel folds with one-thread estimators are the better choice when the folds are refitted and the workers are reused.

• The min and max columns show how much the times vary between calls on this laptop: one sequential fit of ForecasterDirect took 60% longer than another. Read the ratios as orders of magnitude.

Recommendations¶

• Keep one level of parallelism: an estimator with one thread and the folds in parallel, which is what 'auto' does with LGBMRegressor(n_jobs=1), or the threads of the estimator and n_jobs=1.

• Pass an integer n_jobs, such as cpu_count() - 1, when you run several backtestings with refit in the same session, with a one-thread estimator. The workers are already started, and 'auto' may still choose one process for a small backtesting.

• Do not expect gains from n_jobs without refit, where the folds only predict. The exception is a hyperparameter search with a HistGradientBoostingRegressor or a CatBoostRegressor, which 'auto' runs in parallel. If the search has few folds, the first one of a session can take a few seconds longer than with n_jobs=1, because it starts the workers. With 'auto', the backtesting of ForecasterStats, ForecasterEquivalentDate and ForecasterRnn runs in one process. With slow statistical models and refit (for example, a seasonal ARIMA), an integer n_jobs in backtesting_stats can be faster: measure it. backtesting_foundation has no n_jobs argument.

• On a shared machine or in a container, limit the processors with LOKY_MAX_CPU_COUNT or the affinity of the process instead of changing n_jobs in the code.