model_selection

skforecast.model_selection._validation.backtesting_forecaster

backtesting_forecaster(
    forecaster,
    y,
    cv,
    metric,
    exog=None,
    interval=None,
    interval_method="bootstrapping",
    n_boot=250,
    use_in_sample_residuals=True,
    use_binned_residuals=True,
    random_state=123,
    return_predictors=False,
    n_jobs="auto",
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
)

Backtesting of forecaster model following the folds generated by the TimeSeriesFold class and using the metric(s) provided.

If forecaster is already trained and initial_train_size is set to None in the TimeSeriesFold class, no initial train will be done and all data will be used to evaluate the model. However, the first len(forecaster.last_window) observations are needed to create the initial predictors, so no predictions are calculated for them.

A copy of the original forecaster is created so that it is not modified during the process.

Parameters:

Name	Type	Description	Default
forecaster	(ForecasterRecursive, ForecasterDirect, ForecasterEquivalentDate, ForecasterRecursiveClassifier)	Forecaster model.	required
y	pandas Series	Training time series.	required
cv	TimeSeriesFold	TimeSeriesFold object with the information needed to split the data into folds. Its window_size is taken from the forecaster, and the object passed is not modified.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: name of the metric, for example 'mean_absolute_error' or 'root_mean_squared_error'. All accepted names are listed in https://skforecast.org/latest/user_guides/metrics.html. If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. y_train is the series before the first forecast origin. mean_absolute_scaled_error and root_mean_squared_scaled_error scale the error of each prediction with the one-step naive forecast over the observations before its own forecast origin, the same for every forecaster. If list: List containing multiple strings and/or Callables.	required
exog	pandas Series, pandas DataFrame	Exogenous variable/s included as predictor/s. Must have the same number of observations as y and should be aligned so that y[i] is regressed on exog[i].	None
interval	(float, list, tuple, str, object)	Specifies whether probabilistic predictions should be estimated and the method to use. The following options are supported: If float, represents the nominal (expected) coverage (between 0 and 1). For instance, interval=0.95 corresponds to [0.025, 0.975] quantiles. If list or tuple: Sequence of quantiles to compute, each value must be between 0 and 1 inclusive. For example, a 95% confidence interval can be specified as interval = [0.025, 0.975] or multiple quantiles (e.g. 0.1, 0.5 and 0.9) as interval = [0.1, 0.5, 0.9]. If 'bootstrapping' (str): n_boot bootstrapping predictions will be generated. If scipy.stats distribution object, the distribution parameters will be estimated for each prediction. If None, no probabilistic predictions are estimated. Changed in version 0.23.0: interval is now expressed as quantiles (0-1) instead of percentiles (0-100). Passing percentiles is not longer supported and will raise a ValueError.	None
interval_method	str	Technique used to estimate prediction intervals. Available options: 'bootstrapping': Bootstrapping is used to generate prediction intervals [1]_. 'conformal': Employs the conformal prediction split method for interval estimation [2]_.	'bootstrapping'
n_boot	int	Number of bootstrapping iterations to perform when estimating prediction intervals.	250
use_in_sample_residuals	bool	If True, residuals from the training data are used as proxy of prediction error to create predictions. If False, out of sample residuals (calibration) are used. Out-of-sample residuals must be precomputed using Forecaster's set_out_sample_residuals() method.	True
use_binned_residuals	bool	If True, residuals are selected based on the predicted values (binned selection). If False, residuals are selected randomly.	True
random_state	int	Seed for the random number generator to ensure reproducibility.	123
return_predictors	bool	If True, the predictors used to make the predictions are also returned.	False
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.model_selection._utils.select_n_jobs_backtesting, and set to 1 when the workload of the folds, estimated from the time of the initial fit, is too small to pay the start of the workers (for example, without refit). See skforecast.model_selection._utils.limit_n_jobs_by_workload.	'auto'
verbose	bool	Print number of folds and index of training and validation sets used for backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the backtesting process. See skforecast.exceptions.warn_skforecast_categories for more information.	False

Returns:

Name	Type	Description
metric_values	pandas DataFrame	Value(s) of the metric(s). If interval has a lower and an upper bound (a float, or a list or tuple of 2 quantiles, optionally with the median 0.5), two more columns are added after the metrics: coverage: share of true values inside the interval (bounds included). interval_width: mean of upper_bound - lower_bound, in the units of the series. The last column, n_evaluated, is the number of points used to compute the metrics: true value and prediction not missing, each date counted once (with the prediction of its last fold when the folds overlap). New in version 0.27.0
backtest_predictions	pandas DataFrame	Value of predictions. The DataFrame includes the following columns, in this order: fold: Indicates the fold number where the prediction was made. step: Step of the forecast horizon at which the prediction was made, from gap + 1 to gap + steps. y_true: True value of the series (NaN if it is missing). pred: Predicted values for the corresponding series and time steps. With a ForecasterRecursiveClassifier, the predicted class, followed by one column with the probability of each class (e.g., a_proba). New in version 0.27.0: columns step and y_true. If interval is not None, additional columns are included depending on the method: For float: Columns lower_bound and upper_bound. For list or tuple of 2 elements: Columns lower_bound and upper_bound. For list or tuple with multiple quantiles: One column per quantile (e.g., q_0.1, q_0.5, q_0.9). For 'bootstrapping': One column per bootstrapping iteration (e.g., pred_boot_0, pred_boot_1, ..., pred_boot_n). For scipy.stats distribution objects: One column for each estimated parameter of the distribution (e.g., loc, scale). If return_predictors is True, one column per predictor is added after the predictions, with the names of X_train_features_names_out_. Select them by name, for example with backtest_predictions[forecaster.X_train_features_names_out_] once the forecaster is fitted, not by position. Depending on the relation between steps and fold_stride, the output may include repeated indexes (if fold_stride < steps) or gaps (if fold_stride > steps). See Notes below for more details.

Notes

Note on fold_stride vs. steps:

• If fold_stride == steps, test sets are placed back-to-back without overlap. Each observation appears only once in the output DataFrame, so the index is unique.
• If fold_stride < steps, test sets overlap. Multiple forecasts are generated for the same observations and, therefore, the output DataFrame contains repeated indexes.
• If fold_stride > steps, there are gaps between consecutive test sets. Some observations in the series will not have associated predictions, so the output DataFrame has non-contiguous indexes.

References

.. [1] Forecasting: Principles and Practice (3^rd ed) Rob J Hyndman and George Athanasopoulos. https://otexts.com/fpp3/prediction-intervals.html

.. [2] MAPIE - Model Agnostic Prediction Interval Estimator. https://mapie.readthedocs.io/en/stable/content/conformal-prediction/regression/#2-the-split-method

Source code in skforecast/model_selection/_validation.py

def backtesting_forecaster(
    forecaster: object,
    y: pd.Series,
    cv: TimeSeriesFold,
    metric: str | Callable | list[str | Callable],
    exog: pd.Series | pd.DataFrame | None = None,
    interval: float | list[float] | tuple[float, ...] | str | object | None = None,
    interval_method: str = 'bootstrapping',
    n_boot: int = 250,
    use_in_sample_residuals: bool = True,
    use_binned_residuals: bool = True,
    random_state: int = 123,
    return_predictors: bool = False,
    n_jobs: int | str = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False
) -> tuple[pd.DataFrame, pd.DataFrame]:
    """
    Backtesting of forecaster model following the folds generated by the TimeSeriesFold
    class and using the metric(s) provided.

    If `forecaster` is already trained and `initial_train_size` is set to `None` in the
    TimeSeriesFold class, no initial train will be done and all data will be used
    to evaluate the model. However, the first `len(forecaster.last_window)` observations
    are needed to create the initial predictors, so no predictions are calculated for
    them.

    A copy of the original forecaster is created so that it is not modified during 
    the process.

    Parameters
    ----------
    forecaster : ForecasterRecursive, ForecasterDirect, ForecasterEquivalentDate, ForecasterRecursiveClassifier
        Forecaster model.
    y : pandas Series
        Training time series.
    cv : TimeSeriesFold
        TimeSeriesFold object with the information needed to split the data into
        folds. Its `window_size` is taken from the forecaster, and the object
        passed is not modified.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: name of the metric, for example `'mean_absolute_error'`
        or `'root_mean_squared_error'`. All accepted names are listed in
        https://skforecast.org/latest/user_guides/metrics.html.
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float. `y_train` is the series before the
        first forecast origin. `mean_absolute_scaled_error` and
        `root_mean_squared_scaled_error` scale the error of each prediction
        with the one-step naive forecast over the observations before its own
        forecast origin, the same for every forecaster.
        - If `list`: List containing multiple strings and/or Callables.
    exog : pandas Series, pandas DataFrame, default None
        Exogenous variable/s included as predictor/s. Must have the same
        number of observations as `y` and should be aligned so that y[i] is
        regressed on exog[i].
    interval : float, list, tuple, str, object, default None
        Specifies whether probabilistic predictions should be estimated and the 
        method to use. The following options are supported:

        - If `float`, represents the nominal (expected) coverage (between 0 and 1). 
        For instance, `interval=0.95` corresponds to `[0.025, 0.975]` quantiles.
        - If `list` or `tuple`: Sequence of quantiles to compute, each value must 
        be between 0 and 1 inclusive. For example, a 95% confidence interval can 
        be specified as `interval = [0.025, 0.975]` or multiple quantiles (e.g. 0.1, 
        0.5 and 0.9) as `interval = [0.1, 0.5, 0.9]`.
        - If 'bootstrapping' (str): `n_boot` bootstrapping predictions will be generated.
        - If scipy.stats distribution object, the distribution parameters will
        be estimated for each prediction.
        - If None, no probabilistic predictions are estimated.

        **Changed in version 0.23.0:** `interval` is now expressed as
        quantiles (0-1) instead of percentiles (0-100). Passing percentiles
        is not longer supported and will raise a `ValueError`.
    interval_method : str, default 'bootstrapping'
        Technique used to estimate prediction intervals. Available options:

        - 'bootstrapping': Bootstrapping is used to generate prediction 
        intervals [1]_.
        - 'conformal': Employs the conformal prediction split method for 
        interval estimation [2]_.
    n_boot : int, default 250
        Number of bootstrapping iterations to perform when estimating prediction
        intervals.
    use_in_sample_residuals : bool, default True
        If `True`, residuals from the training data are used as proxy of
        prediction error to create predictions. 
        If `False`, out of sample residuals (calibration) are used. 
        Out-of-sample residuals must be precomputed using Forecaster's
        `set_out_sample_residuals()` method.
    use_binned_residuals : bool, default True
        If `True`, residuals are selected based on the predicted values 
        (binned selection).
        If `False`, residuals are selected randomly.
    random_state : int, default 123
        Seed for the random number generator to ensure reproducibility.
    return_predictors : bool, default False
        If `True`, the predictors used to make the predictions are also returned.
    n_jobs : int, 'auto', default 'auto'
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.model_selection._utils.select_n_jobs_backtesting, and set to 1
        when the workload of the folds, estimated from the time of the initial fit, is
        too small to pay the start of the workers (for example, without refit).
        See skforecast.model_selection._utils.limit_n_jobs_by_workload.
    verbose : bool, default False
        Print number of folds and index of training and validation sets used 
        for backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings : bool, default False
        If `True`, skforecast warnings will be suppressed during the backtesting 
        process. See skforecast.exceptions.warn_skforecast_categories for more
        information.

    Returns
    -------
    metric_values : pandas DataFrame
        Value(s) of the metric(s). If `interval` has a lower and an upper bound
        (a `float`, or a `list` or `tuple` of 2 quantiles, optionally with the
        median 0.5), two more columns are added after the metrics:

        - coverage: share of true values inside the interval (bounds included).
        - interval_width: mean of `upper_bound - lower_bound`, in the units of
        the series.

        The last column, `n_evaluated`, is the number of points used to
        compute the metrics: true value and prediction not missing, each date
        counted once (with the prediction of its last fold when the folds
        overlap).

        **New in version 0.27.0**
    backtest_predictions : pandas DataFrame
        Value of predictions. The DataFrame includes the following columns, in
        this order:

        - fold: Indicates the fold number where the prediction was made.
        - step: Step of the forecast horizon at which the prediction was made,
        from `gap + 1` to `gap + steps`.
        - y_true: True value of the series (NaN if it is missing).
        - pred: Predicted values for the corresponding series and time steps.
        With a `ForecasterRecursiveClassifier`, the predicted class, followed
        by one column with the probability of each class (e.g., `a_proba`).

        **New in version 0.27.0**: columns `step` and `y_true`.

        If `interval` is not `None`, additional columns are included depending on the method:

        - For `float`: Columns `lower_bound` and `upper_bound`.
        - For `list` or `tuple` of 2 elements: Columns `lower_bound` and `upper_bound`.
        - For `list` or `tuple` with multiple quantiles: One column per quantile 
        (e.g., `q_0.1`, `q_0.5`, `q_0.9`).
        - For `'bootstrapping'`: One column per bootstrapping iteration 
        (e.g., `pred_boot_0`, `pred_boot_1`, ..., `pred_boot_n`).
        - For `scipy.stats` distribution objects: One column for each estimated 
        parameter of the distribution (e.g., `loc`, `scale`).

        If `return_predictors` is `True`, one column per predictor is added
        after the predictions, with the names of `X_train_features_names_out_`.
        Select them by name, for example with
        `backtest_predictions[forecaster.X_train_features_names_out_]` once the
        forecaster is fitted, not by position.

        Depending on the relation between `steps` and `fold_stride`, the output
        may include repeated indexes (if `fold_stride < steps`) or gaps
        (if `fold_stride > steps`). See Notes below for more details.

    Notes
    -----
    Note on `fold_stride` vs. `steps`:

    - If `fold_stride == steps`, test sets are placed back-to-back without overlap. 
    Each observation appears only once in the output DataFrame, so the index is unique.
    - If `fold_stride < steps`, test sets overlap. Multiple forecasts are generated 
    for the same observations and, therefore, the output DataFrame contains repeated 
    indexes.
    - If `fold_stride > steps`, there are gaps between consecutive test sets. 
    Some observations in the series will not have associated predictions, so 
    the output DataFrame has non-contiguous indexes.

    References
    ----------
    .. [1] Forecasting: Principles and Practice (3rd ed) Rob J Hyndman and George Athanasopoulos.
           https://otexts.com/fpp3/prediction-intervals.html

    .. [2] MAPIE - Model Agnostic Prediction Interval Estimator.
           https://mapie.readthedocs.io/en/stable/content/conformal-prediction/regression/#2-the-split-method

    """

    forecasters_allowed = [
        'ForecasterRecursive', 
        'ForecasterDirect',
        'ForecasterEquivalentDate',
        'ForecasterRecursiveClassifier'
    ]

    if type(forecaster).__name__ not in forecasters_allowed:
        raise TypeError(
            f"`forecaster` must be of type {forecasters_allowed}. For all other "
            f"types of forecasters use the other functions available in the "
            f"`model_selection` module."
        )

    check_backtesting_input(
        forecaster              = forecaster,
        cv                      = cv,
        y                       = y,
        metric                  = metric,
        interval                = interval,
        interval_method         = interval_method,
        n_boot                  = n_boot,
        use_in_sample_residuals = use_in_sample_residuals,
        use_binned_residuals    = use_binned_residuals,
        random_state            = random_state,
        return_predictors       = return_predictors,
        n_jobs                  = n_jobs,
        show_progress           = show_progress,
        suppress_warnings       = suppress_warnings
    )

    metric_values, backtest_predictions = _backtesting_forecaster(
        forecaster              = forecaster,
        y                       = y,
        cv                      = cv,
        metric                  = metric,
        exog                    = exog,
        interval                = interval,
        interval_method         = interval_method,
        n_boot                  = n_boot,
        use_in_sample_residuals = use_in_sample_residuals,
        use_binned_residuals    = use_binned_residuals,
        random_state            = random_state,
        return_predictors       = return_predictors,
        n_jobs                  = n_jobs,
        verbose                 = verbose,
        show_progress           = show_progress,
        suppress_warnings       = suppress_warnings
    )

    return metric_values, backtest_predictions

skforecast.model_selection._search.grid_search_forecaster

grid_search_forecaster(
    forecaster,
    y,
    cv,
    param_grid,
    metric,
    exog=None,
    lags_grid=None,
    return_best=True,
    n_jobs="auto",
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
    output_file=None,
)

Exhaustive search over specified parameter values for a Forecaster object. Validation is done using time series backtesting.

Parameters:

Name	Type	Description	Default
forecaster	(ForecasterRecursive, ForecasterDirect)	Forecaster model.	required
y	pandas Series	Training time series.	required
cv	(TimeSeriesFold, OneStepAheadFold)	TimeSeriesFold or OneStepAheadFold object with the information needed to split the data into folds. With a TimeSeriesFold, its window_size is taken from the forecaster; with a OneStepAheadFold, only its initial_train_size is used. The object passed is not modified.	required
param_grid	dict	Dictionary with parameters names (str) as keys and lists of parameter settings to try as values.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: name of the metric, for example 'mean_absolute_error' or 'root_mean_squared_error'. All accepted names are listed in https://skforecast.org/latest/user_guides/metrics.html. If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. y_train is the series before the first forecast origin. mean_absolute_scaled_error and root_mean_squared_scaled_error scale the error of each prediction with the one-step naive forecast over the observations before its own forecast origin, the same for every forecaster. If list: List containing multiple strings and/or Callables. The first metric ranks the candidates. Scikit-learn metrics ending in _score are maximized and those ending in _error, _loss or _deviance are minimized. Other functions are minimized in regression and maximized in classification, unless their attribute greater_is_better is set (True or False).	required
exog	pandas Series, pandas DataFrame	Exogenous variable/s included as predictor/s. Must have the same number of observations as y and should be aligned so that y[i] is regressed on exog[i].	None
lags_grid	(list, dict)	Lists of lags to try, containing int, lists, numpy ndarray, range objects or None. A None element evaluates the forecaster without lags, using only its window_features (a ValueError is raised if it has none). If dict, the keys are used as labels in the results DataFrame, and the values are used as the lists of lags to try. If None, the lags of the forecaster are used.	None
return_best	bool	Refit the forecaster using the best found parameters on the whole data.	True
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.model_selection._utils.select_n_jobs_backtesting. Ignored for OneStepAheadFold.	'auto'
verbose	bool	Print number of folds used for cv or backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information. Otherwise, the warnings of the candidates are shown once, at the end of the search, with the number of candidates in which they are raised. The candidates skipped because they raise an error are always reported, in a single warning.	False
output_file	str	Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file.	None

Returns:

Name	Type	Description
results	pandas DataFrame	Results for each combination of parameters. column lags: lags configuration for each iteration. column lags_label: descriptive label or alias for the lags. column params: parameters configuration for each iteration. column metric: metric value estimated for each iteration. column n_evaluated: number of points that enter the metrics of each iteration (true value and prediction not missing). A MissingValuesWarning is raised at the end of the search if some iterations evaluate less than 95% of the points of the most complete one, whose metrics are not comparable (with a OneStepAheadFold, when dropna_from_series=True drops rows with missing values). additional n columns with param = value.

Source code in skforecast/model_selection/_search.py

def grid_search_forecaster(
    forecaster: object,
    y: pd.Series,
    cv: TimeSeriesFold | OneStepAheadFold,
    param_grid: dict,
    metric: str | Callable | list[str | Callable],
    exog: pd.Series | pd.DataFrame | None = None,
    lags_grid: (
        list[int | list[int] | np.ndarray | range | None]
        | dict[str, int | list[int] | np.ndarray | range | None]
        | None
    ) = None,
    return_best: bool = True,
    n_jobs: int | str = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: str | None = None
) -> pd.DataFrame:
    """
    Exhaustive search over specified parameter values for a Forecaster object.
    Validation is done using time series backtesting.

    Parameters
    ----------
    forecaster : ForecasterRecursive, ForecasterDirect
        Forecaster model.
    y : pandas Series
        Training time series.
    cv : TimeSeriesFold, OneStepAheadFold
        TimeSeriesFold or OneStepAheadFold object with the information needed to split
        the data into folds. With a `TimeSeriesFold`, its `window_size` is taken
        from the forecaster; with a `OneStepAheadFold`, only its `initial_train_size`
        is used. The object passed is not modified.
    param_grid : dict
        Dictionary with parameters names (`str`) as keys and lists of parameter
        settings to try as values.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: name of the metric, for example `'mean_absolute_error'`
        or `'root_mean_squared_error'`. All accepted names are listed in
        https://skforecast.org/latest/user_guides/metrics.html.
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float. `y_train` is the series before the
        first forecast origin. `mean_absolute_scaled_error` and
        `root_mean_squared_scaled_error` scale the error of each prediction
        with the one-step naive forecast over the observations before its own
        forecast origin, the same for every forecaster.
        - If `list`: List containing multiple strings and/or Callables.

        The first metric ranks the candidates. Scikit-learn metrics ending in
        `_score` are maximized and those ending in `_error`, `_loss` or
        `_deviance` are minimized. Other functions are minimized in regression
        and maximized in classification, unless their attribute
        `greater_is_better` is set (`True` or `False`).
    exog : pandas Series, pandas DataFrame, default None
        Exogenous variable/s included as predictor/s. Must have the same
        number of observations as `y` and should be aligned so that y[i] is
        regressed on exog[i].
    lags_grid : list, dict, default None
        Lists of lags to try, containing int, lists, numpy ndarray, range
        objects or `None`. A `None` element evaluates the forecaster without
        lags, using only its `window_features` (a `ValueError` is raised if
        it has none). If `dict`, the keys are used as labels in the `results`
        DataFrame, and the values are used as the lists of lags to try. If
        `None`, the lags of the forecaster are used.
    return_best : bool, default True
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default 'auto'
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.model_selection._utils.select_n_jobs_backtesting. Ignored for
        `OneStepAheadFold`.
    verbose : bool, default False
        Print number of folds used for cv or backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings : bool, default False
        If `True`, skforecast warnings will be suppressed during the hyperparameter 
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.
        Otherwise, the warnings of the candidates are shown once, at the end
        of the search, with the number of candidates in which they are
        raised. The candidates skipped because they raise an error are always
        reported, in a single warning.
    output_file : str, default None
        Specifies the filename or full path where the results should be saved. 
        The results will be saved in a tab-separated values (TSV) format. If 
        `None`, the results will not be saved to a file.

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column lags: lags configuration for each iteration.
        - column lags_label: descriptive label or alias for the lags.
        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration.
        - column n_evaluated: number of points that enter the metrics of each
        iteration (true value and prediction not missing). A
        `MissingValuesWarning` is raised at the end of the search if some
        iterations evaluate less than 95% of the points of the most complete
        one, whose metrics are not comparable (with a `OneStepAheadFold`, when
        `dropna_from_series=True` drops rows with missing values).
        - additional n columns with param = value.

    """

    param_grid = list(ParameterGrid(param_grid))

    results = _evaluate_grid_hyperparameters(
                  forecaster        = forecaster,
                  y                 = y,
                  cv                = cv,
                  param_grid        = param_grid,
                  metric            = metric,
                  exog              = exog,
                  lags_grid         = lags_grid,
                  return_best       = return_best,
                  n_jobs            = n_jobs,
                  verbose           = verbose,
                  show_progress     = show_progress,
                  suppress_warnings = suppress_warnings,
                  output_file       = output_file
              )

    return results

skforecast.model_selection._search.random_search_forecaster

random_search_forecaster(
    forecaster,
    y,
    cv,
    param_distributions,
    metric,
    exog=None,
    lags_grid=None,
    n_iter=10,
    random_state=123,
    return_best=True,
    n_jobs="auto",
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
    output_file=None,
)

Random search over specified parameter values or distributions for a Forecaster object. Validation is done using time series backtesting.

Parameters:

Name	Type	Description	Default
forecaster	(ForecasterRecursive, ForecasterDirect)	Forecaster model.	required
y	pandas Series	Training time series.	required
cv	(TimeSeriesFold, OneStepAheadFold)	TimeSeriesFold or OneStepAheadFold object with the information needed to split the data into folds. With a TimeSeriesFold, its window_size is taken from the forecaster; with a OneStepAheadFold, only its initial_train_size is used. The object passed is not modified.	required
param_distributions	dict	Dictionary with parameters names (str) as keys and distributions or lists of parameters to try.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: name of the metric, for example 'mean_absolute_error' or 'root_mean_squared_error'. All accepted names are listed in https://skforecast.org/latest/user_guides/metrics.html. If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. y_train is the series before the first forecast origin. mean_absolute_scaled_error and root_mean_squared_scaled_error scale the error of each prediction with the one-step naive forecast over the observations before its own forecast origin, the same for every forecaster. If list: List containing multiple strings and/or Callables. The first metric ranks the candidates. Scikit-learn metrics ending in _score are maximized and those ending in _error, _loss or _deviance are minimized. Other functions are minimized in regression and maximized in classification, unless their attribute greater_is_better is set (True or False).	required
exog	pandas Series, pandas DataFrame	Exogenous variable/s included as predictor/s. Must have the same number of observations as y and should be aligned so that y[i] is regressed on exog[i].	None
lags_grid	(list, dict)	Lists of lags to try, containing int, lists, numpy ndarray, range objects or None. A None element evaluates the forecaster without lags, using only its window_features (a ValueError is raised if it has none). If dict, the keys are used as labels in the results DataFrame, and the values are used as the lists of lags to try. If None, the lags of the forecaster are used.	None
n_iter	int	Number of parameter settings that are sampled per lags configuration. n_iter trades off runtime vs quality of the solution.	10
random_state	int	Sets a seed to the random sampling for reproducible output.	123
return_best	bool	Refit the forecaster using the best found parameters on the whole data.	True
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.model_selection._utils.select_n_jobs_backtesting. Ignored for OneStepAheadFold.	'auto'
verbose	bool	Print number of folds used for cv or backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information. Otherwise, the warnings of the candidates are shown once, at the end of the search, with the number of candidates in which they are raised. The candidates skipped because they raise an error are always reported, in a single warning.	False
output_file	str	Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file.	None

Returns:

Name	Type	Description
results	pandas DataFrame	Results for each combination of parameters. column lags: lags configuration for each iteration. column lags_label: descriptive label or alias for the lags. column params: parameters configuration for each iteration. column metric: metric value estimated for each iteration. column n_evaluated: number of points that enter the metrics of each iteration (true value and prediction not missing). A MissingValuesWarning is raised at the end of the search if some iterations evaluate less than 95% of the points of the most complete one, whose metrics are not comparable (with a OneStepAheadFold, when dropna_from_series=True drops rows with missing values). additional n columns with param = value.

Source code in skforecast/model_selection/_search.py

def random_search_forecaster(
    forecaster: object,
    y: pd.Series,
    cv: TimeSeriesFold | OneStepAheadFold,
    param_distributions: dict,
    metric: str | Callable | list[str | Callable],
    exog: pd.Series | pd.DataFrame | None = None,
    lags_grid: (
        list[int | list[int] | np.ndarray | range | None]
        | dict[str, int | list[int] | np.ndarray | range | None]
        | None
    ) = None,
    n_iter: int = 10,
    random_state: int = 123,
    return_best: bool = True,
    n_jobs: int | str = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: str | None = None
) -> pd.DataFrame:
    """
    Random search over specified parameter values or distributions for a Forecaster 
    object. Validation is done using time series backtesting.

    Parameters
    ----------
    forecaster : ForecasterRecursive, ForecasterDirect
        Forecaster model.
    y : pandas Series
        Training time series.
    cv : TimeSeriesFold, OneStepAheadFold
        TimeSeriesFold or OneStepAheadFold object with the information needed to split
        the data into folds. With a `TimeSeriesFold`, its `window_size` is taken
        from the forecaster; with a `OneStepAheadFold`, only its `initial_train_size`
        is used. The object passed is not modified.
    param_distributions : dict
        Dictionary with parameters names (`str`) as keys and 
        distributions or lists of parameters to try.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: name of the metric, for example `'mean_absolute_error'`
        or `'root_mean_squared_error'`. All accepted names are listed in
        https://skforecast.org/latest/user_guides/metrics.html.
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float. `y_train` is the series before the
        first forecast origin. `mean_absolute_scaled_error` and
        `root_mean_squared_scaled_error` scale the error of each prediction
        with the one-step naive forecast over the observations before its own
        forecast origin, the same for every forecaster.
        - If `list`: List containing multiple strings and/or Callables.

        The first metric ranks the candidates. Scikit-learn metrics ending in
        `_score` are maximized and those ending in `_error`, `_loss` or
        `_deviance` are minimized. Other functions are minimized in regression
        and maximized in classification, unless their attribute
        `greater_is_better` is set (`True` or `False`).
    exog : pandas Series, pandas DataFrame, default None
        Exogenous variable/s included as predictor/s. Must have the same
        number of observations as `y` and should be aligned so that y[i] is
        regressed on exog[i]. 
    lags_grid : list, dict, default None
        Lists of lags to try, containing int, lists, numpy ndarray, range
        objects or `None`. A `None` element evaluates the forecaster without
        lags, using only its `window_features` (a `ValueError` is raised if
        it has none). If `dict`, the keys are used as labels in the `results`
        DataFrame, and the values are used as the lists of lags to try. If
        `None`, the lags of the forecaster are used.
    n_iter : int, default 10
        Number of parameter settings that are sampled per lags configuration. 
        n_iter trades off runtime vs quality of the solution.
    random_state : int, default 123
        Sets a seed to the random sampling for reproducible output.
    return_best : bool, default True
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default 'auto'
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.model_selection._utils.select_n_jobs_backtesting. Ignored for
        `OneStepAheadFold`.
    verbose : bool, default False
        Print number of folds used for cv or backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings : bool, default False
        If `True`, skforecast warnings will be suppressed during the hyperparameter 
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.
        Otherwise, the warnings of the candidates are shown once, at the end
        of the search, with the number of candidates in which they are
        raised. The candidates skipped because they raise an error are always
        reported, in a single warning.
    output_file : str, default None
        Specifies the filename or full path where the results should be saved. 
        The results will be saved in a tab-separated values (TSV) format. If 
        `None`, the results will not be saved to a file.

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column lags: lags configuration for each iteration.
        - column lags_label: descriptive label or alias for the lags.
        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration.
        - column n_evaluated: number of points that enter the metrics of each
        iteration (true value and prediction not missing). A
        `MissingValuesWarning` is raised at the end of the search if some
        iterations evaluate less than 95% of the points of the most complete
        one, whose metrics are not comparable (with a `OneStepAheadFold`, when
        `dropna_from_series=True` drops rows with missing values).
        - additional n columns with param = value.

    """

    param_grid = _sample_param_grid(
                     param_distributions = param_distributions,
                     n_iter              = n_iter,
                     random_state        = random_state,
                     grid_search_name    = 'grid_search_forecaster',
                     lags_grid           = lags_grid is not None
                 )

    results = _evaluate_grid_hyperparameters(
                  forecaster        = forecaster,
                  y                 = y,
                  cv                = cv,
                  param_grid        = param_grid,
                  metric            = metric,
                  exog              = exog,
                  lags_grid         = lags_grid,
                  return_best       = return_best,
                  n_jobs            = n_jobs,
                  verbose           = verbose,
                  show_progress     = show_progress,
                  suppress_warnings = suppress_warnings,
                  output_file       = output_file
              )

    return results

skforecast.model_selection._search.bayesian_search_forecaster

bayesian_search_forecaster(
    forecaster,
    y,
    cv,
    search_space,
    metric,
    exog=None,
    n_trials=20,
    random_state=123,
    return_best=True,
    n_jobs="auto",
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
    output_file=None,
    kwargs_create_study=None,
    kwargs_study_optimize=None,
)

Bayesian search for hyperparameters of a Forecaster object using optuna library.

Parameters:

Name	Type	Description	Default
forecaster	(ForecasterRecursive, ForecasterDirect)	Forecaster model.	required
y	pandas Series	Training time series.	required
cv	(TimeSeriesFold, OneStepAheadFold)	TimeSeriesFold or OneStepAheadFold object with the information needed to split the data into folds. With a TimeSeriesFold, its window_size is taken from the forecaster; with a OneStepAheadFold, only its initial_train_size is used. The object passed is not modified.	required
search_space	Callable(optuna)	Function with argument trial which returns a dictionary with parameters names (str) as keys and Trial object from optuna (trial.suggest_float, trial.suggest_int, trial.suggest_categorical) as values.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: name of the metric, for example 'mean_absolute_error' or 'root_mean_squared_error'. All accepted names are listed in https://skforecast.org/latest/user_guides/metrics.html. If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. y_train is the series before the first forecast origin. mean_absolute_scaled_error and root_mean_squared_scaled_error scale the error of each prediction with the one-step naive forecast over the observations before its own forecast origin, the same for every forecaster. If list: List containing multiple strings and/or Callables. The first metric ranks the candidates. Scikit-learn metrics ending in _score are maximized and those ending in _error, _loss or _deviance are minimized. Other functions are minimized in regression and maximized in classification, unless their attribute greater_is_better is set (True or False).	required
exog	pandas Series, pandas DataFrame	Exogenous variable/s included as predictor/s. Must have the same number of observations as y and should be aligned so that y[i] is regressed on exog[i].	None
n_trials	int	Number of parameter settings (configurations) sampled during the search. With the default TPESampler, the first n_startup_trials (10 by default) are drawn by random sampling to seed the search, and the TPE algorithm only guides sampling from the 11th trial onward. For meaningful Bayesian optimization, n_trials should therefore be clearly larger than 10 (for example 30 to 50 or more). This threshold changes if a custom sampler with a different n_startup_trials is passed in kwargs_create_study. Larger values increase run time, since each trial triggers a full evaluation.	20
random_state	int	Seed for the TPESampler for reproducible output. When a custom sampler is passed in kwargs_create_study, the seed must be set within that sampler. For example {'sampler': TPESampler(seed=145)}.	123
return_best	bool	Refit the forecaster using the best found parameters on the whole data.	True
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.model_selection._utils.select_n_jobs_backtesting. Ignored for OneStepAheadFold.	'auto'
verbose	bool	Print number of folds used for cv or backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information. Otherwise, the warnings of the trials are shown once, at the end of the search, with the number of trials in which they are raised. The trials skipped because they raise an error are always reported, in a single warning.	False
output_file	str	Specifies the filename or full path where the optuna logging output should be saved. If None, logging output will not be redirected to a file.	None
kwargs_create_study	dict	Additional keyword arguments (key, value mappings) to pass to optuna.create_study(). If default, the direction is inferred from the first metric (see metric), and a TPESampler(multivariate=True, group=True, seed=random_state) sampler is used during optimization. If direction is passed, the order of results and the parameters selected with return_best follow it (a warning is raised if it contradicts the direction of the metric).	None
kwargs_study_optimize	dict	Additional keyword arguments (key, value mappings) to pass to study.optimize(). The trials whose evaluation raises an error are marked as failed and skipped, also when catch is passed (its errors are kept). An error in search_space stops the search.	None

Returns:

Name	Type	Description
results	pandas DataFrame	Results for each combination of parameters. column trial_number: optuna trial number for each iteration. Use study.trials[trial_number] to access the full optuna trial object. column lags: lags configuration for each iteration. column params: parameters configuration for each iteration. column metric: metric value estimated for each iteration. column n_evaluated: number of points that enter the metrics of each iteration (true value and prediction not missing). A MissingValuesWarning is raised at the end of the search if some iterations evaluate less than 95% of the points of the most complete one, whose metrics are not comparable (with a OneStepAheadFold, when dropna_from_series=True drops rows with missing values). additional n columns with param = value.
study	optuna Study	The optuna study object containing all optimization trials. Access the best trial via study.best_trial.

Source code in skforecast/model_selection/_search.py

@manage_warnings
def bayesian_search_forecaster(
    forecaster: object,
    y: pd.Series,
    cv: TimeSeriesFold | OneStepAheadFold,
    search_space: Callable,
    metric: str | Callable | list[str | Callable],
    exog: pd.Series | pd.DataFrame | None = None,
    n_trials: int = 20,
    random_state: int = 123,
    return_best: bool = True,
    n_jobs: int | str = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: str | None = None,
    kwargs_create_study: dict | None = None,
    kwargs_study_optimize: dict | None = None
) -> tuple[pd.DataFrame, object]:
    """
    Bayesian search for hyperparameters of a Forecaster object using optuna library.

    Parameters
    ----------
    forecaster : ForecasterRecursive, ForecasterDirect
        Forecaster model.
    y : pandas Series
        Training time series.
    cv : TimeSeriesFold, OneStepAheadFold
        TimeSeriesFold or OneStepAheadFold object with the information needed to split
        the data into folds. With a `TimeSeriesFold`, its `window_size` is taken
        from the forecaster; with a `OneStepAheadFold`, only its `initial_train_size`
        is used. The object passed is not modified.
    search_space : Callable (optuna)
        Function with argument `trial` which returns a dictionary with parameters names 
        (`str`) as keys and Trial object from optuna (trial.suggest_float, 
        trial.suggest_int, trial.suggest_categorical) as values.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: name of the metric, for example `'mean_absolute_error'`
        or `'root_mean_squared_error'`. All accepted names are listed in
        https://skforecast.org/latest/user_guides/metrics.html.
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float. `y_train` is the series before the
        first forecast origin. `mean_absolute_scaled_error` and
        `root_mean_squared_scaled_error` scale the error of each prediction
        with the one-step naive forecast over the observations before its own
        forecast origin, the same for every forecaster.
        - If `list`: List containing multiple strings and/or Callables.

        The first metric ranks the candidates. Scikit-learn metrics ending in
        `_score` are maximized and those ending in `_error`, `_loss` or
        `_deviance` are minimized. Other functions are minimized in regression
        and maximized in classification, unless their attribute
        `greater_is_better` is set (`True` or `False`).
    exog : pandas Series, pandas DataFrame, default None
        Exogenous variable/s included as predictor/s. Must have the same
        number of observations as `y` and should be aligned so that y[i] is
        regressed on exog[i].
    n_trials : int, default 20
        Number of parameter settings (configurations) sampled during the search.
        With the default `TPESampler`, the first `n_startup_trials` (10 by
        default) are drawn by random sampling to seed the search, and the TPE
        algorithm only guides sampling from the 11th trial onward. For meaningful
        Bayesian optimization, `n_trials` should therefore be clearly larger than
        10 (for example 30 to 50 or more). This threshold changes if a custom
        sampler with a different `n_startup_trials` is passed in
        `kwargs_create_study`. Larger values increase run time, since each trial
        triggers a full evaluation.
    random_state : int, default 123
        Seed for the `TPESampler` for reproducible output. When a custom sampler
        is passed in `kwargs_create_study`, the seed must be set within that
        sampler. For example `{'sampler': TPESampler(seed=145)}`.
    return_best : bool, default True
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default 'auto'
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.model_selection._utils.select_n_jobs_backtesting. Ignored for
        `OneStepAheadFold`.
    verbose : bool, default False
        Print number of folds used for cv or backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings : bool, default False
        If `True`, skforecast warnings will be suppressed during the hyperparameter 
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.
        Otherwise, the warnings of the trials are shown once, at the end of
        the search, with the number of trials in which they are raised. The
        trials skipped because they raise an error are always reported, in a
        single warning.
    output_file : str, default None
        Specifies the filename or full path where the optuna logging output
        should be saved. If `None`, logging output will not be redirected to
        a file.
    kwargs_create_study : dict, default None
        Additional keyword arguments (key, value mappings) to pass to optuna.create_study().
        If default, the direction is inferred from the first metric (see
        `metric`), and a `TPESampler(multivariate=True, group=True, seed=random_state)`
        sampler is used during optimization. If `direction` is passed, the order
        of `results` and the parameters selected with `return_best` follow it
        (a warning is raised if it contradicts the direction of the metric).
    kwargs_study_optimize : dict, default None
        Additional keyword arguments (key, value mappings) to pass to study.optimize().
        The trials whose evaluation raises an error are marked as failed and
        skipped, also when `catch` is passed (its errors are kept). An error in
        `search_space` stops the search.

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column trial_number: optuna trial number for each iteration. Use
          `study.trials[trial_number]` to access the full optuna trial object.
        - column lags: lags configuration for each iteration.
        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration.
        - column n_evaluated: number of points that enter the metrics of each
        iteration (true value and prediction not missing). A
        `MissingValuesWarning` is raised at the end of the search if some
        iterations evaluate less than 95% of the points of the most complete
        one, whose metrics are not comparable (with a `OneStepAheadFold`, when
        `dropna_from_series=True` drops rows with missing values).
        - additional n columns with param = value.
    study : optuna Study
        The optuna study object containing all optimization trials. Access the
        best trial via `study.best_trial`.

    """

    cv_name = type(cv).__name__
    if cv_name not in ['TimeSeriesFold', 'OneStepAheadFold']:
        raise TypeError(
            f"`cv` must be an instance of `TimeSeriesFold` or `OneStepAheadFold`. "
            f"Got {type(cv)}."
        )

    if return_best and exog is not None and (len(exog) != len(y)):
        raise ValueError(
            f"`exog` must have same number of samples as `y`. "
            f"length `exog`: ({len(exog)}), length `y`: ({len(y)})"
        )

    forecaster_search = deepcopy_forecaster(forecaster)
    forecaster_name = type(forecaster_search).__name__

    if cv_name == 'OneStepAheadFold':

        check_one_step_ahead_input(
            forecaster        = forecaster_search,
            cv                = cv,
            metric            = metric,
            y                 = y,
            exog              = exog,
            show_progress     = show_progress,
            suppress_warnings = suppress_warnings
        )

        initial_train_size = date_to_index_position(
                                 index        = cv._extract_index(y), 
                                 date_input   = cv.initial_train_size, 
                                 method       = 'validation',
                                 date_literal = 'initial_train_size'
                             )

    if not isinstance(metric, list):
        metric = [metric]
    metric = [
        _get_metric(metric=m)
        if isinstance(m, str)
        else add_y_train_argument(m) 
        for m in metric
    ]
    metric_names = [(m if isinstance(m, str) else m.__name__) for m in metric]
    _check_metric_names_search(metric_names=metric_names)
    metric_dict = {metric_name: [] for metric_name in metric_names}

    # Objective function using backtesting_forecaster
    if cv_name == 'TimeSeriesFold':

        def _objective(
            trial,
            search_space      = search_space,
            forecaster_search = forecaster_search,
            y                 = y,
            cv                = cv,
            exog              = exog,
            metric            = metric,
            n_jobs            = n_jobs,
            verbose           = verbose,
            suppress_warnings = suppress_warnings,
        ) -> float:

            search_warnings.set_candidate(trial.number)
            sample = search_space(trial)
            if sample.keys() != trial.params.keys():
                raise ValueError(
                    f"`search_space` dict keys must match the names passed to "
                    f"`trial.suggest_*()`.\n"
                    f"  Dict keys    : {list(sample.keys())}\n"
                    f"  Suggest names: {list(trial.params.keys())}"
                )
            sample_params = {k: v for k, v in sample.items() if k != 'lags'}
            forecaster_search.set_params(sample_params)
            if "lags" in sample:
                forecaster_search.set_lags(sample['lags'])

            # NOTE: Same resolution of `n_jobs='auto'` as in grid search.
            n_jobs_candidate = (
                select_n_jobs_backtesting(
                    forecaster=forecaster_search, refit=cv.refit
                )
                if n_jobs == 'auto'
                else n_jobs
            )
            try:
                metrics, _ = backtesting_forecaster(
                                 forecaster        = forecaster_search,
                                 y                 = y,
                                 cv                = cv,
                                 exog              = exog,
                                 metric            = metric,
                                 n_jobs            = n_jobs_candidate,
                                 verbose           = verbose,
                                 show_progress     = False,
                                 suppress_warnings = suppress_warnings
                             )
            except Exception as e:
                raise _skip_trial(search_warnings, trial, error=e) from e
            n_evaluated = int(metrics.loc[0, N_EVALUATED_NAME])
            metrics = metrics.loc[0, list(metric_dict)].to_list()

            # Store all metrics in the trial using optuna's user_attrs mechanism.
            for m_name, m_val in zip(metric_dict, metrics):
                trial.set_user_attr(m_name, float(m_val))
            trial.set_user_attr(N_EVALUATED_NAME, n_evaluated)

            return metrics[0]

    else:

        _SENTINEL = object()
        _MAX_CACHE_SIZE = 10
        _cached_split = {}

        def _objective(
            trial,
            search_space      = search_space,
            forecaster_search = forecaster_search,
            y                 = y,
            cv                = cv,
            exog              = exog,
            metric            = metric
        ) -> float:

            search_warnings.set_candidate(trial.number)
            sample = search_space(trial)
            if sample.keys() != trial.params.keys():
                raise ValueError(
                    f"`search_space` dict keys must match the names passed to "
                    f"`trial.suggest_*()`.\n"
                    f"  Dict keys    : {list(sample.keys())}\n"
                    f"  Suggest names: {list(trial.params.keys())}"
                )
            sample_params = {k: v for k, v in sample.items() if k != 'lags'}
            forecaster_search.set_params(sample_params)

            current_lags = sample.get('lags', _SENTINEL)
            if current_lags is not _SENTINEL:
                forecaster_search.set_lags(current_lags)

            # NOTE: The split is cached by lags, also its error. Its warnings
            # are raised outside the trials, so they are shown as they are.
            lags_key = _make_lags_hashable(current_lags, sentinel=_SENTINEL)
            if lags_key not in _cached_split:

                if len(_cached_split) >= _MAX_CACHE_SIZE:
                    _cached_split.pop(next(iter(_cached_split)))

                search_warnings.set_candidate(None)
                try:
                    _cached_split[lags_key] = (
                        *_train_test_split_one_step_ahead_search(
                            forecaster         = forecaster_search,
                            cv                 = cv,
                            initial_train_size = initial_train_size,
                            y                  = y,
                            exog               = exog
                        ),
                        _get_estimator_categorical_set_params(forecaster_search)
                    )
                except Exception as e:
                    _cached_split[lags_key] = e
                finally:
                    search_warnings.set_candidate(trial.number)

            split = _cached_split[lags_key]
            if isinstance(split, Exception):
                raise _skip_trial(search_warnings, trial, error=split) from split
            (
                X_train, y_train, X_test, y_test, sample_weight, fit_kwargs,
                test_index, _estimator_cat_params
            ) = split
            _restore_estimator_categorical_set_params(
                forecaster_search, _estimator_cat_params
            )

            try:
                metrics, n_evaluated = _calculate_metrics_one_step_ahead(
                                           forecaster    = forecaster_search,
                                           metrics       = metric,
                                           X_train       = X_train,
                                           y_train       = y_train,
                                           X_test        = X_test,
                                           y_test        = y_test,
                                           sample_weight = sample_weight,
                                           fit_kwargs    = fit_kwargs,
                                           y             = y,
                                           test_index    = test_index
                                       )
            except Exception as e:
                raise _skip_trial(search_warnings, trial, error=e) from e

            # Store all metrics in the trial using optuna's user_attrs mechanism.
            for m_name, m_val in zip(metric_dict, metrics):
                trial.set_user_attr(m_name, float(m_val))
            trial.set_user_attr(N_EVALUATED_NAME, n_evaluated)

            return metrics[0]

    kwargs_create_study = kwargs_create_study.copy() if kwargs_create_study is not None else {}
    if 'direction' not in kwargs_create_study.keys():
        kwargs_create_study['direction'] = _get_search_direction(
            metric=metric[0], forecaster=forecaster_search
        )
    if 'sampler' not in kwargs_create_study:
        with warnings.catch_warnings():
            warnings.filterwarnings(
                'ignore',
                message='.*multivariate.*|.*group.*'
            )
            kwargs_create_study['sampler'] = TPESampler(
                multivariate=True, group=True, seed=random_state
            )

    kwargs_study_optimize = kwargs_study_optimize.copy() if kwargs_study_optimize is not None else {}
    if show_progress:
        kwargs_study_optimize['show_progress_bar'] = True
    else:
        kwargs_study_optimize.setdefault('show_progress_bar', False)

    if output_file is not None:
        # Redirect optuna logging to file
        optuna.logging.disable_default_handler()
        logger = logging.getLogger('optuna')
        logger.setLevel(logging.INFO)
        for handler in logger.handlers.copy():
            if isinstance(handler, logging.StreamHandler):
                logger.removeHandler(handler)
        handler = logging.FileHandler(output_file, mode="w")
        logger.addHandler(handler)
    else:
        logging.getLogger("optuna").setLevel(logging.WARNING)
        optuna.logging.disable_default_handler()

    study = optuna.create_study(**kwargs_create_study)
    direction = _check_study_direction(study=study, metric=metric[0])

    # NOTE: The warnings of the trials and of the refit are collected and
    # shown once at the end of the search, see `_SearchWarnings`. A study
    # loaded from storage can have trials of previous calls.
    search_warnings = _SearchWarnings(n_candidates=n_trials, candidate_name='trial')
    n_trials_previous = len(study.trials)
    try:
        with search_warnings:
            _optimize_study(
                study                 = study,
                objective             = _objective,
                n_trials              = n_trials,
                kwargs_study_optimize = kwargs_study_optimize,
                filter_logs           = output_file is None
            )
    finally:
        if output_file is not None:
            handler.close()
            logger.removeHandler(handler)

    lags_list = []
    params_list = []
    trial_number_list = []
    n_evaluated_list = []
    for trial in study.get_trials(states=[TrialState.COMPLETE]):
        estimator_params = {k: v for k, v in trial.params.items() if k != 'lags'}
        lags = trial.params.get(
            'lags',
            forecaster_search.lags if hasattr(forecaster_search, 'lags') else None
        )
        params_list.append(estimator_params)
        lags_list.append(initialize_lags(forecaster_name=forecaster_name, lags=lags)[0])
        trial_number_list.append(trial.number)
        for m_name in metric_dict:
            metric_dict[m_name].append(trial.user_attrs[m_name])
        # NOTE: NaN for the trials of a study loaded from storage, run with a
        # previous version without `n_evaluated`.
        n_evaluated_list.append(trial.user_attrs.get(N_EVALUATED_NAME, np.nan))

    results = pd.DataFrame({
                  'trial_number': trial_number_list,
                  'lags': lags_list,
                  'params': params_list,
                  **metric_dict,
                  N_EVALUATED_NAME: n_evaluated_list
              })

    results = (
        results
        .sort_values(by=list(metric_dict.keys())[0], ascending=direction == 'minimize')
        .reset_index(drop=True)
    )
    results = pd.concat([results, results['params'].apply(pd.Series)], axis=1)

    # NOTE: The results are empty when every trial is skipped.
    if return_best and not results.empty:

        best_lags = results.loc[0, 'lags']
        best_params = results.loc[0, 'params']
        best_metric = results.loc[0, list(metric_dict.keys())[0]]

        # NOTE: Here we use the actual forecaster passed by the user
        forecaster.set_lags(best_lags)
        forecaster.set_params(best_params)

        search_warnings.set_candidate(None)
        with search_warnings:
            forecaster.fit(
                y                         = y,
                exog                      = exog,
                store_in_sample_residuals = True,
                suppress_warnings         = suppress_warnings
            )

        if verbose:
            print(
                f"`Forecaster` refitted using the best-found lags and parameters, "
                f"and the whole data set: \n"
                f"  Lags: {best_lags} \n"
                f"  Parameters: {best_params}\n"
                f"  {'Backtesting' if cv_name == 'TimeSeriesFold' else 'One-step-ahead'} "
                f"metric: {best_metric}"
            )

    search_warnings.emit(n_candidates=len(study.trials) - n_trials_previous)

    # NOTE: Only the trials run in this call, a study loaded from storage
    # (`load_if_exists`) can have others, without `n_evaluated`.
    results_trials = results.loc[results['trial_number'] >= n_trials_previous]
    _warn_n_evaluated_search(
        n_evaluated    = results_trials[N_EVALUATED_NAME].to_numpy(),
        lags           = results_trials['lags'].tolist(),
        candidate_ids  = results_trials['trial_number'].tolist(),
        candidate_name = 'trial',
        best_is_first  = len(results_trials) > 0 and results_trials.index[0] == 0
    )

    return results, study

skforecast.model_selection._validation.backtesting_forecaster_multiseries

backtesting_forecaster_multiseries(
    forecaster,
    series,
    cv,
    metric,
    levels=None,
    add_aggregated_metric=True,
    exog=None,
    interval=None,
    interval_method="conformal",
    n_boot=250,
    use_in_sample_residuals=True,
    use_binned_residuals=True,
    random_state=123,
    return_predictors=False,
    n_jobs="auto",
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
)

Backtesting of forecaster model following the folds generated by the TimeSeriesFold class and using the metric(s) provided.

If forecaster is already trained and initial_train_size is set to None in the TimeSeriesFold class, no initial train will be done and all data will be used to evaluate the model. However, the first len(forecaster.last_window) observations are needed to create the initial predictors, so no predictions are calculated for them.

A copy of the original forecaster is created so that it is not modified during the process.

Parameters:

Name	Type	Description	Default
forecaster	(ForecasterRecursiveMultiSeries, ForecasterDirectMultiVariate, ForecasterRnn)	Forecaster model.	required
series	pandas DataFrame, dict	Training time series.	required
cv	TimeSeriesFold	TimeSeriesFold object with the information needed to split the data into folds. Its window_size is taken from the forecaster, and the object passed is not modified.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: name of the metric, for example 'mean_absolute_error' or 'root_mean_squared_error'. All accepted names are listed in https://skforecast.org/latest/user_guides/metrics.html. If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. y_train is the series before the first forecast origin. mean_absolute_scaled_error and root_mean_squared_scaled_error scale the error of each prediction with the one-step naive forecast over the observations before its own forecast origin, the same for every forecaster. If list: List containing multiple strings and/or Callables.	required
levels	(str, list)	Time series to be predicted. If None all levels will be predicted.	None
add_aggregated_metric	bool	If True, and multiple series (levels) are predicted, the aggregated metrics (average, weighted average and pooled) are also returned. 'average': the average (arithmetic mean) of all levels. 'weighted_average': the average of the metrics weighted by the number of points evaluated in each level. 'pooling': the values of all levels are pooled and then the metric is calculated.	True
exog	pandas Series, pandas DataFrame, dict	Exogenous variables.	None
interval	(float, list, tuple, str, object)	Specifies whether probabilistic predictions should be estimated and the method to use. The following options are supported: If float, represents the nominal (expected) coverage (between 0 and 1). For instance, interval=0.95 corresponds to [0.025, 0.975] quantiles. If list or tuple: Sequence of quantiles to compute, each value must be between 0 and 1 inclusive. For example, a 95% confidence interval can be specified as interval = [0.025, 0.975] or multiple quantiles (e.g. 0.1, 0.5 and 0.9) as interval = [0.1, 0.5, 0.9]. If 'bootstrapping' (str): n_boot bootstrapping predictions will be generated. If scipy.stats distribution object, the distribution parameters will be estimated for each prediction. If None, no probabilistic predictions are estimated. Changed in version 0.23.0: interval is now expressed as quantiles (0-1) instead of percentiles (0-100). Passing percentiles is not longer supported and will raise a ValueError.	None
interval_method	str	Technique used to estimate prediction intervals. Available options: 'bootstrapping': Bootstrapping is used to generate prediction intervals [1]_. 'conformal': Employs the conformal prediction split method for interval estimation [2]_.	'conformal'
n_boot	int	Number of bootstrapping iterations to perform when estimating prediction intervals.	250
use_in_sample_residuals	bool	If True, residuals from the training data are used as proxy of prediction error to create predictions. If False, out of sample residuals (calibration) are used. Out-of-sample residuals must be precomputed using Forecaster's set_out_sample_residuals() method.	True
use_binned_residuals	bool	If True, residuals are selected based on the predicted values (binned selection). If False, residuals are selected randomly.	True
random_state	int	Seed for the random number generator to ensure reproducibility.	123
return_predictors	bool	If True, the predictors used to make the predictions are also returned.	False
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.model_selection._utils.select_n_jobs_backtesting, and set to 1 when the workload of the folds, estimated from the time of the initial fit, is too small to pay the start of the workers (for example, without refit). See skforecast.model_selection._utils.limit_n_jobs_by_workload.	'auto'
verbose	bool	Print number of folds and index of training and validation sets used for backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the backtesting process. See skforecast.exceptions.warn_skforecast_categories for more information.	False

Returns:

Name	Type	Description
metrics_levels	pandas DataFrame	Value(s) of the metric(s): one row per level, named in the column levels, followed by the rows of the aggregated metrics if add_aggregated_metric is True, and one column per metric. A level that is not predicted in any fold (for example, a series shorter than the window size of the forecaster) has NaN metrics and is left out of the aggregated metrics. A MissingValuesWarning names the levels not predicted in a fold where they have observed values. A fold in which no level can be predicted is skipped (no predictions, and the forecaster is not trained only for it). If no fold can be predicted, a ValueError is raised. If interval has a lower and an upper bound (a float, or a list or tuple of 2 quantiles, optionally with the median 0.5), two more columns are added after the metrics, for each level and aggregated in the same way as the other metrics: coverage: share of true values inside the interval (bounds included). interval_width: mean of upper_bound - lower_bound, in the units of the series. The last column, n_evaluated, is the number of points used to compute the metrics of each level: true value and prediction not missing, each date counted once (with the prediction of its last fold when the folds overlap). In the aggregated rows, it is the sum over the levels; weighted_average is weighted by it. New in version 0.27.0
backtest_predictions	pandas DataFrame	Long-format DataFrame containing the predicted values for each series. The DataFrame includes the following columns, in this order: level: Identifier for the time series or level being predicted. fold: Indicates the fold number where the prediction was made. step: Step of the forecast horizon at which the prediction was made, from gap + 1 to gap + steps. y_true: True value of the series (NaN if it is missing). pred: Predicted values for the corresponding series and time steps. New in version 0.27.0: columns step and y_true. If interval is not None, additional columns are included depending on the method: For float: Columns lower_bound and upper_bound. For list or tuple of 2 elements: Columns lower_bound and upper_bound. For list or tuple with multiple quantiles: One column per quantile (e.g., q_0.1, q_0.5, q_0.9). For 'bootstrapping': One column per bootstrapping iteration (e.g., pred_boot_0, pred_boot_1, ..., pred_boot_n). For scipy.stats distribution objects: One column for each estimated parameter of the distribution (e.g., loc, scale). If return_predictors is True, one column per predictor is added after the predictions, with the names of X_train_features_names_out_. Select them by name, for example with backtest_predictions[forecaster.X_train_features_names_out_] once the forecaster is fitted, not by position. Depending on the relation between steps and fold_stride, the output may include repeated indexes (if fold_stride < steps) or gaps (if fold_stride > steps). See Notes below for more details.

Notes

Note on fold_stride vs. steps:

• If fold_stride == steps, test sets are placed back-to-back without overlap. Each observation appears only once in the output DataFrame, so the index is unique.
• If fold_stride < steps, test sets overlap. Multiple forecasts are generated for the same observations and, therefore, the output DataFrame contains repeated indexes.
• If fold_stride > steps, there are gaps between consecutive test sets. Some observations in the series will not have associated predictions, so the output DataFrame has non-contiguous indexes.

References

.. [1] Forecasting: Principles and Practice (3^rd ed) Rob J Hyndman and George Athanasopoulos. https://otexts.com/fpp3/prediction-intervals.html

.. [2] MAPIE - Model Agnostic Prediction Interval Estimator. https://mapie.readthedocs.io/en/stable/content/conformal-prediction/regression/#2-the-split-method

Source code in skforecast/model_selection/_validation.py

@manage_warnings
def backtesting_forecaster_multiseries(
    forecaster: object,
    series: pd.DataFrame | dict[str, pd.Series | pd.DataFrame],
    cv: TimeSeriesFold,
    metric: str | Callable | list[str | Callable],
    levels: str | list[str] | None = None,
    add_aggregated_metric: bool = True,
    exog: pd.Series | pd.DataFrame | dict[str, pd.Series | pd.DataFrame] | None = None,
    interval: float | list[float] | tuple[float, ...] | str | object | None = None,
    interval_method: str = 'conformal',
    n_boot: int = 250,
    use_in_sample_residuals: bool = True,
    use_binned_residuals: bool = True,
    random_state: int = 123,
    return_predictors: bool = False,
    n_jobs: int | str = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False
) -> tuple[pd.DataFrame, pd.DataFrame]:
    """
    Backtesting of forecaster model following the folds generated by the TimeSeriesFold
    class and using the metric(s) provided.

    If `forecaster` is already trained and `initial_train_size` is set to `None` in the
    TimeSeriesFold class, no initial train will be done and all data will be used
    to evaluate the model. However, the first `len(forecaster.last_window)` observations
    are needed to create the initial predictors, so no predictions are calculated for
    them.

    A copy of the original forecaster is created so that it is not modified during 
    the process.

    Parameters
    ----------
    forecaster : ForecasterRecursiveMultiSeries, ForecasterDirectMultiVariate, ForecasterRnn
        Forecaster model.
    series : pandas DataFrame, dict
        Training time series.
    cv : TimeSeriesFold
        TimeSeriesFold object with the information needed to split the data into
        folds. Its `window_size` is taken from the forecaster, and the object
        passed is not modified.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: name of the metric, for example `'mean_absolute_error'`
        or `'root_mean_squared_error'`. All accepted names are listed in
        https://skforecast.org/latest/user_guides/metrics.html.
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float. `y_train` is the series before the
        first forecast origin. `mean_absolute_scaled_error` and
        `root_mean_squared_scaled_error` scale the error of each prediction
        with the one-step naive forecast over the observations before its own
        forecast origin, the same for every forecaster.
        - If `list`: List containing multiple strings and/or Callables.
    levels : str, list, default None
        Time series to be predicted. If `None` all levels will be predicted.
    add_aggregated_metric : bool, default True
        If `True`, and multiple series (`levels`) are predicted, the aggregated
        metrics (average, weighted average and pooled) are also returned.

        - 'average': the average (arithmetic mean) of all levels.
        - 'weighted_average': the average of the metrics weighted by the number of
        points evaluated in each level.
        - 'pooling': the values of all levels are pooled and then the metric is
        calculated.
    exog : pandas Series, pandas DataFrame, dict, default None
        Exogenous variables.
    interval : float, list, tuple, str, object, default None
        Specifies whether probabilistic predictions should be estimated and the 
        method to use. The following options are supported:

        - If `float`, represents the nominal (expected) coverage (between 0 and 1). 
        For instance, `interval=0.95` corresponds to `[0.025, 0.975]` quantiles.
        - If `list` or `tuple`: Sequence of quantiles to compute, each value must 
        be between 0 and 1 inclusive. For example, a 95% confidence interval can 
        be specified as `interval = [0.025, 0.975]` or multiple quantiles (e.g. 0.1, 
        0.5 and 0.9) as `interval = [0.1, 0.5, 0.9]`.
        - If 'bootstrapping' (str): `n_boot` bootstrapping predictions will be generated.
        - If scipy.stats distribution object, the distribution parameters will
        be estimated for each prediction.
        - If None, no probabilistic predictions are estimated.

        **Changed in version 0.23.0:** `interval` is now expressed as
        quantiles (0-1) instead of percentiles (0-100). Passing percentiles
        is not longer supported and will raise a `ValueError`.
    interval_method : str, default 'conformal'
        Technique used to estimate prediction intervals. Available options:

        - 'bootstrapping': Bootstrapping is used to generate prediction 
        intervals [1]_.
        - 'conformal': Employs the conformal prediction split method for 
        interval estimation [2]_.
    n_boot : int, default 250
        Number of bootstrapping iterations to perform when estimating prediction 
        intervals.
    use_in_sample_residuals : bool, default True
        If `True`, residuals from the training data are used as proxy of
        prediction error to create predictions. 
        If `False`, out of sample residuals (calibration) are used. 
        Out-of-sample residuals must be precomputed using Forecaster's
        `set_out_sample_residuals()` method.
    use_binned_residuals : bool, default True
        If `True`, residuals are selected based on the predicted values 
        (binned selection).
        If `False`, residuals are selected randomly.
    random_state : int, default 123
        Seed for the random number generator to ensure reproducibility.
    return_predictors : bool, default False
        If `True`, the predictors used to make the predictions are also returned.
    n_jobs : int, 'auto', default 'auto'
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.model_selection._utils.select_n_jobs_backtesting, and set to 1
        when the workload of the folds, estimated from the time of the initial fit, is
        too small to pay the start of the workers (for example, without refit).
        See skforecast.model_selection._utils.limit_n_jobs_by_workload.
    verbose : bool, default False
        Print number of folds and index of training and validation sets used 
        for backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings: bool, default False
        If `True`, skforecast warnings will be suppressed during the backtesting 
        process. See skforecast.exceptions.warn_skforecast_categories for more
        information.

    Returns
    -------
    metrics_levels : pandas DataFrame
        Value(s) of the metric(s): one row per level, named in the column
        `levels`, followed by the rows of the aggregated metrics if
        `add_aggregated_metric` is `True`, and one column per metric.
        A level that is not predicted in any fold (for example, a series
        shorter than the window size of the forecaster) has NaN metrics and is
        left out of the aggregated metrics. A `MissingValuesWarning` names the
        levels not predicted in a fold where they have observed values. A fold
        in which no level can be predicted is skipped (no predictions, and the
        forecaster is not trained only for it). If no fold can be predicted, a
        `ValueError` is raised.
        If `interval` has a lower and an upper bound (a `float`, or a `list` or
        `tuple` of 2 quantiles, optionally with the median 0.5), two more 
        columns are added after the metrics, for each level and aggregated in 
        the same way as the other metrics:

        - coverage: share of true values inside the interval (bounds included).
        - interval_width: mean of `upper_bound - lower_bound`, in the units of
        the series.

        The last column, `n_evaluated`, is the number of points used to
        compute the metrics of each level: true value and prediction not
        missing, each date counted once (with the prediction of its last fold
        when the folds overlap). In the aggregated rows, it is the sum over the
        levels; `weighted_average` is weighted by it.

        **New in version 0.27.0**
    backtest_predictions : pandas DataFrame
        Long-format DataFrame containing the predicted values for each series. The 
        DataFrame includes the following columns, in this order:

        - level: Identifier for the time series or level being predicted.
        - fold: Indicates the fold number where the prediction was made.
        - step: Step of the forecast horizon at which the prediction was made,
        from `gap + 1` to `gap + steps`.
        - y_true: True value of the series (NaN if it is missing).
        - pred: Predicted values for the corresponding series and time steps.

        **New in version 0.27.0**: columns `step` and `y_true`.

        If `interval` is not `None`, additional columns are included depending on the method:

        - For `float`: Columns `lower_bound` and `upper_bound`.
        - For `list` or `tuple` of 2 elements: Columns `lower_bound` and `upper_bound`.
        - For `list` or `tuple` with multiple quantiles: One column per quantile 
        (e.g., `q_0.1`, `q_0.5`, `q_0.9`).
        - For `'bootstrapping'`: One column per bootstrapping iteration 
        (e.g., `pred_boot_0`, `pred_boot_1`, ..., `pred_boot_n`).
        - For `scipy.stats` distribution objects: One column for each estimated 
        parameter of the distribution (e.g., `loc`, `scale`).

        If `return_predictors` is `True`, one column per predictor is added
        after the predictions, with the names of `X_train_features_names_out_`.
        Select them by name, for example with
        `backtest_predictions[forecaster.X_train_features_names_out_]` once the
        forecaster is fitted, not by position.

        Depending on the relation between `steps` and `fold_stride`, the output
        may include repeated indexes (if `fold_stride < steps`) or gaps
        (if `fold_stride > steps`). See Notes below for more details.

    Notes
    -----
    Note on `fold_stride` vs. `steps`:

    - If `fold_stride == steps`, test sets are placed back-to-back without overlap. 
    Each observation appears only once in the output DataFrame, so the index is unique.
    - If `fold_stride < steps`, test sets overlap. Multiple forecasts are generated 
    for the same observations and, therefore, the output DataFrame contains repeated 
    indexes.
    - If `fold_stride > steps`, there are gaps between consecutive test sets. 
    Some observations in the series will not have associated predictions, so 
    the output DataFrame has non-contiguous indexes.

    References
    ----------
    .. [1] Forecasting: Principles and Practice (3rd ed) Rob J Hyndman and George Athanasopoulos.
           https://otexts.com/fpp3/prediction-intervals.html

    .. [2] MAPIE - Model Agnostic Prediction Interval Estimator.
           https://mapie.readthedocs.io/en/stable/content/conformal-prediction/regression/#2-the-split-method

    """

    multi_series_forecasters = [
        'ForecasterRecursiveMultiSeries', 
        'ForecasterDirectMultiVariate',
        'ForecasterRnn'
    ]

    forecaster_name = type(forecaster).__name__

    if forecaster_name not in multi_series_forecasters:
        raise TypeError(
            f"`forecaster` must be of type {multi_series_forecasters}, "
            f"for all other types of forecasters use the functions available in "
            f"the `model_selection` module. Got {forecaster_name}"
        )

    if forecaster_name == 'ForecasterRecursiveMultiSeries':
        series, series_indexes = check_preprocess_series(series)
        if exog is not None:
            series_names_in_ = list(series.keys())
            exog_dict = {serie: None for serie in series_names_in_}
            exog, _ = check_preprocess_exog_multiseries(
                          series_names_in_  = series_names_in_,
                          series_index_type = type(series_indexes[series_names_in_[0]]),
                          exog              = exog,
                          exog_dict         = exog_dict
                      )

    check_backtesting_input(
        forecaster              = forecaster,
        cv                      = cv,
        metric                  = metric,
        add_aggregated_metric   = add_aggregated_metric,
        series                  = series,
        exog                    = exog,
        interval                = interval,
        interval_method         = interval_method,
        n_boot                  = n_boot,
        use_in_sample_residuals = use_in_sample_residuals,
        use_binned_residuals    = use_binned_residuals,
        random_state            = random_state,
        return_predictors       = return_predictors,
        n_jobs                  = n_jobs,
        show_progress           = show_progress,
        suppress_warnings       = suppress_warnings
    )

    metrics_levels, backtest_predictions = _backtesting_forecaster_multiseries(
        forecaster              = forecaster,
        series                  = series,
        cv                      = cv,
        levels                  = levels,
        metric                  = metric,
        add_aggregated_metric   = add_aggregated_metric,
        exog                    = exog,
        interval                = interval,
        interval_method         = interval_method,
        n_boot                  = n_boot,
        use_in_sample_residuals = use_in_sample_residuals,
        use_binned_residuals    = use_binned_residuals,
        random_state            = random_state,
        return_predictors       = return_predictors,
        n_jobs                  = n_jobs,
        verbose                 = verbose,
        show_progress           = show_progress,
        suppress_warnings       = suppress_warnings
    )

    return metrics_levels, backtest_predictions

skforecast.model_selection._search.grid_search_forecaster_multiseries

grid_search_forecaster_multiseries(
    forecaster,
    series,
    cv,
    param_grid,
    metric,
    aggregate_metric=None,
    levels=None,
    exog=None,
    lags_grid=None,
    return_best=True,
    n_jobs="auto",
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
    output_file=None,
)

Exhaustive search over specified parameter values for a Forecaster object. Validation is done using multi-series backtesting.

Parameters:

Name	Type	Description	Default
forecaster	(ForecasterRecursiveMultiSeries, ForecasterDirectMultiVariate)	Forecaster model.	required
series	pandas DataFrame, dict	Training time series.	required
cv	(TimeSeriesFold, OneStepAheadFold)	TimeSeriesFold or OneStepAheadFold object with the information needed to split the data into folds. With a TimeSeriesFold, its window_size is taken from the forecaster; with a OneStepAheadFold, only its initial_train_size is used. The object passed is not modified.	required
param_grid	dict	Dictionary with parameters names (str) as keys and lists of parameter settings to try as values.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: name of the metric, for example 'mean_absolute_error' or 'root_mean_squared_error'. All accepted names are listed in https://skforecast.org/latest/user_guides/metrics.html. If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. y_train is the series before the first forecast origin. mean_absolute_scaled_error and root_mean_squared_scaled_error scale the error of each prediction with the one-step naive forecast over the observations before its own forecast origin, the same for every forecaster. If list: List containing multiple strings and/or Callables. The first metric ranks the candidates. Scikit-learn metrics ending in _score are maximized and those ending in _error, _loss or _deviance are minimized. Other functions are minimized in regression and maximized in classification, unless their attribute greater_is_better is set (True or False).	required
aggregate_metric	(str, list)	Aggregation method/s used to combine the metric/s of all levels (series) when multiple levels are predicted. If list, the first aggregation method is used to select the best parameters. If None, ['weighted_average', 'average', 'pooling'] is used. 'average': the average (arithmetic mean) of all levels. 'weighted_average': the average of the metrics weighted by the number of points evaluated in each level. 'pooling': the values of all levels are pooled and then the metric is calculated.	None
levels	(str, list)	level (str) or levels (list) at which the forecaster is optimized. If None, all levels are taken into account.	None
exog	pandas Series, pandas DataFrame, dict	Exogenous variables.	None
lags_grid	(list, dict)	Lists of lags to try, containing int, lists, numpy ndarray, range objects or None. A None element evaluates the forecaster without lags, using only its window_features (a ValueError is raised if it has none). If dict, the keys are used as labels in the results DataFrame, and the values are used as the lists of lags to try. If None, the lags of the forecaster are used.	None
return_best	bool	Refit the forecaster using the best found parameters on the whole data.	True
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.model_selection._utils.select_n_jobs_backtesting. Ignored for OneStepAheadFold.	'auto'
verbose	bool	Print number of folds used for cv or backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information. Otherwise, the warnings of the candidates are shown once, at the end of the search, with the number of candidates in which they are raised. The candidates skipped because they raise an error are always reported, in a single warning.	False
output_file	str	Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file.	None

Returns:

Name	Type	Description
results	pandas DataFrame	Results for each combination of parameters. column levels: levels configuration for each iteration. column lags: lags configuration for each iteration. column lags_label: descriptive label or alias for the lags. column params: parameters configuration for each iteration. column metric: metric value estimated for each iteration. The resulting metric will be the average of the optimization of all levels. column n_evaluated: number of points that enter the metrics of each iteration (true value and prediction not missing), summed over the levels. A MissingValuesWarning is raised at the end of the search if some iterations evaluate less than 95% of the points of the most complete one, whose metrics are not comparable. additional n columns with param = value.

Source code in skforecast/model_selection/_search.py

def grid_search_forecaster_multiseries(
    forecaster: object,
    series: pd.DataFrame | dict[str, pd.Series | pd.DataFrame],
    cv: TimeSeriesFold | OneStepAheadFold,
    param_grid: dict,
    metric: str | Callable | list[str | Callable],
    aggregate_metric: str | list[str] | None = None,
    levels: str | list[str] | None = None,
    exog: pd.Series | pd.DataFrame | dict[str, pd.Series | pd.DataFrame] | None = None,
    lags_grid: (
        list[int | list[int] | np.ndarray | range | None]
        | dict[str, int | list[int] | np.ndarray | range | None]
        | None
    ) = None,
    return_best: bool = True,
    n_jobs: int | str = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: str | None = None
) -> pd.DataFrame:
    """
    Exhaustive search over specified parameter values for a Forecaster object.
    Validation is done using multi-series backtesting.

    Parameters
    ----------
    forecaster : ForecasterRecursiveMultiSeries, ForecasterDirectMultiVariate
        Forecaster model.
    series : pandas DataFrame, dict
        Training time series.
    cv : TimeSeriesFold, OneStepAheadFold
        TimeSeriesFold or OneStepAheadFold object with the information needed to split
        the data into folds. With a `TimeSeriesFold`, its `window_size` is taken
        from the forecaster; with a `OneStepAheadFold`, only its `initial_train_size`
        is used. The object passed is not modified.
    param_grid : dict
        Dictionary with parameters names (`str`) as keys and lists of parameter
        settings to try as values.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: name of the metric, for example `'mean_absolute_error'`
        or `'root_mean_squared_error'`. All accepted names are listed in
        https://skforecast.org/latest/user_guides/metrics.html.
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float. `y_train` is the series before the
        first forecast origin. `mean_absolute_scaled_error` and
        `root_mean_squared_scaled_error` scale the error of each prediction
        with the one-step naive forecast over the observations before its own
        forecast origin, the same for every forecaster.
        - If `list`: List containing multiple strings and/or Callables.

        The first metric ranks the candidates. Scikit-learn metrics ending in
        `_score` are maximized and those ending in `_error`, `_loss` or
        `_deviance` are minimized. Other functions are minimized in regression
        and maximized in classification, unless their attribute
        `greater_is_better` is set (`True` or `False`).
    aggregate_metric : str, list, default None
        Aggregation method/s used to combine the metric/s of all levels (series)
        when multiple levels are predicted. If list, the first aggregation method
        is used to select the best parameters. If `None`,
        `['weighted_average', 'average', 'pooling']` is used.

        - 'average': the average (arithmetic mean) of all levels.
        - 'weighted_average': the average of the metrics weighted by the number of
        points evaluated in each level.
        - 'pooling': the values of all levels are pooled and then the metric is
        calculated.
    levels : str, list, default None
        level (`str`) or levels (`list`) at which the forecaster is optimized. 
        If `None`, all levels are taken into account.
    exog : pandas Series, pandas DataFrame, dict, default None
        Exogenous variables.
    lags_grid : list, dict, default None
        Lists of lags to try, containing int, lists, numpy ndarray, range
        objects or `None`. A `None` element evaluates the forecaster without
        lags, using only its `window_features` (a `ValueError` is raised if
        it has none). If `dict`, the keys are used as labels in the `results`
        DataFrame, and the values are used as the lists of lags to try. If
        `None`, the lags of the forecaster are used.
    return_best : bool, default True
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default 'auto'
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.model_selection._utils.select_n_jobs_backtesting. Ignored for
        `OneStepAheadFold`.
    verbose : bool, default False
        Print number of folds used for cv or backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings: bool, default False
        If `True`, skforecast warnings will be suppressed during the hyperparameter 
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.
        Otherwise, the warnings of the candidates are shown once, at the end
        of the search, with the number of candidates in which they are
        raised. The candidates skipped because they raise an error are always
        reported, in a single warning.
    output_file : str, default None
        Specifies the filename or full path where the results should be saved. 
        The results will be saved in a tab-separated values (TSV) format. If 
        `None`, the results will not be saved to a file.

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column levels: levels configuration for each iteration.
        - column lags: lags configuration for each iteration.
        - column lags_label: descriptive label or alias for the lags.
        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration. The resulting 
        metric will be the average of the optimization of all levels.
        - column n_evaluated: number of points that enter the metrics of each
        iteration (true value and prediction not missing), summed over the
        levels. A `MissingValuesWarning` is raised at the end of the search if
        some iterations evaluate less than 95% of the points of the most
        complete one, whose metrics are not comparable.
        - additional n columns with param = value.

    """

    param_grid = list(ParameterGrid(param_grid))

    results = _evaluate_grid_hyperparameters_multiseries(
                  forecaster        = forecaster,
                  series            = series,
                  cv                = cv,
                  param_grid        = param_grid,
                  metric            = metric,
                  aggregate_metric  = aggregate_metric,
                  levels            = levels,
                  exog              = exog,
                  lags_grid         = lags_grid,
                  n_jobs            = n_jobs,
                  return_best       = return_best,
                  verbose           = verbose,
                  show_progress     = show_progress,
                  suppress_warnings = suppress_warnings,
                  output_file       = output_file
              )

    return results

skforecast.model_selection._search.random_search_forecaster_multiseries

random_search_forecaster_multiseries(
    forecaster,
    series,
    cv,
    param_distributions,
    metric,
    aggregate_metric=None,
    levels=None,
    exog=None,
    lags_grid=None,
    n_iter=10,
    random_state=123,
    return_best=True,
    n_jobs="auto",
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
    output_file=None,
)

Random search over specified parameter values or distributions for a Forecaster object. Validation is done using multi-series backtesting.

Parameters:

Name	Type	Description	Default
forecaster	(ForecasterRecursiveMultiSeries, ForecasterDirectMultiVariate)	Forecaster model.	required
series	pandas DataFrame, dict	Training time series.	required
cv	(TimeSeriesFold, OneStepAheadFold)	TimeSeriesFold or OneStepAheadFold object with the information needed to split the data into folds. With a TimeSeriesFold, its window_size is taken from the forecaster; with a OneStepAheadFold, only its initial_train_size is used. The object passed is not modified.	required
param_distributions	dict	Dictionary with parameters names (str) as keys and distributions or lists of parameters to try.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: name of the metric, for example 'mean_absolute_error' or 'root_mean_squared_error'. All accepted names are listed in https://skforecast.org/latest/user_guides/metrics.html. If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. y_train is the series before the first forecast origin. mean_absolute_scaled_error and root_mean_squared_scaled_error scale the error of each prediction with the one-step naive forecast over the observations before its own forecast origin, the same for every forecaster. If list: List containing multiple strings and/or Callables. The first metric ranks the candidates. Scikit-learn metrics ending in _score are maximized and those ending in _error, _loss or _deviance are minimized. Other functions are minimized in regression and maximized in classification, unless their attribute greater_is_better is set (True or False).	required
aggregate_metric	(str, list)	Aggregation method/s used to combine the metric/s of all levels (series) when multiple levels are predicted. If list, the first aggregation method is used to select the best parameters. If None, ['weighted_average', 'average', 'pooling'] is used. 'average': the average (arithmetic mean) of all levels. 'weighted_average': the average of the metrics weighted by the number of points evaluated in each level. 'pooling': the values of all levels are pooled and then the metric is calculated.	None
levels	(str, list)	level (str) or levels (list) at which the forecaster is optimized. If None, all levels are taken into account.	None
exog	pandas Series, pandas DataFrame, dict	Exogenous variables.	None
lags_grid	(list, dict)	Lists of lags to try, containing int, lists, numpy ndarray, range objects or None. A None element evaluates the forecaster without lags, using only its window_features (a ValueError is raised if it has none). If dict, the keys are used as labels in the results DataFrame, and the values are used as the lists of lags to try. If None, the lags of the forecaster are used.	None
n_iter	int	Number of parameter settings that are sampled per lags configuration. n_iter trades off runtime vs quality of the solution.	10
random_state	int	Sets a seed to the random sampling for reproducible output.	123
return_best	bool	Refit the forecaster using the best found parameters on the whole data.	True
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.model_selection._utils.select_n_jobs_backtesting. Ignored for OneStepAheadFold.	'auto'
verbose	bool	Print number of folds used for cv or backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information. Otherwise, the warnings of the candidates are shown once, at the end of the search, with the number of candidates in which they are raised. The candidates skipped because they raise an error are always reported, in a single warning.	False
output_file	str	Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file.	None

Returns:

Name	Type	Description
results	pandas DataFrame	Results for each combination of parameters. column levels: levels configuration for each iteration. column lags: lags configuration for each iteration. column lags_label: descriptive label or alias for the lags. column params: parameters configuration for each iteration. column metric: metric value estimated for each iteration. The resulting metric will be the average of the optimization of all levels. column n_evaluated: number of points that enter the metrics of each iteration (true value and prediction not missing), summed over the levels. A MissingValuesWarning is raised at the end of the search if some iterations evaluate less than 95% of the points of the most complete one, whose metrics are not comparable. additional n columns with param = value.

Source code in skforecast/model_selection/_search.py

def random_search_forecaster_multiseries(
    forecaster: object,
    series: pd.DataFrame | dict[str, pd.Series | pd.DataFrame],
    cv: TimeSeriesFold | OneStepAheadFold,
    param_distributions: dict,
    metric: str | Callable | list[str | Callable],
    aggregate_metric: str | list[str] | None = None,
    levels: str | list[str] | None = None,
    exog: pd.Series | pd.DataFrame | dict[str, pd.Series | pd.DataFrame] | None = None,
    lags_grid: (
        list[int | list[int] | np.ndarray | range | None]
        | dict[str, int | list[int] | np.ndarray | range | None]
        | None
    ) = None,
    n_iter: int = 10,
    random_state: int = 123,
    return_best: bool = True,
    n_jobs: int | str = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: str | None = None
) -> pd.DataFrame:
    """
    Random search over specified parameter values or distributions for a Forecaster 
    object. Validation is done using multi-series backtesting.

    Parameters
    ----------
    forecaster : ForecasterRecursiveMultiSeries, ForecasterDirectMultiVariate
        Forecaster model.
    series : pandas DataFrame, dict
        Training time series.
    cv : TimeSeriesFold, OneStepAheadFold
        TimeSeriesFold or OneStepAheadFold object with the information needed to split
        the data into folds. With a `TimeSeriesFold`, its `window_size` is taken
        from the forecaster; with a `OneStepAheadFold`, only its `initial_train_size`
        is used. The object passed is not modified.
    param_distributions : dict
        Dictionary with parameters names (`str`) as keys and distributions or 
        lists of parameters to try.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: name of the metric, for example `'mean_absolute_error'`
        or `'root_mean_squared_error'`. All accepted names are listed in
        https://skforecast.org/latest/user_guides/metrics.html.
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float. `y_train` is the series before the
        first forecast origin. `mean_absolute_scaled_error` and
        `root_mean_squared_scaled_error` scale the error of each prediction
        with the one-step naive forecast over the observations before its own
        forecast origin, the same for every forecaster.
        - If `list`: List containing multiple strings and/or Callables.

        The first metric ranks the candidates. Scikit-learn metrics ending in
        `_score` are maximized and those ending in `_error`, `_loss` or
        `_deviance` are minimized. Other functions are minimized in regression
        and maximized in classification, unless their attribute
        `greater_is_better` is set (`True` or `False`).
    aggregate_metric : str, list, default None
        Aggregation method/s used to combine the metric/s of all levels (series)
        when multiple levels are predicted. If list, the first aggregation method
        is used to select the best parameters. If `None`,
        `['weighted_average', 'average', 'pooling']` is used.

        - 'average': the average (arithmetic mean) of all levels.
        - 'weighted_average': the average of the metrics weighted by the number of
        points evaluated in each level.
        - 'pooling': the values of all levels are pooled and then the metric is
        calculated.
    levels : str, list, default None
        level (`str`) or levels (`list`) at which the forecaster is optimized. 
        If `None`, all levels are taken into account.
    exog : pandas Series, pandas DataFrame, dict, default None
        Exogenous variables.
    lags_grid : list, dict, default None
        Lists of lags to try, containing int, lists, numpy ndarray, range
        objects or `None`. A `None` element evaluates the forecaster without
        lags, using only its `window_features` (a `ValueError` is raised if
        it has none). If `dict`, the keys are used as labels in the `results`
        DataFrame, and the values are used as the lists of lags to try. If
        `None`, the lags of the forecaster are used.
    n_iter : int, default 10
        Number of parameter settings that are sampled per lags configuration. 
        n_iter trades off runtime vs quality of the solution.
    random_state : int, default 123
        Sets a seed to the random sampling for reproducible output.
    return_best : bool, default True
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default 'auto'
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.model_selection._utils.select_n_jobs_backtesting. Ignored for
        `OneStepAheadFold`.
    verbose : bool, default False
        Print number of folds used for cv or backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings: bool, default False
        If `True`, skforecast warnings will be suppressed during the hyperparameter 
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.
        Otherwise, the warnings of the candidates are shown once, at the end
        of the search, with the number of candidates in which they are
        raised. The candidates skipped because they raise an error are always
        reported, in a single warning.
    output_file : str, default None
        Specifies the filename or full path where the results should be saved. 
        The results will be saved in a tab-separated values (TSV) format. If 
        `None`, the results will not be saved to a file.

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column levels: levels configuration for each iteration.
        - column lags: lags configuration for each iteration.
        - column lags_label: descriptive label or alias for the lags.
        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration. The resulting 
        metric will be the average of the optimization of all levels.
        - column n_evaluated: number of points that enter the metrics of each
        iteration (true value and prediction not missing), summed over the
        levels. A `MissingValuesWarning` is raised at the end of the search if
        some iterations evaluate less than 95% of the points of the most
        complete one, whose metrics are not comparable.
        - additional n columns with param = value.

    """

    param_grid = _sample_param_grid(
                     param_distributions = param_distributions,
                     n_iter              = n_iter,
                     random_state        = random_state,
                     grid_search_name    = 'grid_search_forecaster_multiseries',
                     lags_grid           = lags_grid is not None
                 )

    results = _evaluate_grid_hyperparameters_multiseries(
                  forecaster        = forecaster,
                  series            = series,
                  cv                = cv,
                  param_grid        = param_grid,
                  metric            = metric,
                  aggregate_metric  = aggregate_metric,
                  levels            = levels,
                  exog              = exog,
                  lags_grid         = lags_grid,
                  return_best       = return_best,
                  n_jobs            = n_jobs,
                  verbose           = verbose,
                  show_progress     = show_progress,
                  suppress_warnings = suppress_warnings,
                  output_file       = output_file
              )

    return results

skforecast.model_selection._search.bayesian_search_forecaster_multiseries

bayesian_search_forecaster_multiseries(
    forecaster,
    series,
    cv,
    search_space,
    metric,
    aggregate_metric=None,
    levels=None,
    exog=None,
    n_trials=20,
    random_state=123,
    return_best=True,
    n_jobs="auto",
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
    output_file=None,
    kwargs_create_study=None,
    kwargs_study_optimize=None,
)

Bayesian search for hyperparameters of a Forecaster object using optuna library.

Parameters:

Name	Type	Description	Default
forecaster	(ForecasterRecursiveMultiSeries, ForecasterDirectMultiVariate)	Forecaster model.	required
series	pandas DataFrame, dict	Training time series.	required
cv	(TimeSeriesFold, OneStepAheadFold)	TimeSeriesFold or OneStepAheadFold object with the information needed to split the data into folds. With a TimeSeriesFold, its window_size is taken from the forecaster; with a OneStepAheadFold, only its initial_train_size is used. The object passed is not modified.	required
search_space	Callable	Function with argument trial which returns a dictionary with parameters names (str) as keys and Trial object from optuna (trial.suggest_float, trial.suggest_int, trial.suggest_categorical) as values.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: name of the metric, for example 'mean_absolute_error' or 'root_mean_squared_error'. All accepted names are listed in https://skforecast.org/latest/user_guides/metrics.html. If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. y_train is the series before the first forecast origin. mean_absolute_scaled_error and root_mean_squared_scaled_error scale the error of each prediction with the one-step naive forecast over the observations before its own forecast origin, the same for every forecaster. If list: List containing multiple strings and/or Callables. The first metric ranks the candidates. Scikit-learn metrics ending in _score are maximized and those ending in _error, _loss or _deviance are minimized. Other functions are minimized in regression and maximized in classification, unless their attribute greater_is_better is set (True or False).	required
aggregate_metric	(str, list)	Aggregation method/s used to combine the metric/s of all levels (series) when multiple levels are predicted. If list, the first aggregation method is used to select the best parameters. If None, ['weighted_average', 'average', 'pooling'] is used. 'average': the average (arithmetic mean) of all levels. 'weighted_average': the average of the metrics weighted by the number of points evaluated in each level. 'pooling': the values of all levels are pooled and then the metric is calculated.	None
levels	(str, list)	level (str) or levels (list) at which the forecaster is optimized. If None, all levels are taken into account.	None
exog	pandas Series, pandas DataFrame, dict	Exogenous variables.	None
n_trials	int	Number of parameter settings (configurations) sampled during the search. With the default TPESampler, the first n_startup_trials (10 by default) are drawn by random sampling to seed the search, and the TPE algorithm only guides sampling from the 11th trial onward. For meaningful Bayesian optimization, n_trials should therefore be clearly larger than 10 (for example 30 to 50 or more). This threshold changes if a custom sampler with a different n_startup_trials is passed in kwargs_create_study. Larger values increase run time, since each trial triggers a full evaluation.	20
random_state	int	Seed for the TPESampler for reproducible output. When a custom sampler is passed in kwargs_create_study, the seed must be set within that sampler. For example {'sampler': TPESampler(seed=145)}.	123
return_best	bool	Refit the forecaster using the best found parameters on the whole data.	True
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.model_selection._utils.select_n_jobs_backtesting. Ignored for OneStepAheadFold.	'auto'
verbose	bool	Print number of folds used for cv or backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information. Otherwise, the warnings of the trials are shown once, at the end of the search, with the number of trials in which they are raised. The trials skipped because they raise an error are always reported, in a single warning.	False
output_file	str	Specifies the filename or full path where the optuna logging output should be saved. If None, logging output will not be redirected to a file.	None
kwargs_create_study	dict	Additional keyword arguments (key, value mappings) to pass to optuna.create_study(). If default, the direction is inferred from the first metric (see metric), and a TPESampler(multivariate=True, group=True, seed=random_state) sampler is used during optimization. If direction is passed, the order of results and the parameters selected with return_best follow it (a warning is raised if it contradicts the direction of the metric).	None
kwargs_study_optimize	dict	Additional keyword arguments (key, value mappings) to pass to study.optimize(). The trials whose evaluation raises an error are marked as failed and skipped, also when catch is passed (its errors are kept). An error in search_space stops the search.	None

Returns:

Name	Type	Description
results	pandas DataFrame	Results for each combination of parameters. column trial_number: optuna trial number for each iteration. Use study.trials[trial_number] to access the full optuna trial object. column levels: levels configuration for each iteration. column lags: lags configuration for each iteration. column params: parameters configuration for each iteration. column metric: metric value estimated for each iteration. The resulting metric will be the average of the optimization of all levels. column n_evaluated: number of points that enter the metrics of each iteration (true value and prediction not missing), summed over the levels. A MissingValuesWarning is raised at the end of the search if some iterations evaluate less than 95% of the points of the most complete one, whose metrics are not comparable. additional n columns with param = value.
study	optuna Study	The optuna study object containing all optimization trials. Access the best trial via study.best_trial.

Source code in skforecast/model_selection/_search.py

@manage_warnings
def bayesian_search_forecaster_multiseries(
    forecaster: object,
    series: pd.DataFrame | dict[str, pd.Series | pd.DataFrame],
    cv: TimeSeriesFold | OneStepAheadFold,
    search_space: Callable,
    metric: str | Callable | list[str | Callable],
    aggregate_metric: str | list[str] | None = None,
    levels: str | list[str] | None = None,
    exog: pd.Series | pd.DataFrame | dict[str, pd.Series | pd.DataFrame] | None = None,
    n_trials: int = 20,
    random_state: int = 123,
    return_best: bool = True,
    n_jobs: int | str = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: str | None = None,
    kwargs_create_study: dict | None = None,
    kwargs_study_optimize: dict | None = None
) -> tuple[pd.DataFrame, object]:
    """
    Bayesian search for hyperparameters of a Forecaster object using optuna library.

    Parameters
    ----------
    forecaster : ForecasterRecursiveMultiSeries, ForecasterDirectMultiVariate
        Forecaster model.
    series : pandas DataFrame, dict
        Training time series.
    cv : TimeSeriesFold, OneStepAheadFold
        TimeSeriesFold or OneStepAheadFold object with the information needed to split
        the data into folds. With a `TimeSeriesFold`, its `window_size` is taken
        from the forecaster; with a `OneStepAheadFold`, only its `initial_train_size`
        is used. The object passed is not modified.
    search_space : Callable
        Function with argument `trial` which returns a dictionary with parameters names 
        (`str`) as keys and Trial object from optuna (trial.suggest_float, 
        trial.suggest_int, trial.suggest_categorical) as values.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: name of the metric, for example `'mean_absolute_error'`
        or `'root_mean_squared_error'`. All accepted names are listed in
        https://skforecast.org/latest/user_guides/metrics.html.
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float. `y_train` is the series before the
        first forecast origin. `mean_absolute_scaled_error` and
        `root_mean_squared_scaled_error` scale the error of each prediction
        with the one-step naive forecast over the observations before its own
        forecast origin, the same for every forecaster.
        - If `list`: List containing multiple strings and/or Callables.

        The first metric ranks the candidates. Scikit-learn metrics ending in
        `_score` are maximized and those ending in `_error`, `_loss` or
        `_deviance` are minimized. Other functions are minimized in regression
        and maximized in classification, unless their attribute
        `greater_is_better` is set (`True` or `False`).
    aggregate_metric : str, list, default None
        Aggregation method/s used to combine the metric/s of all levels (series)
        when multiple levels are predicted. If list, the first aggregation method
        is used to select the best parameters. If `None`,
        `['weighted_average', 'average', 'pooling']` is used.

        - 'average': the average (arithmetic mean) of all levels.
        - 'weighted_average': the average of the metrics weighted by the number of
        points evaluated in each level.
        - 'pooling': the values of all levels are pooled and then the metric is
        calculated.
    levels : str, list, default None
        level (`str`) or levels (`list`) at which the forecaster is optimized. 
        If `None`, all levels are taken into account.
    exog : pandas Series, pandas DataFrame, dict, default None
        Exogenous variables.
    n_trials : int, default 20
        Number of parameter settings (configurations) sampled during the search.
        With the default `TPESampler`, the first `n_startup_trials` (10 by
        default) are drawn by random sampling to seed the search, and the TPE
        algorithm only guides sampling from the 11th trial onward. For meaningful
        Bayesian optimization, `n_trials` should therefore be clearly larger than
        10 (for example 30 to 50 or more). This threshold changes if a custom
        sampler with a different `n_startup_trials` is passed in
        `kwargs_create_study`. Larger values increase run time, since each trial
        triggers a full evaluation.
    random_state : int, default 123
        Seed for the `TPESampler` for reproducible output. When a custom sampler
        is passed in `kwargs_create_study`, the seed must be set within that
        sampler. For example `{'sampler': TPESampler(seed=145)}`.
    return_best : bool, default True
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default 'auto'
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.model_selection._utils.select_n_jobs_backtesting. Ignored for
        `OneStepAheadFold`.
    verbose : bool, default False
        Print number of folds used for cv or backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings: bool, default False
        If `True`, skforecast warnings will be suppressed during the hyperparameter
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.
        Otherwise, the warnings of the trials are shown once, at the end of
        the search, with the number of trials in which they are raised. The
        trials skipped because they raise an error are always reported, in a
        single warning.
    output_file : str, default None
        Specifies the filename or full path where the optuna logging output
        should be saved. If `None`, logging output will not be redirected to
        a file.
    kwargs_create_study : dict, default None
        Additional keyword arguments (key, value mappings) to pass to optuna.create_study().
        If default, the direction is inferred from the first metric (see
        `metric`), and a `TPESampler(multivariate=True, group=True, seed=random_state)`
        sampler is used during optimization. If `direction` is passed, the order
        of `results` and the parameters selected with `return_best` follow it
        (a warning is raised if it contradicts the direction of the metric).
    kwargs_study_optimize : dict, default None
        Additional keyword arguments (key, value mappings) to pass to study.optimize().
        The trials whose evaluation raises an error are marked as failed and
        skipped, also when `catch` is passed (its errors are kept). An error in
        `search_space` stops the search.

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column trial_number: optuna trial number for each iteration. Use
          `study.trials[trial_number]` to access the full optuna trial object.
        - column levels: levels configuration for each iteration.
        - column lags: lags configuration for each iteration.
        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration. The resulting 
        metric will be the average of the optimization of all levels.
        - column n_evaluated: number of points that enter the metrics of each
        iteration (true value and prediction not missing), summed over the
        levels. A `MissingValuesWarning` is raised at the end of the search if
        some iterations evaluate less than 95% of the points of the most
        complete one, whose metrics are not comparable.
        - additional n columns with param = value.
    study : optuna Study
        The optuna study object containing all optimization trials. Access the
        best trial via `study.best_trial`.

    """

    cv_name = type(cv).__name__
    if cv_name not in ['TimeSeriesFold', 'OneStepAheadFold']:
        raise TypeError(
            f"`cv` must be an instance of `TimeSeriesFold` or `OneStepAheadFold`. "
            f"Got {type(cv)}."
        )

    forecaster_search = deepcopy_forecaster(forecaster)
    forecaster_name = forecaster_search.__skforecast_tags__['forecaster_name']

    if forecaster_name == 'ForecasterRecursiveMultiSeries':
        series, series_indexes = check_preprocess_series(series)
        if exog is not None:
            series_names_in_ = list(series.keys())
            exog_dict = {serie: None for serie in series_names_in_}
            exog, _ = check_preprocess_exog_multiseries(
                          series_names_in_  = series_names_in_,
                          series_index_type = type(series_indexes[series_names_in_[0]]),
                          exog              = exog,
                          exog_dict         = exog_dict
                      )
    else:
        # TODO: This only applies to wide DataFrames. Delete when input is always dict
        # in all forecasters.
        if return_best and exog is not None and (len(exog) != len(series)):
            raise ValueError(
                f"`exog` must have same number of samples as `series`. "
                f"length `exog`: ({len(exog)}), length `series`: ({len(series)})"
            )

    if cv_name == 'OneStepAheadFold':

        check_one_step_ahead_input(
            forecaster        = forecaster_search,
            cv                = cv,
            metric            = metric,
            series            = series,
            exog              = exog,
            show_progress     = show_progress,
            suppress_warnings = suppress_warnings
        )

        initial_train_size = date_to_index_position(
                                 index        = cv._extract_index(series), 
                                 date_input   = cv.initial_train_size, 
                                 method       = 'validation',
                                 date_literal = 'initial_train_size'
                             )

    if aggregate_metric is None:
        aggregate_metric = ['weighted_average', 'average', 'pooling']
    if isinstance(aggregate_metric, str):
        aggregate_metric = [aggregate_metric]
    allowed_aggregate_metrics = ['average', 'weighted_average', 'pooling']
    if not set(aggregate_metric).issubset(allowed_aggregate_metrics):
        raise ValueError(
            f"Allowed `aggregate_metric` are: {allowed_aggregate_metrics}. "
            f"Got: {aggregate_metric}."
        )

    if not isinstance(metric, list):
        metric = [metric]
    metric = [
        _get_metric(metric=m)
        if isinstance(m, str)
        else add_y_train_argument(m) 
        for m in metric
    ]
    metric_names = [(m if isinstance(m, str) else m.__name__) for m in metric]
    _check_metric_names_search(metric_names=metric_names)

    levels = _initialize_levels_model_selection_multiseries(
                 forecaster = forecaster_search,
                 series     = series,
                 levels     = levels
             )
    add_aggregated_metric = True if len(levels) > 1 else False
    # NOTE: Columns of the metrics table of the backtesting, and rows selected
    # from it, in the order of the columns of the results.
    metric_columns = metric_names
    metrics_rows = aggregate_metric if add_aggregated_metric else [levels[0]]
    if add_aggregated_metric:
        metric_names = [
            f"{metric_name}__{aggregation}"
            for metric_name in metric_names
            for aggregation in aggregate_metric
        ]

    # NOTE: Points evaluated of each level of each trial, for the warning about
    # the points evaluated at the end of the search.
    n_evaluated_levels_trials = {}

    # Objective function using backtesting_forecaster_multiseries
    if cv_name == 'TimeSeriesFold':

        def _objective(
            trial,
            search_space          = search_space,
            forecaster_search     = forecaster_search,
            series                = series,
            cv                    = cv,
            exog                  = exog,
            levels                = levels,
            metric                = metric,
            add_aggregated_metric = add_aggregated_metric,
            aggregate_metric      = aggregate_metric,
            metric_names          = metric_names,
            n_jobs                = n_jobs,
            verbose               = verbose,
            suppress_warnings     = suppress_warnings
        ) -> float:

            search_warnings.set_candidate(trial.number)
            sample = search_space(trial)
            if sample.keys() != trial.params.keys():
                raise ValueError(
                    f"`search_space` dict keys must match the names passed to "
                    f"`trial.suggest_*()`.\n"
                    f"  Dict keys    : {list(sample.keys())}\n"
                    f"  Suggest names: {list(trial.params.keys())}"
                )
            sample_params = {k: v for k, v in sample.items() if k != 'lags'}
            forecaster_search.set_params(sample_params)
            if "lags" in sample:
                forecaster_search.set_lags(sample['lags'])

            # NOTE: Same resolution of `n_jobs='auto'` as in grid search.
            n_jobs_candidate = (
                select_n_jobs_backtesting(
                    forecaster=forecaster_search, refit=cv.refit
                )
                if n_jobs == 'auto'
                else n_jobs
            )
            try:
                metrics, _ = backtesting_forecaster_multiseries(
                                 forecaster            = forecaster_search,
                                 series                = series,
                                 cv                    = cv,
                                 exog                  = exog,
                                 levels                = levels,
                                 metric                = metric,
                                 add_aggregated_metric = add_aggregated_metric,
                                 n_jobs                = n_jobs_candidate,
                                 verbose               = verbose,
                                 show_progress         = False,
                                 suppress_warnings     = suppress_warnings
                             )
            except Exception as e:
                raise _skip_trial(search_warnings, trial, error=e) from e

            n_evaluated, n_evaluated_levels = _get_n_evaluated_search_multiseries(
                                                  metrics = metrics,
                                                  row     = metrics_rows[0],
                                                  levels  = levels
                                              )
            n_evaluated_levels_trials[trial.number] = n_evaluated_levels
            metrics = pd.DataFrame(
                          data    = [_select_metrics_search_multiseries(
                                         metrics      = metrics,
                                         metric_names = metric_columns,
                                         rows         = metrics_rows
                                     )],
                          columns = metric_names
                      )

            # Store all metrics in the trial using optuna's user_attrs mechanism.
            # The number of points evaluated is stored, but not optimized.
            for m_name in metric_names:
                trial.set_user_attr(m_name, float(metrics.loc[0, m_name]))
            trial.set_user_attr(N_EVALUATED_NAME, n_evaluated)

            return metrics.loc[0, metric_names[0]]

    else:

        _SENTINEL = object()
        _MAX_CACHE_SIZE = 10
        _cached_split = {}

        def _objective(
            trial,
            search_space          = search_space,
            forecaster_search     = forecaster_search,
            series                = series,
            cv                    = cv,
            exog                  = exog,
            levels                = levels,
            metric                = metric,
            add_aggregated_metric = add_aggregated_metric,
            aggregate_metric      = aggregate_metric,
            metric_names          = metric_names
        ) -> float:

            search_warnings.set_candidate(trial.number)
            sample = search_space(trial)
            if sample.keys() != trial.params.keys():
                raise ValueError(
                    f"`search_space` dict keys must match the names passed to "
                    f"`trial.suggest_*()`.\n"
                    f"  Dict keys    : {list(sample.keys())}\n"
                    f"  Suggest names: {list(trial.params.keys())}"
                )
            sample_params = {k: v for k, v in sample.items() if k != 'lags'}
            forecaster_search.set_params(sample_params)

            current_lags = sample.get('lags', _SENTINEL)
            if current_lags is not _SENTINEL:
                forecaster_search.set_lags(current_lags)

            # NOTE: The split is cached by lags, also its error. Its warnings
            # are raised outside the trials, so they are shown as they are.
            lags_key = _make_lags_hashable(current_lags, sentinel=_SENTINEL)
            if lags_key not in _cached_split:

                if len(_cached_split) >= _MAX_CACHE_SIZE:
                    _cached_split.pop(next(iter(_cached_split)))

                search_warnings.set_candidate(None)
                try:
                    _cached_split[lags_key] = (
                        *_train_test_split_one_step_ahead_search(
                            forecaster         = forecaster_search,
                            cv                 = cv,
                            initial_train_size = initial_train_size,
                            series             = series,
                            exog               = exog,
                            levels             = levels
                        ),
                        _get_estimator_categorical_set_params(forecaster_search)
                    )
                except Exception as e:
                    _cached_split[lags_key] = e
                finally:
                    search_warnings.set_candidate(trial.number)

            split = _cached_split[lags_key]
            if isinstance(split, Exception):
                raise _skip_trial(search_warnings, trial, error=split) from split
            (
                X_train, y_train, X_test, y_test,
                X_train_encoding, X_test_encoding,
                sample_weight, fit_kwargs,
                _estimator_cat_params
            ) = split
            _restore_estimator_categorical_set_params(
                forecaster_search, _estimator_cat_params
            )

            try:
                metrics, _ = _predict_and_calculate_metrics_one_step_ahead_multiseries(
                                 forecaster            = forecaster_search,
                                 series                = series,
                                 X_train               = X_train,
                                 y_train               = y_train,
                                 X_test                = X_test,
                                 y_test                = y_test,
                                 X_train_encoding      = X_train_encoding,
                                 X_test_encoding       = X_test_encoding,
                                 levels                = levels,
                                 metrics               = metric,
                                 add_aggregated_metric = add_aggregated_metric,
                                 sample_weight         = sample_weight,
                                 fit_kwargs            = fit_kwargs,
                                 return_predictions    = False
                             )
            except Exception as e:
                raise _skip_trial(search_warnings, trial, error=e) from e

            n_evaluated, n_evaluated_levels = _get_n_evaluated_search_multiseries(
                                                  metrics = metrics,
                                                  row     = metrics_rows[0],
                                                  levels  = levels
                                              )
            n_evaluated_levels_trials[trial.number] = n_evaluated_levels
            metrics = pd.DataFrame(
                          data    = [_select_metrics_search_multiseries(
                                         metrics      = metrics,
                                         metric_names = metric_columns,
                                         rows         = metrics_rows
                                     )],
                          columns = metric_names
                      )

            # Store all metrics in the trial using optuna's user_attrs mechanism.
            # The number of points evaluated is stored, but not optimized.
            for m_name in metric_names:
                trial.set_user_attr(m_name, float(metrics.loc[0, m_name]))
            trial.set_user_attr(N_EVALUATED_NAME, n_evaluated)

            return metrics.loc[0, metric_names[0]]

    kwargs_create_study = kwargs_create_study.copy() if kwargs_create_study is not None else {}
    if 'direction' not in kwargs_create_study:
        kwargs_create_study['direction'] = _get_search_direction(
            metric=metric[0], forecaster=forecaster_search
        )
    if 'sampler' not in kwargs_create_study:
        with warnings.catch_warnings():
            warnings.filterwarnings(
                'ignore',
                message='.*multivariate.*|.*group.*'
            )
            kwargs_create_study['sampler'] = TPESampler(
                multivariate=True, group=True, seed=random_state
            )

    kwargs_study_optimize = kwargs_study_optimize.copy() if kwargs_study_optimize is not None else {}
    if show_progress:
        kwargs_study_optimize['show_progress_bar'] = True
    else:
        kwargs_study_optimize.setdefault('show_progress_bar', False)

    if output_file is not None:
        # Redirect optuna logging to file
        optuna.logging.disable_default_handler()
        logger = logging.getLogger('optuna')
        logger.setLevel(logging.INFO)
        for handler in logger.handlers.copy():
            if isinstance(handler, logging.StreamHandler):
                logger.removeHandler(handler)
        handler = logging.FileHandler(output_file, mode="w")
        logger.addHandler(handler)
    else:
        logging.getLogger("optuna").setLevel(logging.WARNING)
        optuna.logging.disable_default_handler()

    study = optuna.create_study(**kwargs_create_study)
    direction = _check_study_direction(study=study, metric=metric[0])

    # NOTE: The warnings of the trials and of the refit are collected and
    # shown once at the end of the search, see `_SearchWarnings`. A study
    # loaded from storage can have trials of previous calls.
    search_warnings = _SearchWarnings(n_candidates=n_trials, candidate_name='trial')
    n_trials_previous = len(study.trials)
    try:
        with search_warnings:
            _optimize_study(
                study                 = study,
                objective             = _objective,
                n_trials              = n_trials,
                kwargs_study_optimize = kwargs_study_optimize,
                filter_logs           = output_file is None
            )
    finally:
        if output_file is not None:
            handler.close()
            logger.removeHandler(handler)

    lags_list = []
    params_list = []
    metrics_data = []
    trial_number_list = []
    for trial in study.get_trials(states=[TrialState.COMPLETE]):
        estimator_params = {k: v for k, v in trial.params.items() if k != 'lags'}
        lags = trial.params.get(
            'lags',
            forecaster_search.lags if hasattr(forecaster_search, 'lags') else None
        )
        params_list.append(estimator_params)
        lags_list.append(lags)
        trial_number_list.append(trial.number)
        metrics_data.append(
            {
                **{m_name: trial.user_attrs[m_name] for m_name in metric_names},
                # NOTE: A study loaded from storage can have trials of a
                # previous version without `n_evaluated`.
                N_EVALUATED_NAME: trial.user_attrs.get(N_EVALUATED_NAME, np.nan)
            }
        )

    if forecaster_name not in ['ForecasterDirectMultiVariate']:
        lags_list = [
            initialize_lags(forecaster_name=forecaster_name, lags=lag)[0]
            for lag in lags_list
        ]
    else:
        lags_list_initialized = []
        for lags in lags_list:
            if isinstance(lags, dict):
                for key in lags:
                    if lags[key] is None:
                        lags[key] = None
                    else:
                        lags[key] = initialize_lags(
                                        forecaster_name = forecaster_name,
                                        lags            = lags[key]
                                    )[0]
            else:
                lags = initialize_lags(
                           forecaster_name = forecaster_name,
                           lags            = lags
                       )[0]
            lags_list_initialized.append(lags)

        lags_list = lags_list_initialized

    results = pd.DataFrame(metrics_data, columns=[*metric_names, N_EVALUATED_NAME])
    results.insert(0, 'trial_number', trial_number_list)
    results.insert(1, 'levels', [levels] * len(results))
    results.insert(2, 'lags', lags_list)
    results.insert(3, 'params', params_list)
    results = (
        results
        .sort_values(by=metric_names[0], ascending=direction == 'minimize')
        .reset_index(drop=True)
    )
    results = pd.concat([results, results['params'].apply(pd.Series)], axis=1)

    # NOTE: The results are empty when every trial is skipped.
    if return_best and not results.empty:

        best_lags = results.loc[0, 'lags']
        best_params = results.loc[0, 'params']
        best_metric = results.loc[0, metric_names[0]]

        # NOTE: Here we use the actual forecaster passed by the user
        forecaster.set_lags(best_lags)
        forecaster.set_params(best_params)

        search_warnings.set_candidate(None)
        with search_warnings:
            series_refit, exog_refit = _drop_series_too_short_refit(
                                           forecaster = forecaster,
                                           series     = series,
                                           exog       = exog
                                       )
            forecaster.fit(
                series                    = series_refit, 
                exog                      = exog_refit, 
                store_in_sample_residuals = True, 
                suppress_warnings         = suppress_warnings
            )

        if len(levels) > 20:
            levels_print = levels[:10] + ["..."] + levels[-10:]
        else:
            levels_print = levels

        if verbose:
            print(
                f"`Forecaster` refitted using the best-found lags and parameters, "
                f"and the whole data set: \n"
                f"  Lags: {best_lags} \n"
                f"  Parameters: {best_params}\n"
                f"  {'Backtesting' if cv_name == 'TimeSeriesFold' else 'One-step-ahead'} "
                f"metric: {best_metric}\n"
                f"  Levels: {levels_print}"
            )

    # NOTE: Only the trials run in this call have the points evaluated of each
    # level, a study loaded from storage (`load_if_exists`) can have others.
    results_trials = results.loc[
        results['trial_number'].isin(n_evaluated_levels_trials)
    ]
    n_evaluated_levels = [
        n_evaluated_levels_trials[trial_number]
        for trial_number in results_trials['trial_number']
    ]
    if _n_evaluated_covers_levels_not_predicted(
        n_evaluated        = results_trials[N_EVALUATED_NAME].to_numpy(),
        n_evaluated_levels = n_evaluated_levels
    ):
        search_warnings.discard(MissingValuesWarning, LEVELS_NOT_PREDICTED_WARNING)
    search_warnings.emit(n_candidates=len(study.trials) - n_trials_previous)

    _warn_n_evaluated_search(
        n_evaluated        = results_trials[N_EVALUATED_NAME].to_numpy(),
        n_evaluated_levels = n_evaluated_levels,
        lags               = results_trials['lags'].tolist(),
        candidate_ids      = results_trials['trial_number'].tolist(),
        levels             = levels,
        candidate_name     = 'trial',
        best_is_first      = len(results_trials) > 0 and results_trials.index[0] == 0
    )

    return results, study

skforecast.model_selection._validation.backtesting_foundation

backtesting_foundation(
    forecaster,
    series,
    cv,
    metric,
    levels=None,
    add_aggregated_metric=True,
    exog=None,
    quantiles=None,
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
)

Backtesting of ForecasterFoundation.

The original forecaster is used directly (no copy) and is not modified: its loaded model weights are reused across the entire backtesting run and fit is never called. Since foundation models are zero-shot, cv is forced to refit=True and fixed_train_size=False only so that the context expands with each fold, and per-fold predictions receive context explicitly, so neither the fit state nor the stored context change during the fold loop.

Parameters:

Name	Type	Description	Default
forecaster	ForecasterFoundation	Forecaster model.	required
series	pandas Series, pandas DataFrame, or dict	Training time series. A single pd.Series runs in single-series mode. A wide pd.DataFrame, a long-format pd.DataFrame with a MultiIndex (series IDs in the first level, DatetimeIndex in the second), or a dict[str, pd.Series] runs in multi-series mode. Long-format DataFrames are normalised internally to a dict before processing.	required
cv	TimeSeriesFold	TimeSeriesFold object with the information needed to split the data into folds. Its window_size is taken from the forecaster (the context length), and the object passed is not modified. refit and fixed_train_size are ignored: the context grows with each fold.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: name of the metric, for example 'mean_absolute_error' or 'root_mean_squared_error'. All accepted names are listed in https://skforecast.org/latest/user_guides/metrics.html. If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. y_train is the series before the first forecast origin. mean_absolute_scaled_error and root_mean_squared_scaled_error scale the error of each prediction with the one-step naive forecast over the observations before its own forecast origin, the same for every forecaster. If list: List containing multiple strings and/or Callables.	required
levels	(str, list)	Time series to be predicted and evaluated. Must be a subset of the names in series, otherwise a ValueError is raised. If None, all series are used.	None
add_aggregated_metric	bool	If True, and multiple series (levels) are predicted, the aggregated metrics (average, weighted average and pooled) are also returned. 'average': the average (arithmetic mean) of all levels. 'weighted_average': the average of the metrics weighted by the number of points evaluated in each level. 'pooling': the values of all levels are pooled and then the metric is calculated.	True
exog	pandas Series, pandas DataFrame, dict	Exogenous variable/s included as predictor/s. Must have the same number of observations as series and should be aligned so that series[i] is regressed on exog[i]. Must also cover the forecast horizon of each fold.	None
quantiles	list	Sequence of quantile levels (between 0 and 1 inclusive) to estimate. For example, quantiles = [0.1, 0.5, 0.9].	None
verbose	bool	Print number of folds and index of training and validation sets used for backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the backtesting process. See skforecast.exceptions.warn_skforecast_categories for more information.	False

Returns:

Name	Type	Description
metrics_levels	pandas DataFrame	Value(s) of the metric(s). In multi-series mode, one row per level, named in the column levels, followed by the rows of the aggregated metrics if add_aggregated_metric is True. In multi-series mode, a level that is not predicted in any fold (for example, a series with no observed values in its context window) has NaN metrics and is left out of the aggregated metrics. A MissingValuesWarning names the levels not predicted in a fold where they have observed values. If quantiles has 2 quantiles, optionally with the median 0.5 (for example, [0.1, 0.9] or [0.1, 0.5, 0.9]), two more columns are added after the metrics, computed with those 2 quantiles as the bounds of the interval (for each level and aggregated in multi-series mode): coverage: share of true values inside the interval (bounds included). interval_width: mean of upper_bound - lower_bound, in the units of the series. The last column, n_evaluated, is the number of points used to compute the metrics (of each level in multi-series mode): true value and prediction not missing, each date counted once (with the prediction of its last fold when the folds overlap). In the aggregated rows, it is the sum over the levels. New in version 0.27.0
backtest_predictions	pandas DataFrame	Value of predictions. The DataFrame includes the following columns, in this order: level: Identifies the series (also with a single series). fold: Indicates the fold number where the prediction was made. step: Step of the forecast horizon at which the prediction was made, from gap + 1 to gap + steps. y_true: True value of the series (NaN if it is missing). pred: Predicted values. New in version 0.27.0: columns step and y_true. If quantiles is provided, one column per quantile is included (e.g. q_0.1, q_0.5, q_0.9) instead of pred. Depending on the relation between steps and fold_stride, the output may include repeated indexes (if fold_stride < steps) or gaps (if fold_stride > steps). See Notes below for more details.

Notes

Note on fold_stride vs. steps:

• If fold_stride == steps, test sets are placed back-to-back without overlap. Each observation appears only once in the output DataFrame, so the index is unique.
• If fold_stride < steps, test sets overlap. Multiple forecasts are generated for the same observations and, therefore, the output DataFrame contains repeated indexes.
• If fold_stride > steps, there are gaps between consecutive test sets. Some observations in the series will not have associated predictions, so the output DataFrame has non-contiguous indexes.

Source code in skforecast/model_selection/_validation.py

@manage_warnings
def backtesting_foundation(
    forecaster: object,
    series: pd.Series | pd.DataFrame | dict,
    cv: TimeSeriesFold,
    metric: str | Callable | list[str | Callable],
    levels: str | list[str] | None = None,
    add_aggregated_metric: bool = True,
    exog: pd.Series | pd.DataFrame | dict | None = None,
    quantiles: list[float] | None = None,
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False,
) -> tuple[pd.DataFrame, pd.DataFrame]:
    """
    Backtesting of ForecasterFoundation.

    The original forecaster is used directly (no copy) and is not modified:
    its loaded model weights are reused across the entire backtesting run
    and `fit` is never called. Since foundation models are zero-shot, `cv`
    is forced to `refit=True` and `fixed_train_size=False` only so that the
    context expands with each fold, and per-fold predictions receive
    `context` explicitly, so neither the fit state nor the stored context
    change during the fold loop.

    Parameters
    ----------
    forecaster : ForecasterFoundation
        Forecaster model.
    series : pandas Series, pandas DataFrame, or dict
        Training time series. A single `pd.Series` runs in single-series
        mode. A wide `pd.DataFrame`, a long-format `pd.DataFrame` with a
        MultiIndex (series IDs in the first level, `DatetimeIndex` in the
        second), or a `dict[str, pd.Series]` runs in multi-series mode.
        Long-format DataFrames are normalised internally to a dict before
        processing.
    cv : TimeSeriesFold
        TimeSeriesFold object with the information needed to split the data into
        folds. Its `window_size` is taken from the forecaster (the context
        length), and the object passed is not modified. `refit` and
        `fixed_train_size` are ignored: the context grows with each fold.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: name of the metric, for example `'mean_absolute_error'`
        or `'root_mean_squared_error'`. All accepted names are listed in
        https://skforecast.org/latest/user_guides/metrics.html.
        - If `Callable`: Function with arguments `y_true`, `y_pred` and
        `y_train` (Optional) that returns a float. `y_train` is the series
        before the first forecast origin. `mean_absolute_scaled_error` and
        `root_mean_squared_scaled_error` scale the error of each prediction
        with the one-step naive forecast over the observations before its own
        forecast origin, the same for every forecaster.
        - If `list`: List containing multiple strings and/or Callables.
    levels : str, list, default None
        Time series to be predicted and evaluated. Must be a subset of the
        names in `series`, otherwise a `ValueError` is raised. If `None`,
        all series are used.
    add_aggregated_metric : bool, default True
        If `True`, and multiple series (`levels`) are predicted, the aggregated
        metrics (average, weighted average and pooled) are also returned.

        - 'average': the average (arithmetic mean) of all levels.
        - 'weighted_average': the average of the metrics weighted by the
        number of points evaluated in each level.
        - 'pooling': the values of all levels are pooled and then the metric
        is calculated.
    exog : pandas Series, pandas DataFrame, dict, default None
        Exogenous variable/s included as predictor/s. Must have the same
        number of observations as `series` and should be aligned so that
        `series[i]` is regressed on `exog[i]`. Must also cover the
        forecast horizon of each fold.
    quantiles : list, default None
        Sequence of quantile levels (between 0 and 1 inclusive) to estimate.
        For example, `quantiles = [0.1, 0.5, 0.9]`.
    verbose : bool, default False
        Print number of folds and index of training and validation sets used
        for backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings : bool, default False
        If `True`, skforecast warnings will be suppressed during the
        backtesting process. See
        skforecast.exceptions.warn_skforecast_categories for more
        information.

    Returns
    -------
    metrics_levels : pandas DataFrame
        Value(s) of the metric(s). In multi-series mode, one row per level,
        named in the column `levels`, followed by the rows of the aggregated
        metrics if `add_aggregated_metric` is `True`.
        In multi-series mode, a level that is not predicted in any fold (for
        example, a series with no observed values in its context window) has
        NaN metrics and is left out of the aggregated metrics. A
        `MissingValuesWarning` names the levels not predicted in a fold where
        they have observed values.
        If `quantiles` has 2 quantiles, optionally with the median 0.5 (for
        example, `[0.1, 0.9]` or `[0.1, 0.5, 0.9]`), two more columns are 
        added after the metrics, computed with those 2 quantiles as the bounds
        of the interval (for each level and aggregated in multi-series mode):

        - coverage: share of true values inside the interval (bounds included).
        - interval_width: mean of `upper_bound - lower_bound`, in the units of
        the series.

        The last column, `n_evaluated`, is the number of points used to
        compute the metrics (of each level in multi-series mode): true value
        and prediction not missing, each date counted once (with the
        prediction of its last fold when the folds overlap). In the aggregated
        rows, it is the sum over the levels.

        **New in version 0.27.0**
    backtest_predictions : pandas DataFrame
        Value of predictions. The DataFrame includes the following columns, in
        this order:

        - level: Identifies the series (also with a single series).
        - fold: Indicates the fold number where the prediction was made.
        - step: Step of the forecast horizon at which the prediction was made,
        from `gap + 1` to `gap + steps`.
        - y_true: True value of the series (NaN if it is missing).
        - pred: Predicted values.

        **New in version 0.27.0**: columns `step` and `y_true`.

        If `quantiles` is provided, one column per quantile is included
        (e.g. `q_0.1`, `q_0.5`, `q_0.9`) instead of pred.

        Depending on the relation between `steps` and `fold_stride`, the
        output may include repeated indexes (if `fold_stride < steps`) or
        gaps (if `fold_stride > steps`). See Notes below for more details.

    Notes
    -----
    Note on `fold_stride` vs. `steps`:

    - If `fold_stride == steps`, test sets are placed back-to-back without
    overlap. Each observation appears only once in the output DataFrame, so
    the index is unique.
    - If `fold_stride < steps`, test sets overlap. Multiple forecasts are
    generated for the same observations and, therefore, the output DataFrame
    contains repeated indexes.
    - If `fold_stride > steps`, there are gaps between consecutive test sets.
    Some observations in the series will not have associated predictions, so
    the output DataFrame has non-contiguous indexes.

    """

    if type(forecaster).__name__ != 'ForecasterFoundation':
        raise TypeError(
            "`forecaster` must be of type `ForecasterFoundation`. For all "
            "other types of forecasters use the other functions available in "
            "the `model_selection` module."
        )

    series_dict, series_indexes = check_preprocess_series_foundation(series)
    series_names_in_ = list(series_dict.keys())

    exog_dict = None
    if exog is not None:
        exog_dict, _ = check_preprocess_exog_multiseries(
            series_names_in_  = series_names_in_,
            series_index_type = type(series_indexes[series_names_in_[0]]),
            exog              = exog,
            exog_dict         = {series_name: None for series_name in series_names_in_},
        )

        # NOTE: As no trim is applied to the series, it is only needed to align exog.
        series_dict, exog_dict = align_series_and_exog_multiseries(
                                     series_dict      = series_dict,
                                     exog_dict        = exog_dict,
                                     trim_series_nan  = False,
                                 )

    if quantiles is not None:
        if 0.5 not in quantiles:
            warnings.warn(
                "The median quantile (0.5) is required to compute the main "
                "predictions and metrics. Since it is not included in `quantiles`, "
                "it will be added automatically.",
                IgnoredArgumentWarning
            )
            quantiles = sorted(quantiles + [0.5])

    check_backtesting_input(
        forecaster             = forecaster,
        cv                     = cv,
        series                 = series_dict,
        metric                 = metric,
        add_aggregated_metric  = add_aggregated_metric,
        exog                   = exog_dict,
        interval               = quantiles,
        show_progress          = show_progress,
        suppress_warnings      = suppress_warnings,
    )

    metrics_levels, backtest_predictions = _backtesting_foundation(
        forecaster            = forecaster,
        series                = series_dict,
        cv                    = cv,
        metric                = metric,
        levels                = levels,
        add_aggregated_metric = add_aggregated_metric,
        exog                  = exog_dict,
        quantiles             = quantiles,
        verbose               = verbose,
        show_progress         = show_progress,
        suppress_warnings     = suppress_warnings,
    )

    return metrics_levels, backtest_predictions

skforecast.model_selection._search.bayesian_search_foundation

bayesian_search_foundation(
    forecaster,
    series,
    cv,
    search_space,
    metric,
    aggregate_metric=None,
    levels=None,
    exog=None,
    n_trials=20,
    random_state=123,
    return_best=True,
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
    output_file=None,
    kwargs_create_study=None,
    kwargs_study_optimize=None,
)

Bayesian search for inference-time configuration of a ForecasterFoundation object using the optuna library.

Foundation models are zero-shot (no weight training), so only inference-time parameters are tuned. The most impactful parameter is context_length, which controls how many past observations are fed to the model; shorter contexts reduce memory and latency while potentially sacrificing accuracy. Additional adapter-specific parameters may be tuned (see search_space).

For each trial, the sampled configuration is evaluated with backtesting_foundation (TimeSeriesFold). Returns a ranked results DataFrame and the optuna Study object.

Only TimeSeriesFold is supported; OneStepAheadFold is incompatible with zero-shot foundation semantics.

Parameters:

Name	Type	Description	Default
forecaster	ForecasterFoundation	Forecaster model. Must be a ForecasterFoundation instance.	required
series	pandas Series, pandas DataFrame, dict	Training time series. Passed directly to backtesting_foundation.	required
cv	TimeSeriesFold	TimeSeriesFold object with the information needed to split the data into folds. Its window_size is taken from the forecaster (the context length), and the object passed is not modified. refit and fixed_train_size are ignored: the context grows with each fold. OneStepAheadFold is not accepted and raises TypeError.	required
search_space	Callable	Function with argument trial which returns a dictionary with parameter names (str) as keys and Trial object from optuna (trial.suggest_float, trial.suggest_int, trial.suggest_categorical) as values. All keys must match the names passed to the suggest calls (trial.params keys). Tunable parameters per model: All models: context_length (int or categorical list). Amazon Chronos-2: cross_learning (bool). Soda-INRIA TabICL, Prior Labs TabPFN-TS: point_estimate (str), temporal_features (list). Synthefy Nori: point_estimate (str), add_calendar_features (bool), n_fourier_terms (int). Note: changing model_id, device arguments, or torch_dtype forces a full model reload and is expensive. Changing context_length also forces a reload on TimesFM 2.5, Moirai-2, TabICL and TabPFN-TS (TimesFM 3.0 is unaffected, since context_length does not affect its loaded model).	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If str: name of the metric, for example 'mean_absolute_error' or 'root_mean_squared_error'. All accepted names are listed in https://skforecast.org/latest/user_guides/metrics.html. If Callable: Function with arguments y_true, y_pred and y_train (optional) that returns a float. y_train is the series before the first forecast origin. mean_absolute_scaled_error and root_mean_squared_scaled_error scale the error of each prediction with the one-step naive forecast over the observations before its own forecast origin, the same for every forecaster. If list: List containing multiple strings and/or callables. The first metric ranks the candidates. Scikit-learn metrics ending in _score are maximized and those ending in _error, _loss or _deviance are minimized. Other functions are minimized in regression and maximized in classification, unless their attribute greater_is_better is set (True or False).	required
aggregate_metric	(str, list)	Aggregation method(s) used to combine the metric(s) of all levels (series) when multiple levels are predicted. If list, the first aggregation method is used to select the best parameters. If None, ['weighted_average', 'average', 'pooling'] is used. 'average': the arithmetic mean of all levels. 'weighted_average': the mean weighted by the number of points evaluated in each level. 'pooling': values of all levels are pooled and then the metric is calculated.	None
levels	(str, list)	Level(s) at which the forecaster is optimized. If None, all levels present in series are used.	None
exog	pandas Series, pandas DataFrame, dict	Exogenous variables.	None
n_trials	int	Number of parameter settings (configurations) sampled during the search. With the default TPESampler, the first n_startup_trials (10 by default) are drawn by random sampling to seed the search, and the TPE algorithm only guides sampling from the 11th trial onward. For meaningful Bayesian optimization, n_trials should therefore be clearly larger than 10 (for example 30 to 50 or more). This threshold changes if a custom sampler with a different n_startup_trials is passed in kwargs_create_study. Larger values increase run time, since each trial triggers a full evaluation.	20
random_state	int	Seed for the TPESampler for reproducible output. When a custom sampler is passed in kwargs_create_study, the seed must be set within that sampler. For example {'sampler': TPESampler(seed=145)}.	123
return_best	bool	Refit the original forecaster using the best-found parameters on the whole data set after the search.	True
verbose	bool	Print number of folds used for cross-validation.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the search. See skforecast.exceptions.warn_skforecast_categories for details. Otherwise, the warnings of the trials are shown once, at the end of the search, with the number of trials in which they are raised. The trials skipped because they raise an error are always reported, in a single warning.	False
output_file	str	Filename or full path where optuna logging output should be saved. If None, logging output is not redirected.	None
kwargs_create_study	dict	Additional keyword arguments passed to optuna.create_study(). If not provided, the direction is inferred from the first metric (see metric), and a TPESampler(multivariate=True, group=True, seed=random_state) sampler is used. If direction is passed, the order of results and the parameters selected with return_best follow it (a warning is raised if it contradicts the direction of the metric).	None
kwargs_study_optimize	dict	Additional keyword arguments passed to study.optimize(). The trials whose evaluation raises an error are marked as failed and skipped, also when catch is passed (its errors are kept). An error in search_space stops the search.	None

Returns:

Name	Type	Description
results	pandas DataFrame	Results for each combination of parameters, sorted by the first metric. column trial_number: optuna trial number. Use study.trials[trial_number] to access the full optuna trial object. column levels: levels evaluated in each trial. column params: parameter configuration for each trial. column metric: metric value estimated for each trial. column n_evaluated: number of points that enter the metrics of each trial (true value and prediction not missing), summed over the levels. additional columns with one column per searched parameter (expanded from params).
study	optuna Study	The optuna study object containing all optimization trials. Access the best trial via study.best_trial.

Source code in skforecast/model_selection/_search.py

@manage_warnings
def bayesian_search_foundation(
    forecaster: object,
    series: pd.Series | pd.DataFrame | dict[str, pd.Series | pd.DataFrame],
    cv: TimeSeriesFold,
    search_space: Callable,
    metric: str | Callable | list[str | Callable],
    aggregate_metric: str | list[str] | None = None,
    levels: str | list[str] | None = None,
    exog: pd.Series | pd.DataFrame | dict | None = None,
    n_trials: int = 20,
    random_state: int = 123,
    return_best: bool = True,
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: str | None = None,
    kwargs_create_study: dict | None = None,
    kwargs_study_optimize: dict | None = None,
) -> tuple[pd.DataFrame, object]:
    """
    Bayesian search for inference-time configuration of a ForecasterFoundation
    object using the optuna library.

    Foundation models are zero-shot (no weight training), so only inference-time
    parameters are tuned. The most impactful parameter is `context_length`, which
    controls how many past observations are fed to the model; shorter contexts
    reduce memory and latency while potentially sacrificing accuracy. Additional
    adapter-specific parameters may be tuned (see `search_space`).

    For each trial, the sampled configuration is evaluated with
    `backtesting_foundation` (`TimeSeriesFold`). Returns a ranked results 
    DataFrame and the optuna `Study` object.

    Only `TimeSeriesFold` is supported; `OneStepAheadFold` is incompatible
    with zero-shot foundation semantics.

    Parameters
    ----------
    forecaster : ForecasterFoundation
        Forecaster model. Must be a `ForecasterFoundation` instance.
    series : pandas Series, pandas DataFrame, dict
        Training time series. Passed directly to `backtesting_foundation`.
    cv : TimeSeriesFold
        TimeSeriesFold object with the information needed to split the data into
        folds. Its `window_size` is taken from the forecaster (the context
        length), and the object passed is not modified. `refit` and
        `fixed_train_size` are ignored: the context grows with each fold.
        `OneStepAheadFold` is not accepted and raises `TypeError`.
    search_space : Callable
        Function with argument `trial` which returns a dictionary with parameter
        names (`str`) as keys and Trial object from optuna
        (`trial.suggest_float`, `trial.suggest_int`,
        `trial.suggest_categorical`) as values. All keys must match the names
        passed to the suggest calls (`trial.params` keys).

        Tunable parameters per model:

        - All models: `context_length` (int or categorical list).
        - Amazon Chronos-2: `cross_learning` (bool).
        - Soda-INRIA TabICL, Prior Labs TabPFN-TS: `point_estimate` (str),
        `temporal_features` (list).
        - Synthefy Nori: `point_estimate` (str), `add_calendar_features`
        (bool), `n_fourier_terms` (int).

        Note: changing `model_id`, device arguments, or `torch_dtype` forces a
        full model reload and is expensive. Changing `context_length` also
        forces a reload on TimesFM 2.5, Moirai-2, TabICL and TabPFN-TS (TimesFM
        3.0 is unaffected, since `context_length` does not affect its loaded
        model).
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `str`: name of the metric, for example `'mean_absolute_error'` or
          `'root_mean_squared_error'`. All accepted names are listed in
          https://skforecast.org/latest/user_guides/metrics.html.
        - If `Callable`: Function with arguments `y_true`, `y_pred` and
          `y_train` (optional) that returns a float. `y_train` is the series
          before the first forecast origin. `mean_absolute_scaled_error` and
          `root_mean_squared_scaled_error` scale the error of each prediction
          with the one-step naive forecast over the observations before its
          own forecast origin, the same for every forecaster.
        - If `list`: List containing multiple strings and/or callables.

        The first metric ranks the candidates. Scikit-learn metrics ending in
        `_score` are maximized and those ending in `_error`, `_loss` or
        `_deviance` are minimized. Other functions are minimized in regression
        and maximized in classification, unless their attribute
        `greater_is_better` is set (`True` or `False`).
    aggregate_metric : str, list, default None
        Aggregation method(s) used to combine the metric(s) of all levels (series)
        when multiple levels are predicted. If list, the first aggregation method
        is used to select the best parameters. If `None`,
        `['weighted_average', 'average', 'pooling']` is used.

        - 'average': the arithmetic mean of all levels.
        - 'weighted_average': the mean weighted by the number of points evaluated
        in each level.
        - 'pooling': values of all levels are pooled and then the metric is
        calculated.
    levels : str, list, default None
        Level(s) at which the forecaster is optimized. If `None`, all levels
        present in `series` are used.
    exog : pandas Series, pandas DataFrame, dict, default None
        Exogenous variables.
    n_trials : int, default 20
        Number of parameter settings (configurations) sampled during the search.
        With the default `TPESampler`, the first `n_startup_trials` (10 by
        default) are drawn by random sampling to seed the search, and the TPE
        algorithm only guides sampling from the 11th trial onward. For meaningful
        Bayesian optimization, `n_trials` should therefore be clearly larger than
        10 (for example 30 to 50 or more). This threshold changes if a custom
        sampler with a different `n_startup_trials` is passed in
        `kwargs_create_study`. Larger values increase run time, since each trial
        triggers a full evaluation.
    random_state : int, default 123
        Seed for the `TPESampler` for reproducible output. When a custom sampler
        is passed in `kwargs_create_study`, the seed must be set within that
        sampler. For example `{'sampler': TPESampler(seed=145)}`.
    return_best : bool, default True
        Refit the original `forecaster` using the best-found parameters on the
        whole data set after the search.
    verbose : bool, default False
        Print number of folds used for cross-validation.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings : bool, default False
        If `True`, skforecast warnings will be suppressed during the search.
        See `skforecast.exceptions.warn_skforecast_categories` for details.
        Otherwise, the warnings of the trials are shown once, at the end of
        the search, with the number of trials in which they are raised. The
        trials skipped because they raise an error are always reported, in a
        single warning.
    output_file : str, default None
        Filename or full path where optuna logging output should be saved. If
        `None`, logging output is not redirected.
    kwargs_create_study : dict, default None
        Additional keyword arguments passed to `optuna.create_study()`. If not
        provided, the direction is inferred from the first metric (see
        `metric`), and a `TPESampler(multivariate=True, group=True, seed=random_state)`
        sampler is used. If `direction` is passed, the order of `results` and
        the parameters selected with `return_best` follow it (a warning is
        raised if it contradicts the direction of the metric).
    kwargs_study_optimize : dict, default None
        Additional keyword arguments passed to `study.optimize()`.
        The trials whose evaluation raises an error are marked as failed and
        skipped, also when `catch` is passed (its errors are kept). An error in
        `search_space` stops the search.

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters, sorted by the first metric.

        - column trial_number: optuna trial number. Use
        `study.trials[trial_number]` to access the full optuna trial object.
        - column levels: levels evaluated in each trial.
        - column params: parameter configuration for each trial.
        - column metric: metric value estimated for each trial.
        - column n_evaluated: number of points that enter the metrics of each
        trial (true value and prediction not missing), summed over the levels.
        - additional columns with one column per searched parameter (expanded
        from `params`).
    study : optuna Study
        The optuna study object containing all optimization trials. Access the
        best trial via `study.best_trial`.

    """

    if type(forecaster).__name__ != "ForecasterFoundation":
        raise TypeError(
            f"`forecaster` must be a `ForecasterFoundation` instance. "
            f"Got {type(forecaster).__name__}."
        )

    if type(cv).__name__ != "TimeSeriesFold":
        raise TypeError(
            f"`cv` must be a `TimeSeriesFold` instance. Got {type(cv).__name__}. "
            f"`OneStepAheadFold` is not supported for `ForecasterFoundation`."
        )

    forecaster_search = deepcopy_forecaster(forecaster)

    if aggregate_metric is None:
        aggregate_metric = ["weighted_average", "average", "pooling"]
    if isinstance(aggregate_metric, str):
        aggregate_metric = [aggregate_metric]
    allowed_aggregate_metrics = ["average", "weighted_average", "pooling"]
    if not set(aggregate_metric).issubset(allowed_aggregate_metrics):
        raise ValueError(
            f"Allowed `aggregate_metric` are: {allowed_aggregate_metrics}. "
            f"Got: {aggregate_metric}."
        )

    if not isinstance(metric, list):
        metric = [metric]
    metric = [
        _get_metric(metric=m) 
        if isinstance(m, str) 
        else add_y_train_argument(m)
        for m in metric
    ]
    metric_names = [(m if isinstance(m, str) else m.__name__) for m in metric]
    _check_metric_names_search(metric_names=metric_names)

    if isinstance(series, pd.Series):
        levels_resolved = [series.name]
    else:
        levels_resolved = _initialize_levels_model_selection_multiseries(
                              forecaster = forecaster_search,
                              series     = series,
                              levels     = levels,
                          )

    add_aggregated_metric = len(levels_resolved) > 1
    # NOTE: Columns of the metrics table of the backtesting, and rows selected
    # from it, in the order of the columns of the results.
    metric_columns = metric_names
    metrics_rows = aggregate_metric if add_aggregated_metric else [levels_resolved[0]]
    if add_aggregated_metric:
        metric_names = [
            f"{metric_name}__{aggregation}"
            for metric_name in metric_names
            for aggregation in aggregate_metric
        ]

    def _objective(
        trial,
        search_space          = search_space,
        forecaster_search     = forecaster_search,
        series                = series,
        cv                    = cv,
        exog                  = exog,
        levels                = levels_resolved,
        metric                = metric,
        metric_names          = metric_names,
        add_aggregated_metric = add_aggregated_metric,
        aggregate_metric      = aggregate_metric,
        verbose               = verbose,
        suppress_warnings     = suppress_warnings,
    ) -> float:

        search_warnings.set_candidate(trial.number)
        sample = search_space(trial)
        if sample.keys() != trial.params.keys():
            raise ValueError(
                f"`search_space` dict keys must match the names passed to "
                f"`trial.suggest_*()`.\n"
                f"  Dict keys    : {list(sample.keys())}\n"
                f"  Suggest names: {list(trial.params.keys())}"
            )

        adapter = forecaster_search.estimator.adapter
        _valid_params = set(adapter.get_params().keys())
        _invalid = set(sample.keys()) - _valid_params
        if _invalid:
            raise ValueError(
                f"The following keys in `search_space` are not valid parameters "
                f"for {type(adapter).__name__}: {sorted(_invalid)}. Valid "
                f"parameters are: {sorted(_valid_params)}."
            )

        forecaster_search.set_params(sample)

        try:
            metrics, _ = backtesting_foundation(
                forecaster            = forecaster_search,
                series                = series,
                cv                    = cv,
                exog                  = exog,
                levels                = levels,
                metric                = metric,
                add_aggregated_metric = add_aggregated_metric,
                verbose               = verbose,
                show_progress         = False,
                suppress_warnings     = suppress_warnings,
            )
        except Exception as e:
            raise _skip_trial(search_warnings, trial, error=e) from e

        if "levels" in metrics.columns:
            n_evaluated, _ = _get_n_evaluated_search_multiseries(
                                 metrics = metrics,
                                 row     = metrics_rows[0],
                                 levels  = levels
                             )
            metrics = pd.DataFrame(
                data    = [_select_metrics_search_multiseries(
                              metrics      = metrics,
                              metric_names = metric_columns,
                              rows         = metrics_rows
                          )],
                columns = metric_names,
            )
        else:
            n_evaluated = int(metrics[N_EVALUATED_NAME].iloc[0])
            metrics = pd.DataFrame(
                data    = [metrics[metric_columns].to_numpy().ravel()],
                columns = metric_names,
            )
        metrics = metrics.reset_index(drop=True)

        for m_name in metric_names:
            trial.set_user_attr(m_name, float(metrics.loc[0, m_name]))
        trial.set_user_attr(N_EVALUATED_NAME, n_evaluated)

        return metrics.loc[0, metric_names[0]]

    kwargs_create_study = (
        kwargs_create_study.copy() if kwargs_create_study is not None else {}
    )
    if "direction" not in kwargs_create_study:
        kwargs_create_study["direction"] = _get_search_direction(
            metric=metric[0], forecaster=forecaster_search
        )
    if "sampler" not in kwargs_create_study:
        with warnings.catch_warnings():
            warnings.filterwarnings("ignore", message=".*multivariate.*|.*group.*")
            kwargs_create_study["sampler"] = TPESampler(
                multivariate=True, group=True, seed=random_state
            )

    kwargs_study_optimize = (
        kwargs_study_optimize.copy() if kwargs_study_optimize is not None else {}
    )
    if show_progress:
        kwargs_study_optimize["show_progress_bar"] = True
    else:
        kwargs_study_optimize.setdefault("show_progress_bar", False)

    if output_file is not None:
        optuna.logging.disable_default_handler()
        logger = logging.getLogger("optuna")
        logger.setLevel(logging.INFO)
        for handler in logger.handlers.copy():
            if isinstance(handler, logging.StreamHandler):
                logger.removeHandler(handler)
        handler = logging.FileHandler(output_file, mode="w")
        logger.addHandler(handler)
    else:
        logging.getLogger("optuna").setLevel(logging.WARNING)
        optuna.logging.disable_default_handler()

    study = optuna.create_study(**kwargs_create_study)
    direction = _check_study_direction(study=study, metric=metric[0])

    # NOTE: The warnings of the trials and of the refit are collected and
    # shown once at the end of the search, see `_SearchWarnings`. A study
    # loaded from storage can have trials of previous calls.
    search_warnings = _SearchWarnings(n_candidates=n_trials, candidate_name="trial")
    n_trials_previous = len(study.trials)
    try:
        with search_warnings:
            _optimize_study(
                study                 = study,
                objective             = _objective,
                n_trials              = n_trials,
                kwargs_study_optimize = kwargs_study_optimize,
                filter_logs           = output_file is None
            )
    finally:
        if output_file is not None:
            handler.close()
            logger.removeHandler(handler)

    trial_number_list = []
    params_list = []
    metrics_data = []
    for trial in study.get_trials(states=[TrialState.COMPLETE]):
        trial_number_list.append(trial.number)
        params_list.append(trial.params)
        metrics_data.append(
            {
                **{m_name: trial.user_attrs[m_name] for m_name in metric_names},
                # NOTE: A study loaded from storage can have trials of a
                # previous version without `n_evaluated`.
                N_EVALUATED_NAME: trial.user_attrs.get(N_EVALUATED_NAME, np.nan)
            }
        )

    results = pd.DataFrame(metrics_data, columns=[*metric_names, N_EVALUATED_NAME])
    results.insert(0, "trial_number", trial_number_list)
    results.insert(1, "levels", [levels_resolved] * len(results))
    results.insert(2, "params", params_list)
    results = (
        results.sort_values(
            by=metric_names[0], ascending=direction == "minimize"
        )
        .reset_index(drop=True)
    )
    results = pd.concat([results, results["params"].apply(pd.Series)], axis=1)

    # NOTE: The results are empty when every trial is skipped.
    if return_best and not results.empty:

        best_params = results.loc[0, "params"]
        best_metric = results.loc[0, metric_names[0]]

        # NOTE: Here we use the actual forecaster passed by the user
        forecaster.set_params(best_params)
        search_warnings.set_candidate(None)
        with search_warnings:
            forecaster.fit(series=series, exog=exog)

        if verbose:
            levels_print = (
                levels_resolved[:10] + ["..."] + levels_resolved[-10:]
                if len(levels_resolved) > 20
                else levels_resolved
            )
            print(
                f"`Forecaster` refitted using the best-found parameters, "
                f"and the whole data set: \n"
                f"  Parameters: {best_params}\n"
                f"  Backtesting metric: {best_metric}\n"
                f"  Levels: {levels_print}"
            )

    search_warnings.emit(n_candidates=len(study.trials) - n_trials_previous)

    return results, study

skforecast.model_selection._validation.backtesting_stats

backtesting_stats(
    forecaster,
    y,
    cv,
    metric,
    exog=None,
    interval=None,
    alpha=None,
    freeze_params=True,
    n_jobs="auto",
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
)

Backtesting of ForecasterStats.

A copy of the original forecaster is created so that it is not modified during the process.

Parameters:

Name	Type	Description	Default
forecaster	ForecasterStats	Forecaster model.	required
y	pandas Series	Training time series.	required
cv	TimeSeriesFold	TimeSeriesFold object with the information needed to split the data into folds. Its window_size is taken from the forecaster, and the object passed is not modified. With estimators other than Sarimax, refit is set to True.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: name of the metric, for example 'mean_absolute_error' or 'root_mean_squared_error'. All accepted names are listed in https://skforecast.org/latest/user_guides/metrics.html. If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. y_train is the series before the first forecast origin. mean_absolute_scaled_error and root_mean_squared_scaled_error scale the error of each prediction with the one-step naive forecast over the observations before its own forecast origin, the same for every forecaster. If list: List containing multiple strings and/or Callables.	required
exog	pandas Series, pandas DataFrame	Exogenous variable/s included as predictor/s. Must have the same number of observations as y and should be aligned so that y[i] is regressed on exog[i].	None
interval	(list, tuple)	Confidence of the prediction interval estimated. The values must be symmetric. Sequence of quantiles to compute, which must be between 0 and 1 inclusive. For example, interval of 95% should be as interval = [0.025, 0.975]. If both alpha and interval are None, no intervals are estimated. Changed in version 0.23.0: interval is now expressed as quantiles (0-1) instead of percentiles (0-100). Passing percentiles is not longer supported and will raise a ValueError. Changed in version 0.27.0: passing both alpha and interval raises a ValueError (previously, alpha took precedence), and interval comes before alpha in the signature.	None
alpha	float	Alias of interval expressed as the significance level: the confidence intervals for the forecasts are (1 - alpha) %. For example, alpha=0.05 is equivalent to interval=[0.025, 0.975]. It cannot be used together with interval.	None
freeze_params	bool	Determines whether to freeze the model parameters after the first fit for estimators that perform automatic model selection (Arima with order=None or seasonal_order=None, Ets with model='ZZZ'). If True, the parameters selected in the first fit (best_params_) are reused in all subsequent refits, so the automatic selection runs only once. Arima keeps the orders, intercept, drift and Box-Cox lambda, and re-estimates the coefficients in each refit. Ets keeps the model type, smoothing parameters and Box-Cox lambda, and re-estimates the initial states. If False, automatic model selection is performed independently in each refit, allowing parameters to adapt across folds. This increases runtime and adds an estimator_params column to the output with the model selected in each fold.	True
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.model_selection._utils.select_n_jobs_backtesting.	'auto'
verbose	bool	Print number of folds and index of training and validation sets used for backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the backtesting process. See skforecast.exceptions.warn_skforecast_categories for more information.	False

Returns:

Name	Type	Description
metric_values	pandas DataFrame	Value(s) of the metric(s), one row per estimator, named in the first column, estimator_id, if the forecaster has several. If interval (or alpha) is not None, two more columns are added after the metrics: coverage: share of true values inside the interval (bounds included). interval_width: mean of upper_bound - lower_bound, in the units of the series. The last column, n_evaluated, is the number of points used to compute the metrics: true value and prediction not missing, each date counted once (with the prediction of its last fold when the folds overlap). New in version 0.27.0
backtest_predictions	pandas DataFrame	Value of predictions. The DataFrame includes the following columns, in this order: estimator_id: Identifier of the estimator, only if the forecaster has several (one row per estimator and date). fold: Indicates the fold number where the prediction was made. step: Step of the forecast horizon at which the prediction was made, from gap + 1 to gap + steps. y_true: True value of the series (NaN if it is missing). pred: Predicted values for the corresponding series and time steps. New in version 0.27.0: columns step and y_true. If interval is not None, additional columns are included: lower_bound: lower bound of the interval. upper_bound: upper bound of the interval. If freeze_params is False, an additional column is included: estimator_params: parameters used in the estimator for each fold. Depending on the relation between steps and fold_stride, the output may include repeated indexes (if fold_stride < steps) or gaps (if fold_stride > steps). See Notes below for more details.

Notes

Note on fold_stride vs. steps:

• If fold_stride == steps, test sets are placed back-to-back without overlap. Each observation appears only once in the output DataFrame, so the index is unique.
• If fold_stride < steps, test sets overlap. Multiple forecasts are generated for the same observations and, therefore, the output DataFrame contains repeated indexes.
• If fold_stride > steps, there are gaps between consecutive test sets. Some observations in the series will not have associated predictions, so the output DataFrame has non-contiguous indexes.

Source code in skforecast/model_selection/_validation.py

def backtesting_stats(
    forecaster: object,
    y: pd.Series,
    cv: TimeSeriesFold,
    metric: str | Callable | list[str | Callable],
    exog: pd.Series | pd.DataFrame | None = None,
    interval: list[float] | tuple[float, ...] | None = None,
    alpha: float | None = None,
    freeze_params: bool = True,
    n_jobs: int | str = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False,
) -> tuple[pd.DataFrame, pd.DataFrame]:
    """
    Backtesting of ForecasterStats.

    A copy of the original forecaster is created so that it is not modified during 
    the process.

    Parameters
    ----------
    forecaster : ForecasterStats
        Forecaster model.
    y : pandas Series
        Training time series.
    cv : TimeSeriesFold
        TimeSeriesFold object with the information needed to split the data into
        folds. Its `window_size` is taken from the forecaster, and the object
        passed is not modified. With estimators other than `Sarimax`,
        `refit` is set to `True`.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: name of the metric, for example `'mean_absolute_error'`
        or `'root_mean_squared_error'`. All accepted names are listed in
        https://skforecast.org/latest/user_guides/metrics.html.
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float. `y_train` is the series before the
        first forecast origin. `mean_absolute_scaled_error` and
        `root_mean_squared_scaled_error` scale the error of each prediction
        with the one-step naive forecast over the observations before its own
        forecast origin, the same for every forecaster.
        - If `list`: List containing multiple strings and/or Callables.
    exog : pandas Series, pandas DataFrame, default None
        Exogenous variable/s included as predictor/s. Must have the same
        number of observations as `y` and should be aligned so that y[i] is
        regressed on exog[i].
    interval : list, tuple, default None
        Confidence of the prediction interval estimated. The values must be
        symmetric. Sequence of quantiles to compute, which must be between 
        0 and 1 inclusive. For example, interval of 95% should be as 
        `interval = [0.025, 0.975]`. If both `alpha` and `interval` are
        `None`, no intervals are estimated.

        **Changed in version 0.23.0:** `interval` is now expressed as
        quantiles (0-1) instead of percentiles (0-100). Passing percentiles
        is not longer supported and will raise a `ValueError`.

        **Changed in version 0.27.0:** passing both `alpha` and `interval`
        raises a `ValueError` (previously, `alpha` took precedence), and
        `interval` comes before `alpha` in the signature.
    alpha : float, default None
        Alias of `interval` expressed as the significance level: the
        confidence intervals for the forecasts are (1 - alpha) %. For
        example, `alpha=0.05` is equivalent to `interval=[0.025, 0.975]`.
        It cannot be used together with `interval`.
    freeze_params : bool, default True
        Determines whether to freeze the model parameters after the first fit
        for estimators that perform automatic model selection (`Arima` with
        `order=None` or `seasonal_order=None`, `Ets` with `model='ZZZ'`).

        - If `True`, the parameters selected in the first fit (`best_params_`)
        are reused in all subsequent refits, so the automatic selection runs only
        once. Arima keeps the orders, intercept, drift and Box-Cox lambda, and
        re-estimates the coefficients in each refit. Ets keeps the model type,
        smoothing parameters and Box-Cox lambda, and re-estimates the initial
        states.
        - If `False`, automatic model selection is performed independently in each
        refit, allowing parameters to adapt across folds. This increases runtime
        and adds an `estimator_params` column to the output with the model
        selected in each fold.
    n_jobs : int, 'auto', default 'auto'
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.model_selection._utils.select_n_jobs_backtesting.
    verbose : bool, default False
        Print number of folds and index of training and validation sets used 
        for backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings: bool, default False
        If `True`, skforecast warnings will be suppressed during the backtesting 
        process. See skforecast.exceptions.warn_skforecast_categories for more
        information.

    Returns
    -------
    metric_values : pandas DataFrame
        Value(s) of the metric(s), one row per estimator, named in the first
        column, `estimator_id`, if the forecaster has several. If `interval`
        (or `alpha`) is not `None`, two more columns are added after the
        metrics:

        - coverage: share of true values inside the interval (bounds included).
        - interval_width: mean of `upper_bound - lower_bound`, in the units of
        the series.

        The last column, `n_evaluated`, is the number of points used to
        compute the metrics: true value and prediction not missing, each date
        counted once (with the prediction of its last fold when the folds
        overlap).

        **New in version 0.27.0**
    backtest_predictions : pandas DataFrame
        Value of predictions. The DataFrame includes the following columns, in
        this order:

        - estimator_id: Identifier of the estimator, only if the forecaster has
        several (one row per estimator and date).
        - fold: Indicates the fold number where the prediction was made.
        - step: Step of the forecast horizon at which the prediction was made,
        from `gap + 1` to `gap + steps`.
        - y_true: True value of the series (NaN if it is missing).
        - pred: Predicted values for the corresponding series and time steps.

        **New in version 0.27.0**: columns `step` and `y_true`.

        If `interval` is not `None`, additional columns are included:

        - lower_bound: lower bound of the interval.
        - upper_bound: upper bound of the interval.

        If `freeze_params` is `False`, an additional column is included:

        - estimator_params: parameters used in the estimator for each fold.

        Depending on the relation between `steps` and `fold_stride`, the output
        may include repeated indexes (if `fold_stride < steps`) or gaps
        (if `fold_stride > steps`). See Notes below for more details.

    Notes
    -----
    Note on `fold_stride` vs. `steps`:

    - If `fold_stride == steps`, test sets are placed back-to-back without overlap. 
    Each observation appears only once in the output DataFrame, so the index is unique.
    - If `fold_stride < steps`, test sets overlap. Multiple forecasts are generated 
    for the same observations and, therefore, the output DataFrame contains repeated 
    indexes.
    - If `fold_stride > steps`, there are gaps between consecutive test sets. 
    Some observations in the series will not have associated predictions, so 
    the output DataFrame has non-contiguous indexes.

    """

    if type(forecaster).__name__  != 'ForecasterStats':
        raise TypeError(
            "`forecaster` must be of type `ForecasterStats`. For all other "
            "types of forecasters use the other functions available in the "
            "`model_selection` module."
        )

    if alpha is not None and interval is not None:
        raise ValueError(
            "`alpha` and `interval` cannot be used together. Use only one "
            "of them: `alpha=0.05` is equivalent to `interval=[0.025, 0.975]`."
        )

    check_backtesting_input(
        forecaster        = forecaster,
        cv                = cv,
        y                 = y,
        metric            = metric,
        interval          = interval,
        alpha             = alpha,
        freeze_params     = freeze_params,
        n_jobs            = n_jobs,
        show_progress     = show_progress,
        suppress_warnings = suppress_warnings
    )

    # NOTE: `alpha` is an alias of `interval`. It is translated into the
    # equivalent `interval` so that the rest of the process only uses `interval`.
    if alpha is not None:
        interval = [alpha / 2, 1 - alpha / 2]

    metric_values, backtest_predictions = _backtesting_stats(
        forecaster        = forecaster,
        y                 = y,
        cv                = cv,
        metric            = metric,
        exog              = exog,
        interval          = interval,
        freeze_params     = freeze_params,
        n_jobs            = n_jobs,
        verbose           = verbose,
        show_progress     = show_progress,
        suppress_warnings = suppress_warnings
    )

    return metric_values, backtest_predictions

skforecast.model_selection._search.grid_search_stats

grid_search_stats(
    forecaster,
    y,
    cv,
    param_grid,
    metric,
    exog=None,
    return_best=True,
    n_jobs="auto",
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
    output_file=None,
)

Exhaustive search over specified parameter values for a ForecasterStats object. Validation is done using time series backtesting.

Parameters:

Name	Type	Description	Default
forecaster	ForecasterStats	Forecaster model.	required
y	pandas Series	Training time series.	required
cv	TimeSeriesFold	TimeSeriesFold object with the information needed to split the data into folds. Its window_size is taken from the forecaster, and the object passed is not modified. With estimators other than Sarimax, refit is set to True.	required
param_grid	dict	Dictionary with parameters names (str) as keys and lists of parameter settings to try as values.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: name of the metric, for example 'mean_absolute_error' or 'root_mean_squared_error'. All accepted names are listed in https://skforecast.org/latest/user_guides/metrics.html. If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. y_train is the series before the first forecast origin. mean_absolute_scaled_error and root_mean_squared_scaled_error scale the error of each prediction with the one-step naive forecast over the observations before its own forecast origin, the same for every forecaster. If list: List containing multiple strings and/or Callables. The first metric ranks the candidates. Scikit-learn metrics ending in _score are maximized and those ending in _error, _loss or _deviance are minimized. Other functions are minimized in regression and maximized in classification, unless their attribute greater_is_better is set (True or False).	required
exog	pandas Series, pandas DataFrame	Exogenous variable/s included as predictor/s. Must have the same number of observations as y and should be aligned so that y[i] is regressed on exog[i].	None
return_best	bool	Refit the forecaster using the best found parameters on the whole data.	True
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.model_selection._utils.select_n_jobs_backtesting. Ignored for OneStepAheadFold.	'auto'
verbose	bool	Print number of folds used for cv or backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information. Otherwise, the warnings of the candidates are shown once, at the end of the search, with the number of candidates in which they are raised. The candidates skipped because they raise an error are always reported, in a single warning.	False
output_file	str	Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file.	None

Returns:

Name	Type	Description
results	pandas DataFrame	Results for each combination of parameters. column params: parameters configuration for each iteration. column metric: metric value estimated for each iteration. column n_evaluated: number of points that enter the metrics of each iteration (true value and prediction not missing). additional n columns with param = value.

Source code in skforecast/model_selection/_search.py

def grid_search_stats(
    forecaster: object,
    y: pd.Series,
    cv: TimeSeriesFold,
    param_grid: dict,
    metric: str | Callable | list[str | Callable],
    exog: pd.Series | pd.DataFrame | None = None,
    return_best: bool = True,
    n_jobs: int | str = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: str | None = None
) -> pd.DataFrame:
    """
    Exhaustive search over specified parameter values for a ForecasterStats object.
    Validation is done using time series backtesting.

    Parameters
    ----------
    forecaster : ForecasterStats
        Forecaster model.
    y : pandas Series
        Training time series. 
    cv : TimeSeriesFold
        TimeSeriesFold object with the information needed to split the data into
        folds. Its `window_size` is taken from the forecaster, and the object
        passed is not modified. With estimators other than `Sarimax`,
        `refit` is set to `True`.
    param_grid : dict
        Dictionary with parameters names (`str`) as keys and lists of parameter
        settings to try as values.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: name of the metric, for example `'mean_absolute_error'`
        or `'root_mean_squared_error'`. All accepted names are listed in
        https://skforecast.org/latest/user_guides/metrics.html.
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float. `y_train` is the series before the
        first forecast origin. `mean_absolute_scaled_error` and
        `root_mean_squared_scaled_error` scale the error of each prediction
        with the one-step naive forecast over the observations before its own
        forecast origin, the same for every forecaster.
        - If `list`: List containing multiple strings and/or Callables.

        The first metric ranks the candidates. Scikit-learn metrics ending in
        `_score` are maximized and those ending in `_error`, `_loss` or
        `_deviance` are minimized. Other functions are minimized in regression
        and maximized in classification, unless their attribute
        `greater_is_better` is set (`True` or `False`).
    exog : pandas Series, pandas DataFrame, default None
        Exogenous variable/s included as predictor/s. Must have the same
        number of observations as `y` and should be aligned so that y[i] is
        regressed on exog[i].
    return_best : bool, default True
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default 'auto'
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.model_selection._utils.select_n_jobs_backtesting. Ignored for
        `OneStepAheadFold`.
    verbose : bool, default False
        Print number of folds used for cv or backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings : bool, default False
        If `True`, skforecast warnings will be suppressed during the hyperparameter 
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.
        Otherwise, the warnings of the candidates are shown once, at the end
        of the search, with the number of candidates in which they are
        raised. The candidates skipped because they raise an error are always
        reported, in a single warning.
    output_file : str, default None
        Specifies the filename or full path where the results should be saved. 
        The results will be saved in a tab-separated values (TSV) format. If 
        `None`, the results will not be saved to a file.

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration.
        - column n_evaluated: number of points that enter the metrics of each
        iteration (true value and prediction not missing).
        - additional n columns with param = value.

    """

    param_grid = list(ParameterGrid(param_grid))

    results = _evaluate_grid_hyperparameters_stats(
        forecaster        = forecaster,
        y                 = y,
        cv                = cv,
        param_grid        = param_grid,
        metric            = metric,
        exog              = exog,
        return_best       = return_best,
        n_jobs            = n_jobs,
        verbose           = verbose,
        suppress_warnings = suppress_warnings,
        show_progress     = show_progress,
        output_file       = output_file
    )

    return results

skforecast.model_selection._search.random_search_stats

random_search_stats(
    forecaster,
    y,
    cv,
    param_distributions,
    metric,
    exog=None,
    n_iter=10,
    random_state=123,
    return_best=True,
    n_jobs="auto",
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
    output_file=None,
)

Random search over specified parameter values or distributions for a ForecasterStats object. Validation is done using time series backtesting.

Parameters:

Name	Type	Description	Default
forecaster	ForecasterStats	Forecaster model.	required
y	pandas Series	Training time series.	required
cv	TimeSeriesFold	TimeSeriesFold object with the information needed to split the data into folds. Its window_size is taken from the forecaster, and the object passed is not modified. With estimators other than Sarimax, refit is set to True.	required
param_distributions	dict	Dictionary with parameters names (str) as keys and distributions or lists of parameters to try.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: name of the metric, for example 'mean_absolute_error' or 'root_mean_squared_error'. All accepted names are listed in https://skforecast.org/latest/user_guides/metrics.html. If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. y_train is the series before the first forecast origin. mean_absolute_scaled_error and root_mean_squared_scaled_error scale the error of each prediction with the one-step naive forecast over the observations before its own forecast origin, the same for every forecaster. If list: List containing multiple strings and/or Callables. The first metric ranks the candidates. Scikit-learn metrics ending in _score are maximized and those ending in _error, _loss or _deviance are minimized. Other functions are minimized in regression and maximized in classification, unless their attribute greater_is_better is set (True or False).	required
exog	pandas Series, pandas DataFrame	Exogenous variable/s included as predictor/s. Must have the same number of observations as y and should be aligned so that y[i] is regressed on exog[i].	None
n_iter	int	Number of parameter settings that are sampled. n_iter trades off runtime vs quality of the solution.	10
random_state	int	Sets a seed to the random sampling for reproducible output.	123
return_best	bool	Refit the forecaster using the best found parameters on the whole data.	True
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.model_selection._utils.select_n_jobs_backtesting. Ignored for OneStepAheadFold.	'auto'
verbose	bool	Print number of folds used for cv or backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information. Otherwise, the warnings of the candidates are shown once, at the end of the search, with the number of candidates in which they are raised. The candidates skipped because they raise an error are always reported, in a single warning.	False
output_file	str	Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file.	None

Returns:

Name	Type	Description
results	pandas DataFrame	Results for each combination of parameters. column params: parameters configuration for each iteration. column metric: metric value estimated for each iteration. column n_evaluated: number of points that enter the metrics of each iteration (true value and prediction not missing). additional n columns with param = value.

Source code in skforecast/model_selection/_search.py

def random_search_stats(
    forecaster: object,
    y: pd.Series,
    cv: TimeSeriesFold,
    param_distributions: dict,
    metric: str | Callable | list[str | Callable],
    exog: pd.Series | pd.DataFrame | None = None,
    n_iter: int = 10,
    random_state: int = 123,
    return_best: bool = True,
    n_jobs: int | str = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: str | None = None
) -> pd.DataFrame:
    """
    Random search over specified parameter values or distributions for a ForecasterStats 
    object. Validation is done using time series backtesting.

    Parameters
    ----------
    forecaster : ForecasterStats
        Forecaster model.
    y : pandas Series
        Training time series. 
    cv : TimeSeriesFold
        TimeSeriesFold object with the information needed to split the data into
        folds. Its `window_size` is taken from the forecaster, and the object
        passed is not modified. With estimators other than `Sarimax`,
        `refit` is set to `True`.
    param_distributions : dict
        Dictionary with parameters names (`str`) as keys and 
        distributions or lists of parameters to try.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: name of the metric, for example `'mean_absolute_error'`
        or `'root_mean_squared_error'`. All accepted names are listed in
        https://skforecast.org/latest/user_guides/metrics.html.
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float. `y_train` is the series before the
        first forecast origin. `mean_absolute_scaled_error` and
        `root_mean_squared_scaled_error` scale the error of each prediction
        with the one-step naive forecast over the observations before its own
        forecast origin, the same for every forecaster.
        - If `list`: List containing multiple strings and/or Callables.

        The first metric ranks the candidates. Scikit-learn metrics ending in
        `_score` are maximized and those ending in `_error`, `_loss` or
        `_deviance` are minimized. Other functions are minimized in regression
        and maximized in classification, unless their attribute
        `greater_is_better` is set (`True` or `False`).
    exog : pandas Series, pandas DataFrame, default None
        Exogenous variable/s included as predictor/s. Must have the same
        number of observations as `y` and should be aligned so that y[i] is
        regressed on exog[i].
    n_iter : int, default 10
        Number of parameter settings that are sampled. 
        n_iter trades off runtime vs quality of the solution.
    random_state : int, default 123
        Sets a seed to the random sampling for reproducible output.
    return_best : bool, default True
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default 'auto'
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.model_selection._utils.select_n_jobs_backtesting. Ignored for
        `OneStepAheadFold`.
    verbose : bool, default False
        Print number of folds used for cv or backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings : bool, default False
        If `True`, skforecast warnings will be suppressed during the hyperparameter 
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.
        Otherwise, the warnings of the candidates are shown once, at the end
        of the search, with the number of candidates in which they are
        raised. The candidates skipped because they raise an error are always
        reported, in a single warning.
    output_file : str, default None
        Specifies the filename or full path where the results should be saved. 
        The results will be saved in a tab-separated values (TSV) format. If 
        `None`, the results will not be saved to a file.

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration.
        - column n_evaluated: number of points that enter the metrics of each
        iteration (true value and prediction not missing).
        - additional n columns with param = value.

    """

    param_grid = _sample_param_grid(
                     param_distributions = param_distributions,
                     n_iter              = n_iter,
                     random_state        = random_state,
                     grid_search_name    = 'grid_search_stats',
                     lags_grid           = False
                 )

    results = _evaluate_grid_hyperparameters_stats(
        forecaster        = forecaster,
        y                 = y,
        cv                = cv,
        param_grid        = param_grid,
        metric            = metric,
        exog              = exog,
        return_best       = return_best,
        n_jobs            = n_jobs,
        verbose           = verbose,
        suppress_warnings = suppress_warnings,
        show_progress     = show_progress,
        output_file       = output_file
    )

    return results

skforecast.model_selection._search.grid_search_equivalent_date

grid_search_equivalent_date(
    forecaster,
    y,
    cv,
    param_grid,
    metric,
    return_best=True,
    n_jobs="auto",
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
    output_file=None,
)

Search over specified configurations for a ForecasterEquivalentDate object. Validation is done using time series backtesting.

The parameters offset and n_offsets are coupled: together they define the window used to build the equivalent-date prediction. A third parameter, agg_func, controls how the n_offsets values are aggregated into the prediction (when n_offsets > 1) and can be searched as well. For this reason, param_grid accepts two forms:

• dict: parameter names as keys and lists of values to try. All combinations (Cartesian product) are evaluated. Useful when the combinations are meaningful, for example sweeping agg_func over a set of offset values.
• list of dict: each dictionary defines one explicit configuration (one run) and no Cartesian product is computed. Values must be scalar. To evaluate several values of a parameter, add one dictionary per configuration, for example [{'offset': 7, 'n_offsets': 2, 'agg_func': np.mean}, {'offset': 7, 'n_offsets': 2, 'agg_func': np.median}]. Each dictionary may include an optional alias key (str) to label the configuration in the results; the alias is not passed to the forecaster.

Parameters:

Name	Type	Description	Default
forecaster	ForecasterEquivalentDate	Forecaster model.	required
y	pandas Series	Training time series.	required
cv	TimeSeriesFold	TimeSeriesFold object with the information needed to split the data into folds. Its window_size is taken from the forecaster, and the object passed is not modified.	required
param_grid	(dict, list)	Configurations to try. Valid parameter names are offset, n_offsets and agg_func. If dict: parameter names (str) as keys and lists of settings to try as values. All combinations are evaluated (Cartesian product). For example, {'offset': [1, 7], 'n_offsets': [1, 2], 'agg_func': [np.mean, np.median]}. If list: each element is a dictionary defining a single configuration with scalar values (one run per dictionary). Each dictionary may include an optional alias key (str) to label the configuration in the results. For example, [{'alias': '7-day moving average', 'offset': 1, 'n_offsets': 7, 'agg_func': np.mean}, {'alias': 'mean of lag-7 and lag-14', 'offset': 7, 'n_offsets': 2, 'agg_func': np.mean}].	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: name of the metric, for example 'mean_absolute_error' or 'root_mean_squared_error'. All accepted names are listed in https://skforecast.org/latest/user_guides/metrics.html. If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. y_train is the series before the first forecast origin. mean_absolute_scaled_error and root_mean_squared_scaled_error scale the error of each prediction with the one-step naive forecast over the observations before its own forecast origin, the same for every forecaster. If list: List containing multiple strings and/or Callables. The first metric ranks the candidates. Scikit-learn metrics ending in _score are maximized and those ending in _error, _loss or _deviance are minimized. Other functions are minimized in regression and maximized in classification, unless their attribute greater_is_better is set (True or False).	required
return_best	bool	Refit the forecaster using the best found configuration on the whole data.	True
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.model_selection._utils.select_n_jobs_backtesting.	'auto'
verbose	bool	Print number of folds used for cv or backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the search. See skforecast.exceptions.warn_skforecast_categories for more information. Otherwise, the warnings of the candidates are shown once, at the end of the search, with the number of candidates in which they are raised. The candidates skipped because they raise an error are always reported, in a single warning.	False
output_file	str	Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file.	None

Returns:

Name	Type	Description
results	pandas DataFrame	Results for each configuration. column alias: configuration label, only present as the first column if param_grid is a list and at least one configuration defines an alias. column params: parameters configuration for each iteration. column metric: metric value estimated for each iteration. column n_evaluated: number of points that enter the metrics of each iteration (true value and prediction not missing). additional n columns with param = value.

Source code in skforecast/model_selection/_search.py

@manage_warnings
def grid_search_equivalent_date(
    forecaster: object,
    y: pd.Series,
    cv: TimeSeriesFold,
    param_grid: dict | list[dict],
    metric: str | Callable | list[str | Callable],
    return_best: bool = True,
    n_jobs: int | str = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: str | None = None
) -> pd.DataFrame:
    """
    Search over specified configurations for a ForecasterEquivalentDate object.
    Validation is done using time series backtesting.

    The parameters `offset` and `n_offsets` are coupled: together they define the
    window used to build the equivalent-date prediction. A third parameter,
    `agg_func`, controls how the `n_offsets` values are aggregated into the
    prediction (when `n_offsets > 1`) and can be searched as well. For this
    reason, `param_grid` accepts two forms:

    - `dict`: parameter names as keys and lists of values to try. All
    combinations (Cartesian product) are evaluated. Useful when the combinations
    are meaningful, for example sweeping `agg_func` over a set of `offset` values.
    - `list` of `dict`: each dictionary defines one explicit configuration (one
    run) and no Cartesian product is computed. Values must be scalar. To evaluate
    several values of a parameter, add one dictionary per configuration, for
    example `[{'offset': 7, 'n_offsets': 2, 'agg_func': np.mean},
    {'offset': 7, 'n_offsets': 2, 'agg_func': np.median}]`. Each dictionary may
    include an optional `alias` key (`str`) to label the configuration in the
    results; the alias is not passed to the forecaster.

    Parameters
    ----------
    forecaster : ForecasterEquivalentDate
        Forecaster model.
    y : pandas Series
        Training time series.
    cv : TimeSeriesFold
        TimeSeriesFold object with the information needed to split the data into
        folds. Its `window_size` is taken from the forecaster, and the object
        passed is not modified.
    param_grid : dict, list
        Configurations to try. Valid parameter names are `offset`, `n_offsets`
        and `agg_func`.

        - If `dict`: parameter names (`str`) as keys and lists of settings to try
        as values. All combinations are evaluated (Cartesian product).
        For example, `{'offset': [1, 7], 'n_offsets': [1, 2], 'agg_func': [np.mean, np.median]}`.
        - If `list`: each element is a dictionary defining a single configuration
        with scalar values (one run per dictionary). Each dictionary may include 
        an optional `alias` key (`str`) to label the configuration in the results. 
        For example,
        `[{'alias': '7-day moving average', 'offset': 1, 'n_offsets': 7, 'agg_func': np.mean}, 
        {'alias': 'mean of lag-7 and lag-14', 'offset': 7, 'n_offsets': 2, 'agg_func': np.mean}]`.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: name of the metric, for example `'mean_absolute_error'`
        or `'root_mean_squared_error'`. All accepted names are listed in
        https://skforecast.org/latest/user_guides/metrics.html.
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float. `y_train` is the series before the
        first forecast origin. `mean_absolute_scaled_error` and
        `root_mean_squared_scaled_error` scale the error of each prediction
        with the one-step naive forecast over the observations before its own
        forecast origin, the same for every forecaster.
        - If `list`: List containing multiple strings and/or Callables.

        The first metric ranks the candidates. Scikit-learn metrics ending in
        `_score` are maximized and those ending in `_error`, `_loss` or
        `_deviance` are minimized. Other functions are minimized in regression
        and maximized in classification, unless their attribute
        `greater_is_better` is set (`True` or `False`).
    return_best : bool, default True
        Refit the `forecaster` using the best found configuration on the whole data.
    n_jobs : int, 'auto', default 'auto'
        The number of jobs to run in parallel. If `-1`, then the number of jobs is
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.model_selection._utils.select_n_jobs_backtesting.
    verbose : bool, default False
        Print number of folds used for cv or backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings : bool, default False
        If `True`, skforecast warnings will be suppressed during the search.
        See skforecast.exceptions.warn_skforecast_categories for more information.
        Otherwise, the warnings of the candidates are shown once, at the end
        of the search, with the number of candidates in which they are
        raised. The candidates skipped because they raise an error are always
        reported, in a single warning.
    output_file : str, default None
        Specifies the filename or full path where the results should be saved.
        The results will be saved in a tab-separated values (TSV) format. If
        `None`, the results will not be saved to a file.

    Returns
    -------
    results : pandas DataFrame
        Results for each configuration.

        - column alias: configuration label, only present as the first column if
        `param_grid` is a list and at least one configuration defines an `alias`.
        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration.
        - column n_evaluated: number of points that enter the metrics of each
        iteration (true value and prediction not missing).
        - additional n columns with param = value.

    """

    if type(forecaster).__name__ != 'ForecasterEquivalentDate':
        raise TypeError(
            "`forecaster` must be of type `ForecasterEquivalentDate`, for all "
            "other types of forecasters use the functions available in the "
            "`model_selection` module."
        )

    if type(cv).__name__ != 'TimeSeriesFold':
        raise TypeError(
            f"`cv` must be an instance of `TimeSeriesFold`. Got {type(cv).__name__}."
        )

    if isinstance(param_grid, dict):
        if 'alias' in param_grid:
            raise ValueError(
                "`alias` is only supported when `param_grid` is a list of "
                "configurations, not when `param_grid` is a dict."
            )
        param_grid = list(ParameterGrid(param_grid))
        alias_grid = [None] * len(param_grid)
    elif isinstance(param_grid, list):
        param_grid_clean = []
        alias_grid = []
        for config in param_grid:
            if not isinstance(config, dict):
                raise TypeError(
                    f"When `param_grid` is a list, each element must be a dict "
                    f"of parameters. Got {type(config).__name__}."
                )
            config = config.copy()
            alias = config.pop('alias', None)
            list_valued_params = [
                k for k, v in config.items()
                if isinstance(v, (list, np.ndarray))
            ]
            if list_valued_params:
                raise ValueError(
                    f"When `param_grid` is a list, each dictionary must define a "
                    f"single configuration with scalar values. Parameters "
                    f"{list_valued_params} have list values. To evaluate multiple "
                    f"values, either use the dict form of `param_grid` (Cartesian "
                    f"product) or add one dictionary per configuration."
                )
            param_grid_clean.append(config)
            alias_grid.append(alias)
        param_grid = param_grid_clean
    else:
        raise TypeError(
            f"`param_grid` must be a dict or a list of dicts. "
            f"Got {type(param_grid).__name__}."
        )

    use_alias = any(alias is not None for alias in alias_grid)

    forecaster_search = deepcopy_forecaster(forecaster)

    if not isinstance(metric, list):
        metric = [metric]
    metric = [
        _get_metric(metric=m)
        if isinstance(m, str)
        else add_y_train_argument(m)
        for m in metric
    ]
    metric_names = [(m if isinstance(m, str) else m.__name__) for m in metric]
    _check_metric_names_search(metric_names=metric_names)
    metric_dict = {metric_name: [] for metric_name in metric_names}

    if verbose:
        print(f"Number of models compared: {len(param_grid)}.")

    param_alias_grid = list(zip(param_grid, alias_grid))
    if show_progress:
        param_alias_grid = tqdm(param_alias_grid, desc='params grid', position=0)

    if output_file is not None and os.path.isfile(output_file):
        os.remove(output_file)

    # NOTE: The warnings of the candidates and of the refit are collected and
    # shown once at the end of the search, see `_SearchWarnings`.
    search_warnings = _SearchWarnings(n_candidates=len(param_grid))
    with search_warnings:

        candidate = 0
        params_list = []
        alias_list = []
        n_evaluated_list = []
        for params, alias in param_alias_grid:

            candidate += 1
            search_warnings.set_candidate(candidate)
            try:
                forecaster_search.set_params(params)
                metric_values = backtesting_forecaster(
                                    forecaster        = forecaster_search,
                                    y                 = y,
                                    cv                = cv,
                                    metric            = metric,
                                    exog              = None,
                                    interval          = None,
                                    n_jobs            = n_jobs,
                                    verbose           = verbose,
                                    show_progress     = False,
                                    suppress_warnings = suppress_warnings
                                )[0]
            except Exception as e:
                search_warnings.add_skipped(lags=None, params=params, error=e)
                continue

            n_evaluated = int(metric_values.loc[0, N_EVALUATED_NAME])
            metric_values = metric_values.loc[0, list(metric_dict)].to_list()

            params_list.append(params)
            alias_list.append(alias)
            n_evaluated_list.append(n_evaluated)
            for m, m_value in zip(metric, metric_values):
                m_name = m if isinstance(m, str) else m.__name__
                metric_dict[m_name].append(m_value)

            if output_file is not None:
                header = ['alias'] if use_alias else []
                header += [
                    'params', *metric_dict.keys(), N_EVALUATED_NAME, *params.keys()
                ]
                row = [alias] if use_alias else []
                row += [
                    params,
                    *metric_values,
                    n_evaluated,
                    *[v.__name__ if callable(v) else v for v in params.values()]
                ]
                if not os.path.isfile(output_file):
                    with open(output_file, 'w', newline='') as f:
                        f.write('\t'.join(header) + '\n')
                        f.write('\t'.join([str(r) for r in row]) + '\n')
                else:
                    with open(output_file, 'a', newline='') as f:
                        f.write('\t'.join([str(r) for r in row]) + '\n')

    results = pd.DataFrame({
                  'params': params_list,
                  **metric_dict,
                  N_EVALUATED_NAME: n_evaluated_list
              })
    if use_alias:
        results.insert(0, 'alias', alias_list)

    direction = _get_search_direction(metric=metric[0], forecaster=forecaster_search)
    results = (
        results
        .sort_values(by=list(metric_dict.keys())[0], ascending=direction == 'minimize')
        .reset_index(drop=True)
    )
    expanded_params = results['params'].apply(pd.Series)
    expanded_params = expanded_params.map(
        lambda v: v.__name__ if callable(v) else v
    )
    results = pd.concat([results, expanded_params], axis=1)

    if results.empty:
        search_warnings.emit()
        return results

    if return_best:

        best_params = results.loc[0, 'params']
        best_metric = results.loc[0, list(metric_dict.keys())[0]]

        # NOTE: Here we use the actual forecaster passed by the user
        forecaster.set_params(best_params)
        search_warnings.set_candidate(None)
        with search_warnings:
            forecaster.fit(
                y                         = y,
                store_in_sample_residuals = True,
                suppress_warnings         = suppress_warnings
            )

        if verbose:
            print(
                f"`Forecaster` refitted using the best-found parameters, "
                f"and the whole data set: \n"
                f"  Parameters: {best_params}\n"
                f"  Backtesting metric: {best_metric}\n"
            )

    search_warnings.emit()

    return results

skforecast.model_selection._split.TimeSeriesFold

TimeSeriesFold(
    steps,
    initial_train_size=None,
    fold_stride=None,
    window_size=None,
    differentiation=None,
    refit=False,
    fixed_train_size=True,
    gap=0,
    skip_folds=None,
    allow_incomplete_fold=True,
    return_all_indexes=False,
    verbose=True,
)

Bases: BaseFold

Generate the train and test folds used for time series cross-validation with a rolling or expanding window scheme. Starting from initial_train_size, the origin advances fold_stride observations in each fold and steps observations are forecasted ahead, optionally leaving a gap between the training data and the forecast horizon.

This is the cross-validation strategy expected by the backtesting functions (backtesting_forecaster, backtesting_forecaster_multiseries, ...) and by the hyperparameter search functions (grid_search_forecaster, random_search_forecaster, bayesian_search_forecaster, ...), where it is passed through the cv argument. The split method can also be called directly to inspect the resulting folds.

The user always provides initial_train_size (it can only be None when reusing an already fitted forecaster). The argument window_size is taken from the forecaster automatically when the object is used inside a backtesting or hyperparameter search, so it does not need to be set manually in that context.

Parameters:

Name	Type	Description	Default
steps	int	Number of observations to predict in each fold, also known as the forecast horizon or test size.	required
initial_train_size	int, str, pandas Timestamp	Number of observations used for initial training. If None, there is no initial training: the forecaster must be already fitted, and window_size is needed to build the first last window. If an integer, the number of observations used for initial training. If a date string or pandas Timestamp, it is the last date included in the initial training set.	None
fold_stride	int	Number of observations that the start of the test set advances between consecutive folds. If None, it defaults to the same value as steps, meaning that folds are placed back-to-back without overlap. If fold_stride < steps, test sets overlap and multiple forecasts will be generated for the same observations. If fold_stride > steps, gaps are left between consecutive test sets. New in version 0.18.0	None
window_size	int, pandas DateOffset	Number of observations needed to generate the autoregressive predictors. It sets the start of the last window of each fold. In the backtesting and hyperparameter search functions, it is taken from the forecaster. To get the same folds when calling split directly, use forecaster.window_size, which already includes the differentiation order of the forecaster. A pandas DateOffset (the window_size of a ForecasterEquivalentDate with a DateOffset as offset) is converted by split into the equivalent number of observations, which replaces the value of this attribute.	None
refit	(bool, int)	Whether to refit the forecaster in each fold. If True, the forecaster is refitted in each fold. If False, the forecaster is trained only in the first fold. If an integer, the forecaster is trained in the first fold and then refitted every refit folds.	False
fixed_train_size	bool	Whether the training set keeps its size and moves forward with each fold (rolling window, True) or grows with each fold (expanding window, False). Only used when the forecaster is refitted (refit is not False).	True
gap	int	Number of observations between the end of the training set and the start of the test set.	0
skip_folds	(int, list)	Folds to leave out of the output. If an integer, only one of every skip_folds folds is kept, starting with the first one. If a list, the indexes of the folds to leave out. The first fold (0) cannot be skipped because it is needed to train the forecaster. For example, if skip_folds=3 and there are 10 folds, the returned folds are 0, 3, 6, and 9. If skip_folds=[1, 2, 3], the returned folds are 0, 4, 5, 6, 7, 8, and 9.	None
allow_incomplete_fold	bool	Whether to allow the last fold to include fewer observations than steps. If False, the last fold is excluded if it is incomplete.	True
return_all_indexes	bool	If True, each partition of a fold is a range with all its positions. If False, it is a list with its start and end positions. In the backtesting and hyperparameter search functions, it is set to False.	False
verbose	bool	Whether to print information about the folds when split is called. In the backtesting and hyperparameter search functions, it is replaced by their own verbose argument.	True
differentiation	int	Deprecated since version 0.27.0, it has no effect and will be removed in version 0.29.0. The window_size of the forecaster already includes its differentiation order.	None

Attributes:

Name	Type	Description
steps	int	Number of observations to predict in each fold, also known as the forecast horizon or test size.
initial_train_size	int, str, pandas Timestamp	Number of observations used for initial training, or a date (string or pandas Timestamp parseable by pandas) marking the last observation of the training set. If None, the forecaster is not trained in the first fold.
initial_train_size_as_int	int	Resolved integer position of initial_train_size, set when split() is called (None when initial_train_size is None).
fold_stride	int	Number of observations that the start of the test set advances between consecutive folds.
overlapping_folds	bool	Whether the folds overlap.
window_size	int	Number of observations needed to generate the autoregressive predictors.
refit	(bool, int)	Whether to refit the forecaster in each fold.
fixed_train_size	bool	Whether the training size is fixed or increases in each fold.
gap	int	Number of observations between the end of the training set and the start of the test set.
skip_folds	(int, list)	Number of folds to skip.
allow_incomplete_fold	bool	Whether to allow the last fold to include fewer observations than steps.
return_all_indexes	bool	Whether to return all indexes or only the start and end indexes of each fold.
verbose	bool	Whether to print information about generated folds.

Notes

Returned values are the positions of the observations and not the actual values of the index, so they can be used to slice the data directly using iloc. For example, if the input series is X = [10, 11, 12, 13, 14, 15, 16, 17, 18, 19], the initial_train_size = 3, window_size = 2, steps = 4, and gap = 1, the output of the first fold will be: [0, [0, 3], [1, 3], [3, 8], [4, 8], True].

The first element is the fold number, the first list [0, 3] indicates that the training set goes from the first to the third observation. The second list [1, 3] indicates that the last window seen by the forecaster during training goes from the second to the third observation. The third list [3, 8] indicates that the forecaster predicts from the fourth to the eighth observation: the gap (fourth observation) and the test set. The fourth list [4, 8] indicates that the test set, after the gap, goes from the fifth to the eighth observation; these are the observations used to evaluate the forecaster. The boolean True indicates that the forecaster will be trained in this fold.

Following the python convention, the start index is inclusive and the end index is exclusive. This means that the last index is not included in the slice.

As an example, with initial_train_size=50, steps=30, and fold_stride=7, the first test fold will cover observations [50, 80), the second fold [57, 87), and the third fold [64, 94). This configuration produces multiple forecasts for the same observations, which is often desirable in rolling-origin evaluation.

Methods:

Name	Description
split	Split the time series data into train and test folds.

Source code in skforecast/model_selection/_split.py

def __init__(
    self,
    steps: int,
    initial_train_size: int | str | pd.Timestamp | None = None,
    fold_stride: int | None = None,
    window_size: int | pd.DateOffset | None = None,
    differentiation: int | None = None,
    refit: bool | int = False,
    fixed_train_size: bool = True,
    gap: int = 0,
    skip_folds: int | list[int] | None = None,
    allow_incomplete_fold: bool = True,
    return_all_indexes: bool = False,
    verbose: bool = True
) -> None:

    super().__init__(
        steps                 = steps,
        initial_train_size    = initial_train_size,
        fold_stride           = fold_stride,
        window_size           = window_size,
        differentiation       = differentiation,
        refit                 = refit,
        fixed_train_size      = fixed_train_size,
        gap                   = gap,
        skip_folds            = skip_folds,
        allow_incomplete_fold = allow_incomplete_fold,
        return_all_indexes    = return_all_indexes,
        verbose               = verbose
    )

    self.steps                 = steps
    self.fold_stride           = fold_stride if fold_stride is not None else steps
    self.overlapping_folds     = self.fold_stride < self.steps
    self.refit                 = refit
    self.fixed_train_size      = fixed_train_size
    self.gap                   = gap
    self.skip_folds            = skip_folds
    self.allow_incomplete_fold = allow_incomplete_fold

Attributes

steps instance-attribute

steps = steps

fold_stride instance-attribute

fold_stride = (
    fold_stride if fold_stride is not None else steps
)

overlapping_folds instance-attribute

overlapping_folds = self.fold_stride < self.steps

refit instance-attribute

refit = refit

fixed_train_size instance-attribute

fixed_train_size = fixed_train_size

gap instance-attribute

gap = gap

skip_folds instance-attribute

skip_folds = skip_folds

allow_incomplete_fold instance-attribute

allow_incomplete_fold = allow_incomplete_fold

Methods:

split

split(X, as_pandas=False)

Split the time series data into train and test folds.

Parameters:

Name	Type	Description	Default
X	pandas Series, pandas DataFrame, pandas Index, dict	Time series data or index to split.	required
as_pandas	bool	If True, the folds are returned as a DataFrame. This is useful to visualize the folds in a more interpretable way.	False

Returns:

Name	Type	Description
folds	list, pandas DataFrame	A list with one element per fold, containing the fold number, 4 lists with positions and a boolean: fold: fold number. [train_start, train_end]: list with the start and end positions of the training set. [last_window_start, last_window_end]: list with the start and end positions of the last window seen by the forecaster during training. The last window is used to generate the lags used as predictors. In the backtesting functions, window_size includes the differentiation order of the forecaster, so the interval also covers the observations needed to differentiate. If the argument window_size is None, this list is empty. [test_start, test_end]: list with the start and end positions of the observations predicted in the fold, from the end of the last window: the gap followed by the test set. [test_start_with_gap, test_end_with_gap]: list with the start and end positions of the test set, after the gap. These are the observations used to evaluate the forecaster. Without gap, they are the same as [test_start, test_end]. fit_forecaster: boolean indicating whether the forecaster should be fitted in this fold. It is important to note that the returned values are the positions of the observations and not the actual values of the index, so they can be used to slice the data directly using iloc. If return_all_indexes is True, the lists are replaced by range objects with all the positions. If as_pandas is True, the folds are returned as a DataFrame with the following columns: 'fold', 'train_start', 'train_end', 'last_window_start', 'last_window_end', 'test_start', 'test_end', 'test_start_with_gap', 'test_end_with_gap', 'fit_forecaster'. With return_all_indexes=True, the columns are 'fold', 'train_index', 'last_window_index', 'test_index', 'test_index_with_gap', 'fit_forecaster'. Following the python convention, the start index is inclusive and the end index is exclusive. This means that the last index is not included in the slice.

Source code in skforecast/model_selection/_split.py

def split(
    self,
    X: pd.Series | pd.DataFrame | pd.Index | dict[str, pd.Series | pd.DataFrame],
    as_pandas: bool = False
) -> list | pd.DataFrame:
    """
    Split the time series data into train and test folds.

    Parameters
    ----------
    X : pandas Series, pandas DataFrame, pandas Index, dict
        Time series data or index to split.
    as_pandas : bool, default False
        If True, the folds are returned as a DataFrame. This is useful to visualize
        the folds in a more interpretable way.

    Returns
    -------
    folds : list, pandas DataFrame
        A list with one element per fold, containing the fold number, 4 lists
        with positions and a boolean:

        - fold: fold number.
        - [train_start, train_end]: list with the start and end positions of the
        training set.
        - [last_window_start, last_window_end]: list with the start and end positions
        of the last window seen by the forecaster during training. The last window
        is used to generate the lags used as predictors. In the backtesting
        functions, `window_size` includes the differentiation order of the
        forecaster, so the interval also covers the observations needed to
        differentiate. If the argument `window_size` is `None`, this list is
        empty.
        - [test_start, test_end]: list with the start and end positions of the
        observations predicted in the fold, from the end of the last window: the
        `gap` followed by the test set.
        - [test_start_with_gap, test_end_with_gap]: list with the start and end
        positions of the test set, after the `gap`. These are the observations
        used to evaluate the forecaster. Without gap, they are the same as
        [test_start, test_end].
        - fit_forecaster: boolean indicating whether the forecaster should be fitted
        in this fold.

        It is important to note that the returned values are the positions of the
        observations and not the actual values of the index, so they can be used to
        slice the data directly using iloc. If `return_all_indexes` is `True`, the
        lists are replaced by `range` objects with all the positions.

        If `as_pandas` is `True`, the folds are returned as a DataFrame with the
        following columns: 'fold', 'train_start', 'train_end', 'last_window_start',
        'last_window_end', 'test_start', 'test_end', 'test_start_with_gap',
        'test_end_with_gap', 'fit_forecaster'. With `return_all_indexes=True`,
        the columns are 'fold', 'train_index', 'last_window_index', 'test_index',
        'test_index_with_gap', 'fit_forecaster'.

        Following the python convention, the start index is inclusive and the end
        index is exclusive. This means that the last index is not included in the
        slice.

    """

    if not isinstance(X, (pd.Series, pd.DataFrame, pd.Index, dict)):
        raise TypeError(
            f"X must be a pandas Series, DataFrame, Index or a dictionary. "
            f"Got {type(X)}."
        )

    window_size_as_date_offset = isinstance(self.window_size, pd.tseries.offsets.DateOffset)
    if window_size_as_date_offset:
        # Calculate the window_size in steps. This is not a exact calculation
        # because the offset follows the calendar rules and the distance between
        # two dates may not be constant.
        first_valid_index = X.index[-1] - self.window_size
        try:
            window_size_idx_start = X.index.get_loc(first_valid_index)
            window_size_idx_end = X.index.get_loc(X.index[-1])
            self.window_size = window_size_idx_end - window_size_idx_start
        except KeyError:
            raise ValueError(
                f"The length of `y` ({len(X)}), must be greater than or equal "
                f"to the window size ({self.window_size}). This is because "
                f"the offset (forecaster.offset) is larger than the available "
                f"data. Try to decrease the size of the offset (forecaster.offset), "
                f"the number of `n_offsets` (forecaster.n_offsets) or increase the "
                f"size of `y`."
            )

    if self.initial_train_size is None:
        if self.window_size is None:
            raise ValueError(
                "To use split method when `initial_train_size` is None, "
                "`window_size` must be an integer greater than 0. "
                "Although no initial training is done and all data is used to "
                "evaluate the model, the first `window_size` observations are "
                "needed to create the initial predictors. Got `window_size` = None."
            )
        if self.refit:
            raise ValueError(
                "`refit` is only allowed when `initial_train_size` is not `None`. "
                "Set `refit` to `False` if you want to use `initial_train_size = None`."
            )
        externally_fitted = True
        initial_train_size = self.window_size
    else:
        if self.window_size is None:
            warnings.warn(
                "Last window cannot be calculated because `window_size` is None.",
                IgnoredArgumentWarning
            )
        externally_fitted = False
        initial_train_size = self.initial_train_size

    index = self._extract_index(X)
    idx = range(len(index))
    folds = []
    i = 0

    initial_train_size_as_int = date_to_index_position(
                                    index        = index,
                                    date_input   = initial_train_size,
                                    method       = 'validation',
                                    date_literal = 'initial_train_size'
                                )

    # NOTE: When `initial_train_size` is None, the forecaster is already
    # fitted and the first fold only needs the last window, so
    # `initial_train_size` is set to `window_size` on purpose.
    if window_size_as_date_offset and self.initial_train_size is not None:
        _check_initial_train_size(
            initial_train_size       = initial_train_size_as_int,
            initial_train_size_input = self.initial_train_size,
            data_length              = len(index),
            data_length_name         = "the length of the series",
            window_size              = self.window_size
        )

    if self.allow_incomplete_fold:
        # At least one observation after the gap to allow incomplete fold
        if len(index) <= initial_train_size_as_int + self.gap:
            raise ValueError(
                f"The time series must have more than `initial_train_size + gap` "
                f"observations to create at least one fold.\n"
                f"    Time series length: {len(index)}\n"
                f"    Required > {initial_train_size_as_int + self.gap}\n"
                f"    initial_train_size: {initial_train_size_as_int}\n"
                f"    gap: {self.gap}\n"
            )
    else:
        # At least one complete fold
        if len(index) < initial_train_size_as_int + self.gap + self.steps:
            raise ValueError(
                f"The time series must have at least `initial_train_size + gap + steps` "
                f"observations to create a minimum of one complete fold "
                f"(allow_incomplete_fold=False).\n"
                f"    Time series length: {len(index)}\n"
                f"    Required >= {initial_train_size_as_int + self.gap + self.steps}\n"
                f"    initial_train_size: {initial_train_size_as_int}\n"
                f"    gap: {self.gap}\n"
                f"    steps: {self.steps}\n"
            )

    while initial_train_size_as_int + (i * self.fold_stride) + self.gap < len(index):

        if self.refit:
            # NOTE: If `fixed_train_size` the train size doesn't increase but
            # moves by `fold_stride` positions in each iteration. If `False`,
            # the train size increases by `fold_stride` in each iteration.
            train_iloc_start = i * (self.fold_stride) if self.fixed_train_size else 0
            train_iloc_end = initial_train_size_as_int + i * (self.fold_stride)
            test_iloc_start = train_iloc_end
        else:
            # NOTE: The train size doesn't increase and doesn't move.
            train_iloc_start = 0
            train_iloc_end = initial_train_size_as_int
            test_iloc_start = initial_train_size_as_int + i * (self.fold_stride)

        if self.window_size is not None:
            # NOTE: When window_size > test_iloc_start (e.g. a large
            # context_length for ForecasterFoundation on a short series),
            # the difference is negative and must be clamped to 0 so the
            # last_window contains all available history up to
            # test_iloc_start rather than exactly window_size
            # observations. A small negative value cannot be passed to the
            # slice directly: Python interprets it as an offset from the
            # end, producing an empty last_window.
            last_window_iloc_start = max(0, test_iloc_start - self.window_size)

        test_iloc_end = test_iloc_start + self.gap + self.steps

        partitions = [
            idx[train_iloc_start : train_iloc_end],
            idx[last_window_iloc_start : test_iloc_start] if self.window_size is not None else [],
            idx[test_iloc_start : test_iloc_end],
            idx[test_iloc_start + self.gap : test_iloc_end]
        ]
        folds.append(partitions)
        i += 1

    # NOTE: Delete all incomplete folds at the end if not allowed
    n_removed_folds = 0
    if not self.allow_incomplete_fold:
        # NOTE: While folds and the last "test_index_with_gap" is incomplete,
        # calculating len of range objects
        while folds and len(folds[-1][3]) < self.steps:
            folds.pop()
            n_removed_folds += 1

    # Replace partitions inside folds with length 0 with `None`
    folds = [
        [partition if len(partition) > 0 else None for partition in fold] 
        for fold in folds
    ]

    # Create a flag to know whether to train the forecaster
    if self.refit == 0:
        self.refit = False

    if isinstance(self.refit, bool):
        fit_forecaster = [self.refit] * len(folds)
        fit_forecaster[0] = True
    else:
        fit_forecaster = [False] * len(folds)
        for i in range(0, len(fit_forecaster), self.refit): 
            fit_forecaster[i] = True

    for i in range(len(folds)): 
        folds[i].insert(0, i)
        folds[i].append(fit_forecaster[i])
        if fit_forecaster[i] is False:
            folds[i][1] = folds[i - 1][1]

    index_to_skip = []
    if self.skip_folds is not None:
        if isinstance(self.skip_folds, (int, np.integer)) and self.skip_folds > 0:
            index_to_keep = np.arange(0, len(folds), self.skip_folds)
            index_to_skip = np.setdiff1d(np.arange(0, len(folds)), index_to_keep, assume_unique=True)
            index_to_skip = [int(x) for x in index_to_skip]  # Required since numpy 2.0
        if isinstance(self.skip_folds, list):
            index_to_skip = [i for i in self.skip_folds if i < len(folds)]

    if self.verbose:
        self._print_info(
            index                     = index,
            folds                     = folds,
            externally_fitted         = externally_fitted,
            n_removed_folds           = n_removed_folds,
            index_to_skip             = index_to_skip,
            initial_train_size_as_int = initial_train_size_as_int
        )

    folds = [fold for i, fold in enumerate(folds) if i not in index_to_skip]
    if not self.return_all_indexes:
        # NOTE: +1 to prevent iloc pandas from deleting the last observation
        folds = [
            [
                fold[0],
                [fold[1][0], fold[1][-1] + 1],
                (
                    [fold[2][0], fold[2][-1] + 1]
                    if self.window_size is not None
                    else []
                ),
                [fold[3][0], fold[3][-1] + 1],
                [fold[4][0], fold[4][-1] + 1],
                fold[5],
            ]
            for fold in folds
        ]

    if externally_fitted:
        self.initial_train_size_as_int = None
        folds[0][5] = False
    else:
        self.initial_train_size_as_int = initial_train_size_as_int

    if as_pandas:
        if self.window_size is None:
            for fold in folds:
                fold[2] = [None, None]

        if not self.return_all_indexes:
            folds = pd.DataFrame(
                data = [[fold[0]] + list(itertools.chain(*fold[1:-1])) + [fold[-1]] for fold in folds],
                columns = [
                    'fold',
                    'train_start',
                    'train_end',
                    'last_window_start',
                    'last_window_end',
                    'test_start',
                    'test_end',
                    'test_start_with_gap',
                    'test_end_with_gap',
                    'fit_forecaster'
                ],
            )
        else:
            folds = pd.DataFrame(
                data = folds,
                columns = [
                    'fold',
                    'train_index',
                    'last_window_index',
                    'test_index',
                    'test_index_with_gap',
                    'fit_forecaster'
                ],
            )

    return folds

skforecast.model_selection._split.OneStepAheadFold

OneStepAheadFold(
    initial_train_size,
    window_size=None,
    differentiation=None,
    return_all_indexes=False,
    verbose=True,
)

Bases: BaseFold

Split time series data into a single train and test fold to evaluate one-step-ahead predictions. The training set contains the first initial_train_size observations and the test set the rest of the series.

It is the faster alternative to TimeSeriesFold in the hyperparameter search functions (grid_search_forecaster, bayesian_search_forecaster, ...): the forecaster is trained once, and each observation of the test set is predicted one step ahead from the observed values. It is not accepted by the backtesting functions. The split method can also be called directly to inspect the fold.

Parameters:

Name	Type	Description	Default
initial_train_size	int, str, pandas Timestamp	Number of observations used for initial training. If an integer, the number of observations used for initial training. If a date string or pandas Timestamp, it is the last date included in the initial training set.	required
return_all_indexes	bool	If True, each partition of the fold is a list with a range with all its positions. If False, it is a list with its start and end positions.	False
verbose	bool	Whether to print information about the fold when split is called. The hyperparameter search functions do not call split, so they print nothing.	True
window_size	int	Deprecated since version 0.27.0, it has no effect and will be removed in version 0.29.0.	None
differentiation	int	Deprecated since version 0.27.0, it has no effect and will be removed in version 0.29.0.	None

Attributes:

Name	Type	Description
initial_train_size	int, str, pandas Timestamp	Number of observations used for initial training, or a date (string or pandas Timestamp parseable by pandas) marking the last observation of the training set.
initial_train_size_as_int	int	Resolved integer position of initial_train_size, set when split() is called.
return_all_indexes	bool	Whether to return all indexes or only the start and end indexes of each fold.
verbose	bool	Whether to print information about generated folds.

Methods:

Name	Description
split	Split the time series data into train and test folds.

Source code in skforecast/model_selection/_split.py

def __init__(
    self,
    initial_train_size: int | str | pd.Timestamp,
    window_size: int | None = None,
    differentiation: int | None = None,
    return_all_indexes: bool = False,
    verbose: bool = True
) -> None:

    super().__init__(
        initial_train_size = initial_train_size,
        window_size        = window_size,
        differentiation    = differentiation,
        return_all_indexes = return_all_indexes,
        verbose            = verbose
    )

Methods:

split

split(X, as_pandas=False)

Split the time series data into train and test folds.

Parameters:

Name	Type	Description	Default
X	pandas Series, DataFrame, Index, or dictionary	Time series data or index to split.	required
as_pandas	bool	If True, the folds are returned as a DataFrame. This is useful to visualize the folds in a more interpretable way.	False

Returns:

Name	Type	Description
fold	list, pandas DataFrame	A list with the fold number, 2 lists with positions and a boolean: fold: fold number. [train_start, train_end]: list with the start and end positions of the training set. [test_start, test_end]: list with the start and end positions of the test set. These are the observations used to evaluate the forecaster. fit_forecaster: boolean indicating whether the forecaster should be fitted in this fold. It is important to note that the returned values are the positions of the observations and not the actual values of the index, so they can be used to slice the data directly using iloc. If return_all_indexes is True, each list contains a range with all the positions instead. If as_pandas is True, the fold is returned as a DataFrame with the following columns: 'fold', 'train_start', 'train_end', 'test_start', 'test_end', 'fit_forecaster'. With return_all_indexes=True, the columns are 'fold', 'train_index', 'test_index', 'fit_forecaster'. Following the python convention, the start index is inclusive and the end index is exclusive. This means that the last index is not included in the slice.

Source code in skforecast/model_selection/_split.py

def split(
    self,
    X: pd.Series | pd.DataFrame | pd.Index | dict[str, pd.Series | pd.DataFrame],
    as_pandas: bool = False
) -> list | pd.DataFrame:
    """
    Split the time series data into train and test folds.

    Parameters
    ----------
    X : pandas Series, DataFrame, Index, or dictionary
        Time series data or index to split.
    as_pandas : bool, default False
        If True, the folds are returned as a DataFrame. This is useful to visualize
        the folds in a more interpretable way.

    Returns
    -------
    fold : list, pandas DataFrame
        A list with the fold number, 2 lists with positions and a boolean:

        - fold: fold number.
        - [train_start, train_end]: list with the start and end positions of the
        training set.
        - [test_start, test_end]: list with the start and end positions of the test
        set. These are the observations used to evaluate the forecaster.
        - fit_forecaster: boolean indicating whether the forecaster should be fitted
        in this fold.

        It is important to note that the returned values are the positions of the
        observations and not the actual values of the index, so they can be used to
        slice the data directly using iloc. If `return_all_indexes` is `True`, each
        list contains a `range` with all the positions instead.

        If `as_pandas` is `True`, the fold is returned as a DataFrame with the
        following columns: 'fold', 'train_start', 'train_end', 'test_start', 
        'test_end', 'fit_forecaster'. With `return_all_indexes=True`, the
        columns are 'fold', 'train_index', 'test_index', 'fit_forecaster'.

        Following the python convention, the start index is inclusive and the end
        index is exclusive. This means that the last index is not included in the
        slice.

    """

    if not isinstance(X, (pd.Series, pd.DataFrame, pd.Index, dict)):
        raise TypeError(
            f"X must be a pandas Series, DataFrame, Index or a dictionary. "
            f"Got {type(X)}."
        )

    index = self._extract_index(X)

    initial_train_size_as_int = date_to_index_position(
                                    index        = index,
                                    date_input   = self.initial_train_size,
                                    method       = 'validation',
                                    date_literal = 'initial_train_size'
                                )
    self.initial_train_size_as_int = initial_train_size_as_int

    fold = [
        0,
        [0, initial_train_size_as_int - 1],
        [initial_train_size_as_int, len(index)],
        True
    ]

    if self.verbose:
        self._print_info(
            index=index, fold=fold, initial_train_size_as_int=initial_train_size_as_int
        )

    # NOTE: +1 to prevent iloc pandas from deleting the last observation
    if self.return_all_indexes:
        fold = [
            fold[0], 
            [range(fold[1][0], fold[1][1] + 1)],
            [range(fold[2][0], fold[2][1])],
            fold[3]
        ]
    else:
        fold = [
            fold[0], 
            [fold[1][0], fold[1][1] + 1],
            [fold[2][0], fold[2][1]],
            fold[3]
        ]

    if as_pandas:
        if not self.return_all_indexes:
            fold = pd.DataFrame(
                data = [[fold[0]] + list(itertools.chain(*fold[1:-1])) + [fold[-1]]],
                columns = [
                    'fold',
                    'train_start',
                    'train_end',
                    'test_start',
                    'test_end',
                    'fit_forecaster'
                ],
            )
        else:
            fold = pd.DataFrame(
                data = [fold],
                columns = [
                    'fold',
                    'train_index',
                    'test_index',
                    'fit_forecaster'
                ],
            )

    return fold

skforecast.model_selection._split.BaseFold

BaseFold(
    steps=None,
    initial_train_size=None,
    fold_stride=None,
    window_size=None,
    differentiation=None,
    refit=False,
    fixed_train_size=True,
    gap=0,
    skip_folds=None,
    allow_incomplete_fold=True,
    return_all_indexes=False,
    verbose=True,
)

Base class for all Fold classes in skforecast. All fold classes should specify all the parameters that can be set at the class level in their __init__.

Parameters:

Name	Type	Description	Default
steps	int	Number of observations to predict in each fold, also known as the forecast horizon or test size.	None
initial_train_size	int, str, pandas Timestamp	Number of observations used for initial training. If an integer, the number of observations used for initial training. If a date string or pandas Timestamp, it is the last date included in the initial training set.	None
fold_stride	int	Number of observations that the start of the test set advances between consecutive folds. If None, it defaults to the same value as steps, meaning that folds are placed back-to-back without overlap. If fold_stride < steps, test sets overlap and multiple forecasts will be generated for the same observations. If fold_stride > steps, gaps are left between consecutive test sets.	None
window_size	int, pandas DateOffset	Number of observations needed to generate the autoregressive predictors. It sets the start of the last window of each fold. In the backtesting and hyperparameter search functions, it is taken from the forecaster. To get the same folds when calling split directly, use forecaster.window_size, which already includes the differentiation order of the forecaster. A pandas DateOffset (the window_size of a ForecasterEquivalentDate with a DateOffset as offset) is converted by split into the equivalent number of observations, which replaces the value of this attribute.	None
refit	(bool, int)	Whether to refit the forecaster in each fold. If True, the forecaster is refitted in each fold. If False, the forecaster is trained only in the first fold. If an integer, the forecaster is trained in the first fold and then refitted every refit folds.	False
fixed_train_size	bool	Whether the training set keeps its size and moves forward with each fold (rolling window, True) or grows with each fold (expanding window, False). Only used when the forecaster is refitted (refit is not False).	True
gap	int	Number of observations between the end of the training set and the start of the test set.	0
skip_folds	(int, list)	Folds to leave out of the output. If an integer, only one of every skip_folds folds is kept, starting with the first one. If a list, the indexes of the folds to leave out. The first fold (0) cannot be skipped because it is needed to train the forecaster. For example, if skip_folds=3 and there are 10 folds, the returned folds are 0, 3, 6, and 9. If skip_folds=[1, 2, 3], the returned folds are 0, 4, 5, 6, 7, 8, and 9.	None
allow_incomplete_fold	bool	Whether to allow the last fold to include fewer observations than steps. If False, the last fold is excluded if it is incomplete.	True
return_all_indexes	bool	If True, each partition of a fold is a range with all its positions. If False, it is a list with its start and end positions. In the backtesting and hyperparameter search functions, it is set to False.	False
verbose	bool	Whether to print information about the folds when split is called. In the backtesting and hyperparameter search functions, it is replaced by their own verbose argument.	True
differentiation	int	Deprecated since version 0.27.0, it has no effect and will be removed in version 0.29.0. The window_size of the forecaster already includes its differentiation order.	None

Attributes:

Name	Type	Description
initial_train_size	int, str, pandas Timestamp	Number of observations used for initial training, or a date (string or pandas Timestamp parseable by pandas) marking the last observation of the training set.
initial_train_size_as_int	int	Resolved integer position of initial_train_size, set when split() is called (None when initial_train_size is None).
window_size	int	Number of observations needed to generate the autoregressive predictors.
return_all_indexes	bool	Whether to return all indexes or only the start and end indexes of each fold.
verbose	bool	Whether to print information about generated folds.

Methods:

Name	Description
set_params	Set the parameters of the Fold object. Before overwriting the current

Source code in skforecast/model_selection/_split.py

def __init__(
    self,
    steps: int | None = None,
    initial_train_size: int | str | pd.Timestamp | None = None,
    fold_stride: int | None = None,
    window_size: int | pd.DateOffset | None = None,
    differentiation: int | None = None,
    refit: bool | int = False,
    fixed_train_size: bool = True,
    gap: int = 0,
    skip_folds: int | list[int] | None = None,
    allow_incomplete_fold: bool = True,
    return_all_indexes: bool = False,
    verbose: bool = True
) -> None:

    self._warn_deprecated_params(
        params     = {'window_size': window_size, 'differentiation': differentiation},
        stacklevel = 4
    )
    self._validate_params(
        cv_name               = type(self).__name__,
        steps                 = steps,
        initial_train_size    = initial_train_size,
        fold_stride           = fold_stride,
        window_size           = window_size,
        differentiation       = differentiation,
        refit                 = refit,
        fixed_train_size      = fixed_train_size,
        gap                   = gap,
        skip_folds            = skip_folds,
        allow_incomplete_fold = allow_incomplete_fold,
        return_all_indexes    = return_all_indexes,
        verbose               = verbose
    )

    self.initial_train_size        = initial_train_size
    self.initial_train_size_as_int = None
    self.window_size               = window_size
    self.differentiation           = differentiation
    self.return_all_indexes        = return_all_indexes
    self.verbose                   = verbose

Attributes

initial_train_size instance-attribute

initial_train_size = initial_train_size

initial_train_size_as_int instance-attribute

initial_train_size_as_int = None

window_size instance-attribute

window_size = window_size

differentiation instance-attribute

differentiation = differentiation

return_all_indexes instance-attribute

return_all_indexes = return_all_indexes

verbose instance-attribute

verbose = verbose

Methods:

set_params

set_params(params)

Set the parameters of the Fold object. Before overwriting the current parameters, the input parameters are validated to ensure correctness.

Parameters:

Name	Type	Description	Default
params	dict	Dictionary with the parameters to set.	required

Returns:

Type	Description
None

Source code in skforecast/model_selection/_split.py

def set_params(
    self, 
    params: dict
) -> None:
    """
    Set the parameters of the Fold object. Before overwriting the current 
    parameters, the input parameters are validated to ensure correctness.

    Parameters
    ----------
    params : dict
        Dictionary with the parameters to set.

    Returns
    -------
    None

    """

    if not isinstance(params, dict):
        raise TypeError(
            f"`params` must be a dictionary. Got {type(params)}."
        )

    self._warn_deprecated_params(params=params, stacklevel=3)

    current_params = dict(vars(self))
    unknown_params = set(params.keys()) - set(current_params.keys())
    if unknown_params:
        warnings.warn(
            f"Unknown parameters: {unknown_params}. They have been ignored.",
            IgnoredArgumentWarning
        )

    filtered_params = {k: v for k, v in params.items() if k in current_params}
    updated_params = {'cv_name': type(self).__name__, **current_params, **filtered_params}

    self._validate_params(**updated_params)
    for key, value in updated_params.items():
        setattr(self, key, value)

skforecast.model_selection._utils.check_backtesting_input

check_backtesting_input(
    forecaster,
    cv,
    metric,
    add_aggregated_metric=True,
    y=None,
    series=None,
    exog=None,
    interval=None,
    interval_method="bootstrapping",
    alpha=None,
    n_boot=250,
    use_in_sample_residuals=True,
    use_binned_residuals=True,
    random_state=123,
    return_predictors=False,
    freeze_params=True,
    n_jobs="auto",
    show_progress=True,
    suppress_warnings=False,
)

This is a helper function to check most inputs of backtesting functions in modules model_selection.

Parameters:

Name	Type	Description	Default
forecaster	Forecaster	Forecaster model.	required
cv	TimeSeriesFold	TimeSeriesFold object with the information needed to split the data into folds.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model.	required
add_aggregated_metric	bool	If True, the aggregated metrics (average, weighted average and pooling) over all levels are also returned (only multiseries).	True
y	pandas Series	Training time series for uni-series forecasters.	None
series	pandas DataFrame, dict	Training time series for multi-series forecasters.	None
exog	pandas Series, pandas DataFrame, dict	Exogenous variables.	None
interval	(float, list, tuple, str, object)	Specifies whether probabilistic predictions should be estimated and the method to use. The following options are supported: If float, represents the nominal (expected) coverage (between 0 and 1). For instance, interval=0.95 corresponds to [0.025, 0.975] quantiles. If list or tuple: Sequence of quantiles to compute, each value must be between 0 and 1 inclusive. For example, a 95% confidence interval can be specified as interval = [0.025, 0.975] or multiple quantiles (e.g. 0.1, 0.5 and 0.9) as interval = [0.1, 0.5, 0.9]. If 'bootstrapping' (str): n_boot bootstrapping predictions will be generated. If scipy.stats distribution object, the distribution parameters will be estimated for each prediction. If None, no probabilistic predictions are estimated.	None
interval_method	str	Technique used to estimate prediction intervals. Available options: 'bootstrapping': Bootstrapping is used to generate prediction intervals. 'conformal': Employs the conformal prediction split method for interval estimation.	'bootstrapping'
alpha	float	The confidence intervals used in ForecasterStats are (1 - alpha) %.	None
n_boot	int	Number of bootstrapping iterations to perform when estimating prediction intervals.	`250`
use_in_sample_residuals	bool	If True, residuals from the training data are used as proxy of prediction error to create prediction intervals. If False, out_sample_residuals are used if they are already stored inside the forecaster.	True
use_binned_residuals	bool	If True, residuals are selected based on the predicted values (binned selection). If False, residuals are selected randomly.	True
random_state	int	Seed for the random number generator to ensure reproducibility.	`123`
return_predictors	bool	If True, the predictors used to make the predictions are also returned.	False
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_fit_forecaster.	`'auto'`
freeze_params	bool	Determines whether to freeze the model parameters after the first fit for estimators that perform automatic model selection (Arima with order=None or seasonal_order=None, Ets with model='ZZZ'). If True, the parameters selected in the first fit (best_params_) are reused in all subsequent refits, so the automatic selection runs only once. Arima keeps the orders, intercept, drift and Box-Cox lambda, and re-estimates the coefficients in each refit. Ets keeps the model type, smoothing parameters and Box-Cox lambda, and re-estimates the initial states. If False, automatic model selection is performed independently in each refit, allowing parameters to adapt across folds. This increases runtime and adds an estimator_params column to the output with the model selected in each fold.	True
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the backtesting process. See skforecast.exceptions.warn_skforecast_categories for more information.	False

Returns:

Type	Description
None

Source code in skforecast/model_selection/_utils.py

def check_backtesting_input(
    forecaster: object,
    cv: object,
    metric: str | Callable | list[str | Callable],
    add_aggregated_metric: bool = True,
    y: pd.Series | None = None,
    series: pd.DataFrame | dict[str, pd.Series | pd.DataFrame] = None,
    exog: pd.Series | pd.DataFrame | dict[str, pd.Series | pd.DataFrame] | None = None,
    interval: float | list[float] | tuple[float, ...] | str | object | None = None,
    interval_method: str = 'bootstrapping',    
    alpha: float | None = None,
    n_boot: int = 250,
    use_in_sample_residuals: bool = True,
    use_binned_residuals: bool = True,
    random_state: int = 123,
    return_predictors: bool = False,
    freeze_params: bool = True,
    n_jobs: int | str = 'auto',
    show_progress: bool = True,
    suppress_warnings: bool = False
) -> None:
    """
    This is a helper function to check most inputs of backtesting functions in 
    modules `model_selection`.

    Parameters
    ----------
    forecaster : Forecaster
        Forecaster model.
    cv : TimeSeriesFold
        TimeSeriesFold object with the information needed to split the data into folds.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.
    add_aggregated_metric : bool, default True
        If `True`, the aggregated metrics (average, weighted average and pooling)
        over all levels are also returned (only multiseries).
    y : pandas Series, default None
        Training time series for uni-series forecasters.
    series : pandas DataFrame, dict, default None
        Training time series for multi-series forecasters.
    exog : pandas Series, pandas DataFrame, dict, default None
        Exogenous variables.
    interval : float, list, tuple, str, object, default None
        Specifies whether probabilistic predictions should be estimated and the 
        method to use. The following options are supported:

        - If `float`, represents the nominal (expected) coverage (between 0 and 1). 
        For instance, `interval=0.95` corresponds to `[0.025, 0.975]` quantiles.
        - If `list` or `tuple`: Sequence of quantiles to compute, each value must 
        be between 0 and 1 inclusive. For example, a 95% confidence interval can 
        be specified as `interval = [0.025, 0.975]` or multiple quantiles (e.g. 0.1, 
        0.5 and 0.9) as `interval = [0.1, 0.5, 0.9]`.
        - If 'bootstrapping' (str): `n_boot` bootstrapping predictions will be generated.
        - If scipy.stats distribution object, the distribution parameters will
        be estimated for each prediction.
        - If None, no probabilistic predictions are estimated.
    interval_method : str, default 'bootstrapping'
        Technique used to estimate prediction intervals. Available options:

        + 'bootstrapping': Bootstrapping is used to generate prediction 
        intervals.
        + 'conformal': Employs the conformal prediction split method for 
        interval estimation.
    alpha : float, default None
        The confidence intervals used in ForecasterStats are (1 - alpha) %. 
    n_boot : int, default `250`
        Number of bootstrapping iterations to perform when estimating prediction
            intervals.
    use_in_sample_residuals : bool, default True
        If `True`, residuals from the training data are used as proxy of prediction 
        error to create prediction intervals.  If `False`, out_sample_residuals 
        are used if they are already stored inside the forecaster.
    use_binned_residuals : bool, default True
        If `True`, residuals are selected based on the predicted values 
        (binned selection).
        If `False`, residuals are selected randomly.
    random_state : int, default `123`
        Seed for the random number generator to ensure reproducibility.
    return_predictors : bool, default False
        If `True`, the predictors used to make the predictions are also returned.
    n_jobs : int, 'auto', default `'auto'`
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_fit_forecaster.
    freeze_params : bool, default True
        Determines whether to freeze the model parameters after the first fit
        for estimators that perform automatic model selection (`Arima` with
        `order=None` or `seasonal_order=None`, `Ets` with `model='ZZZ'`).

        - If `True`, the parameters selected in the first fit (`best_params_`)
        are reused in all subsequent refits, so the automatic selection runs only
        once. Arima keeps the orders, intercept, drift and Box-Cox lambda, and
        re-estimates the coefficients in each refit. Ets keeps the model type,
        smoothing parameters and Box-Cox lambda, and re-estimates the initial
        states.
        - If `False`, automatic model selection is performed independently in each
        refit, allowing parameters to adapt across folds. This increases runtime
        and adds an `estimator_params` column to the output with the model
        selected in each fold.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings: bool, default False
        If `True`, skforecast warnings will be suppressed during the backtesting 
        process. See skforecast.exceptions.warn_skforecast_categories for more
        information.

    Returns
    -------
    None

    """

    forecaster_name = type(forecaster).__name__
    cv_name = type(cv).__name__

    if cv_name != "TimeSeriesFold":
        raise TypeError(f"`cv` must be a 'TimeSeriesFold' object. Got '{cv_name}'.")

    steps = cv.steps
    initial_train_size = cv.initial_train_size
    gap = cv.gap
    allow_incomplete_fold = cv.allow_incomplete_fold
    refit = cv.refit

    forecasters_uni = [
        "ForecasterRecursive",
        "ForecasterDirect",
        "ForecasterStats",
        "ForecasterEquivalentDate",
        "ForecasterRecursiveClassifier"
    ]
    forecasters_direct = [
        "ForecasterDirect",
        "ForecasterDirectMultiVariate",
        "ForecasterRnn"
    ]
    forecasters_multi_no_dict = [
        "ForecasterDirectMultiVariate",
        "ForecasterRnn",
    ]
    forecasters_multi_dict = [
        "ForecasterRecursiveMultiSeries",
        "ForecasterFoundation"
    ]
    # NOTE: ForecasterStats has interval but not with bootstrapping or conformal
    # NOTE: ForecasterRnn has interval with conformal but not bootstrapping
    forecasters_boot_conformal = [
        "ForecasterRecursive",
        "ForecasterDirect",
        "ForecasterRecursiveMultiSeries",
        "ForecasterDirectMultiVariate",
        "ForecasterEquivalentDate",
    ]
    forecasters_return_predictors = [
        "ForecasterRecursive",
        "ForecasterDirect",
        "ForecasterRecursiveMultiSeries",
        "ForecasterDirectMultiVariate",
        "ForecasterRecursiveClassifier"
    ]

    if forecaster_name in forecasters_uni:
        if not isinstance(y, pd.Series):
            raise TypeError("`y` must be a pandas Series.")
        data_name = 'y'
        data_length_name = "the length of `y`"
        index = cv._extract_index(y)

    elif forecaster_name in forecasters_multi_no_dict:
        if not isinstance(series, pd.DataFrame):
            raise TypeError("`series` must be a pandas DataFrame.")
        data_name = 'series'
        data_length_name = "the length of `series`"
        index = cv._extract_index(series)

    elif forecaster_name in forecasters_multi_dict:

        # NOTE: Checks are not need as they are done in the function
        # `check_preprocess_series` that is used before `check_backtesting_input`
        # in the backtesting function.

        # NOTE: The folds are built over the span of all series (from the
        # earliest to the latest index), which is longer than any of them when
        # they do not cover the same period.
        data_name = 'series'
        data_length_name = "the length of the span of `series`"
        index = cv._extract_index(series)

    data_length = len(index)

    if exog is not None:
        if forecaster_name in forecasters_multi_dict:
            # NOTE: Checks are not need as they are done in the function
            # `check_preprocess_exog_multiseries` that is used before
            # `check_backtesting_input` in the backtesting function.
            pass
        else:
            if not isinstance(exog, (pd.Series, pd.DataFrame)):
                raise TypeError(
                    f"`exog` must be a pandas Series, DataFrame or None. Got {type(exog)}."
                )

    if not isinstance(metric, (str, Callable, list)):
        raise TypeError(
            f"`metric` must be a string, a callable function, or a list containing "
            f"multiple strings and/or callables. Got {type(metric)}."
        )

    if forecaster_name == "ForecasterEquivalentDate" and isinstance(
        forecaster.offset, pd.tseries.offsets.DateOffset
    ):
        # NOTE: Checks when initial_train_size is not None cannot be done here
        # because the forecaster is not fitted yet and we don't know the
        # window_size since pd.DateOffset is not a fixed window size.
        if initial_train_size is None:
            raise ValueError(
                f"`initial_train_size` must be an integer greater than "
                f"the `window_size` of the forecaster ({forecaster.window_size}) "
                f"and smaller than {data_length_name} ({data_length}) or "
                f"a date within this range of the index."
            )
    elif initial_train_size is not None:
        initial_train_size = date_to_index_position(
                                 index        = index, 
                                 date_input   = initial_train_size, 
                                 method       = 'validation',
                                 date_literal = 'initial_train_size'
                             )
        # NOTE: For ForecasterFoundation, window_size equals context_length,
        # the *maximum* context the model accepts, not a minimum training size:
        # initial_train_size only needs to be at least 1. The window_size of
        # ForecasterStats is ignored, initial_train_size can be equal to it.
        _check_initial_train_size(
            initial_train_size       = initial_train_size,
            initial_train_size_input = cv.initial_train_size,
            data_length              = data_length,
            data_length_name         = data_length_name,
            window_size              = (
                None if forecaster_name == 'ForecasterFoundation'
                else forecaster.window_size
            ),
            allow_window_size        = forecaster_name == 'ForecasterStats'
        )
        if allow_incomplete_fold:
            # At least one observation after the gap to allow incomplete fold
            if data_length <= initial_train_size + gap:
                raise ValueError(
                    f"`{data_name}` must have more than `initial_train_size + gap` "
                    f"observations to create at least one fold.\n"
                    f"    Time series length: {data_length}\n"
                    f"    Required > {initial_train_size + gap}\n"
                    f"    initial_train_size: {initial_train_size}\n"
                    f"    gap: {gap}\n"
                )
        else:
            # At least one complete fold
            if data_length < initial_train_size + gap + steps:
                raise ValueError(
                    f"`{data_name}` must have at least `initial_train_size + gap + steps` "
                    f"observations to create a minimum of one complete fold "
                    f"(allow_incomplete_fold=False).\n"
                    f"    Time series length: {data_length}\n"
                    f"    Required >= {initial_train_size + gap + steps}\n"
                    f"    initial_train_size: {initial_train_size}\n"
                    f"    gap: {gap}\n"
                    f"    steps: {steps}\n"
                )
    else:
        if forecaster_name in [
            'ForecasterStats', 'ForecasterFoundation', 'ForecasterEquivalentDate'
        ]:
            raise ValueError(
                f"When using {forecaster_name}, `initial_train_size` must be an "
                f"integer smaller than {data_length_name} ({data_length})."
            )
        else:
            if not forecaster.is_fitted:
                raise NotFittedError(
                    "`forecaster` must be already trained if no `initial_train_size` "
                    "is provided."
                )
            if refit:
                raise ValueError(
                    "`refit` is only allowed when `initial_train_size` is not `None`."
                )

    if forecaster_name == 'ForecasterStats' and cv.skip_folds is not None:
        raise ValueError(
            "`skip_folds` is not allowed for ForecasterStats. Set it to `None`."
        )

    # NOTE: The names are those of the columns of the metrics table. A callable
    # without `__name__` (for example, a `functools.partial`) is named after
    # the wrapper of `add_y_train_argument`.
    metric_names = [
        m if isinstance(m, str) else getattr(add_y_train_argument(m), '__name__', None)
        for m in (metric if isinstance(metric, list) else [metric])
    ]
    duplicated_names = list(dict.fromkeys(
        name for name in metric_names
        if name is not None and metric_names.count(name) > 1
    ))
    if duplicated_names:
        raise ValueError(
            f"Each metric name must be unique, they are the columns of the "
            f"metrics table. Duplicated names: {duplicated_names}. Give each "
            f"function a distinct `__name__` (for example, "
            f"`create_mean_pinball_loss` includes the quantile in the name)."
        )
    if N_EVALUATED_NAME in metric_names:
        raise ValueError(
            f"The backtesting metrics include the column '{N_EVALUATED_NAME}' "
            f"with the number of points evaluated. Rename the metric "
            f"'{N_EVALUATED_NAME}' passed in `metric`."
        )

    # NOTE: `backtesting_foundation` passes its `quantiles` as `interval`.
    if _get_interval_bound_quantiles(interval) is not None or alpha is not None:
        reserved_names = [name for name in metric_names if name in INTERVAL_METRIC_NAMES]
        if reserved_names:
            raise ValueError(
                f"When `interval` has a lower and an upper bound, or `alpha` is "
                f"used, the backtesting metrics include {INTERVAL_METRIC_NAMES}. "
                f"Rename the metric(s) {reserved_names} passed in `metric`."
            )

    if not isinstance(add_aggregated_metric, bool):
        raise TypeError("`add_aggregated_metric` must be a boolean: `True`, `False`.")
    if not isinstance(n_boot, (int, np.integer)) or n_boot < 0:
        raise TypeError(f"`n_boot` must be an integer greater than 0. Got {n_boot}.")
    if not isinstance(use_in_sample_residuals, bool):
        raise TypeError("`use_in_sample_residuals` must be a boolean: `True`, `False`.")
    if not isinstance(use_binned_residuals, bool):
        raise TypeError("`use_binned_residuals` must be a boolean: `True`, `False`.")
    if not isinstance(random_state, (int, np.integer)) or random_state < 0:
        raise TypeError(f"`random_state` must be an integer greater than 0. Got {random_state}.")
    if not isinstance(return_predictors, bool):
        raise TypeError("`return_predictors` must be a boolean: `True`, `False`.")
    if not isinstance(freeze_params, bool):
        raise TypeError("`freeze_params` must be a boolean: `True`, `False`.")
    if not isinstance(n_jobs, int) and n_jobs != 'auto':
        raise TypeError(f"`n_jobs` must be an integer or `'auto'`. Got {n_jobs}.")
    if not isinstance(show_progress, bool):
        raise TypeError("`show_progress` must be a boolean: `True`, `False`.")
    if not isinstance(suppress_warnings, bool):
        raise TypeError("`suppress_warnings` must be a boolean: `True`, `False`.")

    if interval is not None or alpha is not None:

        if forecaster_name in forecasters_boot_conformal:

            if interval_method == 'conformal':
                if not isinstance(interval, (float, list, tuple)):
                    raise TypeError(
                        f"When `interval_method` is 'conformal', `interval` must "
                        f"be a float or a list/tuple defining a symmetric interval. "
                        f"Got {type(interval)}."
                    )
                if isinstance(interval, (list, tuple)) and len(interval) != 2:
                    raise ValueError(
                        f"When `interval_method` is 'conformal', `interval` must be "
                        f"a float or a list/tuple of exactly 2 values (lower and "
                        f"upper quantile) defining a symmetric interval. Multiple "
                        f"quantiles (e.g. `interval=[0.1, 0.5, 0.9]`) are only "
                        f"supported when `interval_method='bootstrapping'`. "
                        f"Got {interval}."
                    )
            elif interval_method == 'bootstrapping':
                if (
                    not isinstance(interval, (float, list, tuple, str))
                    and (not hasattr(interval, "_pdf") or not callable(getattr(interval, "fit", None)))
                ):                
                    raise TypeError(
                        f"When `interval_method` is 'bootstrapping', `interval` "
                        f"must be a float, a list or tuple of floats, a "
                        f"scipy.stats distribution object (with methods `_pdf` and "
                        f"`fit`) or the string 'bootstrapping'. Got {type(interval)}."
                    )
                if isinstance(interval, (list, tuple)):
                    if len(interval) == 2:
                        check_interval(
                            interval                   = interval,
                            ensure_symmetric_intervals = False
                        )
                    else:
                        for q in interval:
                            if (q < 0.) or (q > 1.):
                                raise ValueError(
                                    f"When `interval` is a list or tuple, all values must be "
                                    f"between 0 and 1 inclusive. Got {q} in {interval}."
                                )
                elif isinstance(interval, float):
                    if (interval <= 0.) or (interval >= 1.):
                        raise ValueError(
                            f"When `interval` is a float, it must be between 0 and 1 "
                            f"exclusive. Got {interval}."
                        )
                elif isinstance(interval, str):
                    if interval != 'bootstrapping':
                        raise ValueError(
                            f"When `interval` is a string, it must be 'bootstrapping'."
                            f"Got {interval}."
                        )
            else:
                raise ValueError(
                    f"`interval_method` must be 'bootstrapping' or 'conformal'. "
                    f"Got {interval_method}."
                )
        else:
            if forecaster_name == 'ForecasterRecursiveClassifier':
                raise ValueError(
                    f"`interval` is not supported for {forecaster_name}. Class "
                    f"probabilities are returned by default during backtesting, "
                    f"set `interval=None`."
                )
            if forecaster_name == 'ForecasterFoundation':
                check_interval(quantiles=interval)
            else:
                check_interval(interval=interval, alpha=alpha)

    if return_predictors and forecaster_name not in forecasters_return_predictors:
        raise ValueError(
            f"`return_predictors` is only allowed for forecasters of type "
            f"{forecasters_return_predictors}. Got {forecaster_name}."
        )

    if forecaster_name in forecasters_direct and forecaster.max_step < steps + gap:
        raise ValueError(
            f"When using a {forecaster_name}, the combination of steps "
            f"+ gap ({steps + gap}) cannot be greater than the `steps` parameter "
            f"declared when the forecaster is initialized ({forecaster.max_step})."
        )

skforecast.model_selection._utils.check_one_step_ahead_input

check_one_step_ahead_input(
    forecaster,
    cv,
    metric,
    y=None,
    series=None,
    exog=None,
    show_progress=True,
    suppress_warnings=False,
)

This is a helper function to check most inputs of hyperparameter tuning functions in modules model_selection when using a OneStepAheadFold.

Parameters:

Name	Type	Description	Default
forecaster	Forecaster	Forecaster model.	required
cv	OneStepAheadFold	OneStepAheadFold object with the information needed to split the data into folds.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model.	required
y	pandas Series	Training time series for uni-series forecasters.	None
series	pandas DataFrame, dict	Training time series for multi-series forecasters.	None
exog	pandas Series, pandas DataFrame, dict	Exogenous variables.	None
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information.	False

Returns:

Type	Description
None

Source code in skforecast/model_selection/_utils.py

def check_one_step_ahead_input(
    forecaster: object,
    cv: object,
    metric: str | Callable | list[str | Callable],
    y: pd.Series | None = None,
    series: pd.DataFrame | dict[str, pd.Series | pd.DataFrame] = None,
    exog: pd.Series | pd.DataFrame | dict[str, pd.Series | pd.DataFrame] | None = None,
    show_progress: bool = True,
    suppress_warnings: bool = False
) -> None:
    """
    This is a helper function to check most inputs of hyperparameter tuning
    functions in modules `model_selection` when using a `OneStepAheadFold`.

    Parameters
    ----------
    forecaster : Forecaster
        Forecaster model.
    cv : OneStepAheadFold
        OneStepAheadFold object with the information needed to split the data into folds.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.
    y : pandas Series, default None
        Training time series for uni-series forecasters.
    series : pandas DataFrame, dict, default None
        Training time series for multi-series forecasters.
    exog : pandas Series, pandas DataFrame, dict, default None
        Exogenous variables.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings: bool, default False
        If `True`, skforecast warnings will be suppressed during the hyperparameter 
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.

    Returns
    -------
    None

    """

    forecaster_name = type(forecaster).__name__
    cv_name = type(cv).__name__

    if cv_name != "OneStepAheadFold":
        raise TypeError(f"`cv` must be a 'OneStepAheadFold' object. Got '{cv_name}'.")

    initial_train_size = cv.initial_train_size

    forecasters_one_step_ahead = [
        "ForecasterRecursive",
        "ForecasterDirect",
        "ForecasterRecursiveClassifier",
        'ForecasterRecursiveMultiSeries',
        'ForecasterDirectMultiVariate'
    ]
    if forecaster_name not in forecasters_one_step_ahead:
        raise TypeError(
            f"Only forecasters of type {forecasters_one_step_ahead} are allowed "
            f"when using `cv` of type `OneStepAheadFold`. Got {forecaster_name}."
        )

    forecasters_uni = [
        "ForecasterRecursive",
        "ForecasterDirect",
        "ForecasterRecursiveClassifier"
    ]
    forecasters_multi_no_dict = [
        "ForecasterDirectMultiVariate",
    ]
    forecasters_multi_dict = [
        "ForecasterRecursiveMultiSeries"
    ]

    if forecaster_name in forecasters_uni:
        if not isinstance(y, pd.Series):
            raise TypeError(f"`y` must be a pandas Series. Got {type(y)}")
        data_length_name = "the length of `y`"
        index = cv._extract_index(y)

    elif forecaster_name in forecasters_multi_no_dict:
        if not isinstance(series, pd.DataFrame):
            raise TypeError(f"`series` must be a pandas DataFrame. Got {type(series)}")
        data_length_name = "the length of `series`"
        index = cv._extract_index(series)

    elif forecaster_name in forecasters_multi_dict:

        # NOTE: Checks are not need as they are done in the function
        # `check_preprocess_series` that is used before `check_one_step_ahead_input`
        # in the backtesting function.

        # NOTE: The training and test sets are split over the span of all series
        # (from the earliest to the latest index), which is longer than any of
        # them when they do not cover the same period.
        data_length_name = "the length of the span of `series`"
        index = cv._extract_index(series)

    data_length = len(index)

    if exog is not None:
        if forecaster_name in forecasters_multi_dict:
            # NOTE: Checks are not need as they are done in the function 
            # `check_preprocess_exog_multiseries` that is used before 
            # `check_backtesting_input` in the backtesting function.
            pass
        else:
            if not isinstance(exog, (pd.Series, pd.DataFrame)):
                raise TypeError(
                    f"`exog` must be a pandas Series, DataFrame or None. Got {type(exog)}."
                )

    if not isinstance(metric, (str, Callable, list)):
        raise TypeError(
            f"`metric` must be a string, a callable function, or a list containing "
            f"multiple strings and/or callables. Got {type(metric)}."
        )

    initial_train_size = date_to_index_position(
                             index        = index,
                             date_input   = initial_train_size,
                             method       = 'validation',
                             date_literal = 'initial_train_size'
                         )
    # NOTE: The `window_size` is checked for each candidate of the search
    # (`_train_test_split_one_step_ahead_search`), since the candidates can
    # have a different `window_size` than the forecaster passed.
    _check_initial_train_size(
        initial_train_size       = initial_train_size,
        initial_train_size_input = cv.initial_train_size,
        data_length              = data_length,
        data_length_name         = data_length_name,
        window_size              = None
    )

    if not isinstance(show_progress, bool):
        raise TypeError("`show_progress` must be a boolean: `True`, `False`.")
    if not isinstance(suppress_warnings, bool):
        raise TypeError("`suppress_warnings` must be a boolean: `True`, `False`.")

    if not suppress_warnings:
        warnings.warn(
            "One-step-ahead predictions are used for faster model comparison, but they "
            "may not fully represent multi-step prediction performance. It is recommended "
            "to backtest the final model for a more accurate multi-step performance "
            "estimate.",
            OneStepAheadValidationWarning,
            stacklevel = _find_stack_level()
        )

skforecast.model_selection._utils.select_n_jobs_backtesting

select_n_jobs_backtesting(forecaster, refit)

Select the optimal number of jobs to use in the backtesting process. This selection is based on heuristics and is not guaranteed to be optimal.

The number of jobs is chosen as follows:

• If refit is an integer other than 0 or 1, then n_jobs = 1. The forecaster fitted in a fold is reused in the following ones, so the folds cannot run in parallel.
• If forecaster is not 'ForecasterRecursive', 'ForecasterRecursiveClassifier' or 'ForecasterRecursiveMultiSeries', then n_jobs = 1. 'ForecasterDirect' and 'ForecasterDirectMultiVariate' parallelize the fit of their steps instead (see skforecast.utils.select_n_jobs_fit_forecaster), and 'ForecasterStats', 'ForecasterEquivalentDate' and 'ForecasterRnn' are not parallelized.
• If refit is False (or 0), then n_jobs = 1, unless estimator, or the last step of a Pipeline, is a HistGradientBoostingRegressor or a CatBoostRegressor. The folds only predict with the forecaster fitted before them, and sending the fitted forecaster to the workers usually costs more than the predictions they share. These two estimators predict a few rows slowly and their fitted models are small, so their predictions pay for sending them.
• If estimator, or the last step of a Pipeline, is a linear model of scikit-learn (a subclass of LinearModel, such as Ridge), then n_jobs = 1. It is fitted and predicts in less time than the workers take to start. LogisticRegression is not a subclass of LinearModel: it is fitted iteratively, so it is parallelized.
• If estimator, or the last step of a Pipeline, is a LGBMRegressor or LGBMClassifier with internal n_jobs != 1, then n_jobs = 1. LightGBM already parallelizes its operations with threads, and more processes would compete for the same cores.
• Otherwise, n_jobs = max(1, cpu_count() - 1).

The hyperparameter searches use these rules for every candidate. A single backtesting (backtesting_forecaster, backtesting_forecaster_multiseries) then limits the result by its workload with skforecast.model_selection._utils.limit_n_jobs_by_workload.

Parameters:

Name	Type	Description	Default
forecaster	Forecaster	Forecaster model.	required
refit	(bool, int)	If the forecaster is refitted during the backtesting process.	required

Returns:

Name	Type	Description
n_jobs	int	The number of jobs to run in parallel.

Source code in skforecast/model_selection/_utils.py

def select_n_jobs_backtesting(
    forecaster: object,
    refit: bool | int
) -> int:
    """
    Select the optimal number of jobs to use in the backtesting process. This
    selection is based on heuristics and is not guaranteed to be optimal.

    The number of jobs is chosen as follows:

    - If `refit` is an integer other than 0 or 1, then `n_jobs = 1`. The
    forecaster fitted in a fold is reused in the following ones, so the folds
    cannot run in parallel.
    - If forecaster is not 'ForecasterRecursive', 'ForecasterRecursiveClassifier'
    or 'ForecasterRecursiveMultiSeries', then `n_jobs = 1`. 'ForecasterDirect'
    and 'ForecasterDirectMultiVariate' parallelize the fit of their steps
    instead (see `skforecast.utils.select_n_jobs_fit_forecaster`), and
    'ForecasterStats', 'ForecasterEquivalentDate' and 'ForecasterRnn' are not
    parallelized.
    - If `refit` is `False` (or 0), then `n_jobs = 1`, unless estimator, or the
    last step of a `Pipeline`, is a `HistGradientBoostingRegressor` or a
    `CatBoostRegressor`. The folds only predict with the forecaster fitted
    before them, and sending the fitted forecaster to the workers usually costs
    more than the predictions they share. These two estimators predict a few
    rows slowly and their fitted models are small, so their predictions pay
    for sending them.
    - If estimator, or the last step of a `Pipeline`, is a linear model of
    scikit-learn (a subclass of `LinearModel`, such as `Ridge`), then
    `n_jobs = 1`. It is fitted and predicts in less time than the workers
    take to start. `LogisticRegression` is not a subclass of `LinearModel`:
    it is fitted iteratively, so it is parallelized.
    - If estimator, or the last step of a `Pipeline`, is a `LGBMRegressor` or
    `LGBMClassifier` with internal `n_jobs != 1`, then `n_jobs = 1`. LightGBM
    already parallelizes its operations with threads, and more processes
    would compete for the same cores.
    - Otherwise, `n_jobs = max(1, cpu_count() - 1)`.

    The hyperparameter searches use these rules for every candidate. A single
    backtesting (`backtesting_forecaster`, `backtesting_forecaster_multiseries`)
    then limits the result by its workload with
    `skforecast.model_selection._utils.limit_n_jobs_by_workload`.

    Parameters
    ----------
    forecaster : Forecaster
        Forecaster model.
    refit : bool, int
        If the forecaster is refitted during the backtesting process.

    Returns
    -------
    n_jobs : int
        The number of jobs to run in parallel.

    """

    refit = False if refit == 0 else refit
    if not isinstance(refit, bool) and refit != 1:
        return 1

    if type(forecaster).__name__ not in {
        "ForecasterRecursive",
        "ForecasterRecursiveClassifier",
        "ForecasterRecursiveMultiSeries",
    }:
        return 1

    estimator = forecaster.estimator
    if isinstance(estimator, Pipeline):
        estimator = estimator[-1]

    # NOTE: Without refit, each task receives the fitted forecaster and only
    # predicts, so parallelizing only pays off when the prediction is slow and
    # the fitted model is cheap to send to the workers. That is the case of
    # `HistGradientBoostingRegressor` and `CatBoostRegressor` (high fixed cost
    # per predict call, models of a few hundred KB): their searches without
    # refit were 1.5 to 3 times faster in parallel, at the cost of starting the
    # workers in small searches. LightGBM, XGBoost and RandomForest did not
    # gain. Review this exception if these libraries make their prediction
    # faster. A single backtesting without refit still uses one job
    # (`limit_n_jobs_by_workload`).
    if (
        refit is False
        and type(estimator).__name__
        not in {"HistGradientBoostingRegressor", "CatBoostRegressor"}
    ):
        return 1

    if isinstance(estimator, LinearModel):
        return 1

    if (
        type(estimator).__name__ in {"LGBMRegressor", "LGBMClassifier"}
        and estimator.n_jobs != 1
    ):
        return 1

    return max(1, cpu_count() - 1)

skforecast.model_selection._utils.limit_n_jobs_by_workload

limit_n_jobs_by_workload(
    n_jobs, n_tasks, n_fits, fit_seconds
)

Limit the number of jobs selected with n_jobs='auto' by the workload of a single backtesting. Starting the workers of joblib takes seconds (more on Windows than on Linux), so parallelizing only pays off when the folds have enough work to share. The result is never greater than n_jobs.

• If n_jobs is 1, there is at most one task or no task fits the forecaster (no refit), then n_jobs = 1. Predicting the folds of a fitted forecaster takes less time than the workers take to start.
• If the estimated workload, n_fits * fit_seconds, is below the minimum of the platform (4 seconds on Windows, 2 seconds on Linux and macOS), then n_jobs = 1. If the time of the initial fit is not known (fit_seconds=None), the workload is not estimated.
• Otherwise, n_jobs = min(n_jobs, n_tasks): no more workers are started than there are tasks.

Parameters:

Name	Type	Description	Default
n_jobs	int	Number of jobs selected with select_n_jobs_backtesting.	required
n_tasks	int	Number of tasks (folds) sent to joblib.Parallel.	required
n_fits	int	Number of fits made in the tasks (folds with refit), without the initial fit.	required
fit_seconds	(float, None)	Duration of the initial fit, in seconds. None if the forecaster was not fitted before the folds.	required

Returns:

Name	Type	Description
n_jobs	int	The number of jobs to run in parallel.

Source code in skforecast/model_selection/_utils.py

def limit_n_jobs_by_workload(
    n_jobs: int,
    n_tasks: int,
    n_fits: int,
    fit_seconds: float | None
) -> int:
    """
    Limit the number of jobs selected with `n_jobs='auto'` by the workload of
    a single backtesting. Starting the workers of `joblib` takes seconds (more
    on Windows than on Linux), so parallelizing only pays off when the folds
    have enough work to share. The result is never greater than `n_jobs`.

    - If `n_jobs` is 1, there is at most one task or no task fits the
    forecaster (no refit), then `n_jobs = 1`. Predicting the folds of a fitted
    forecaster takes less time than the workers take to start.
    - If the estimated workload, `n_fits * fit_seconds`, is below the minimum
    of the platform (4 seconds on Windows, 2 seconds on Linux and macOS), then
    `n_jobs = 1`. If the time of the initial fit is not known
    (`fit_seconds=None`), the workload is not estimated.
    - Otherwise, `n_jobs = min(n_jobs, n_tasks)`: no more workers are started
    than there are tasks.

    Parameters
    ----------
    n_jobs : int
        Number of jobs selected with `select_n_jobs_backtesting`.
    n_tasks : int
        Number of tasks (folds) sent to `joblib.Parallel`.
    n_fits : int
        Number of fits made in the tasks (folds with refit), without the
        initial fit.
    fit_seconds : float, None
        Duration of the initial fit, in seconds. `None` if the forecaster
        was not fitted before the folds.

    Returns
    -------
    n_jobs : int
        The number of jobs to run in parallel.

    """

    if n_jobs <= 1 or n_tasks <= 1 or n_fits == 0:
        return 1

    if fit_seconds is not None:
        min_workload = _MIN_PARALLEL_WORKLOAD_SECONDS.get(
            sys.platform, _MIN_PARALLEL_WORKLOAD_SECONDS_DEFAULT
        )
        if n_fits * fit_seconds < min_workload:
            return 1

    return min(n_jobs, n_tasks)

skforecast.model_selection._utils.initialize_lags_grid

initialize_lags_grid(forecaster, lags_grid=None)

Initialize lags grid and lags label for model selection.

Parameters:

Name	Type	Description	Default
forecaster	Forecaster	Forecaster model. ForecasterRecursive, ForecasterDirect, ForecasterRecursiveMultiSeries, ForecasterDirectMultiVariate.	required
lags_grid	(list, dict)	Lists of lags to try, containing int, lists, numpy ndarray, range objects or None. A None element evaluates the forecaster without lags, using only its window_features (a ValueError is raised if it has none). If dict, the keys are used as labels in the results DataFrame, and the values are used as the lists of lags to try. If None, the lags of the forecaster are used.	None

Returns:

Name	Type	Description
lags_grid	dict	Dictionary with lags configuration for each iteration.
lags_label	str	Label for lags representation in the results object.

Source code in skforecast/model_selection/_utils.py

def initialize_lags_grid(
    forecaster: object, 
    lags_grid: (
        list[int | list[int] | np.ndarray | range | None]
        | dict[str, int | list[int] | np.ndarray | range | None]
        | None
    ) = None,
) -> tuple[dict[str, int | list[int] | np.ndarray | range | None], str]:
    """
    Initialize lags grid and lags label for model selection. 

    Parameters
    ----------
    forecaster : Forecaster
        Forecaster model. ForecasterRecursive, ForecasterDirect, 
        ForecasterRecursiveMultiSeries, ForecasterDirectMultiVariate.
    lags_grid : list, dict, default None
        Lists of lags to try, containing int, lists, numpy ndarray, range
        objects or `None`. A `None` element evaluates the forecaster without
        lags, using only its `window_features` (a `ValueError` is raised if
        it has none). If `dict`, the keys are used as labels in the `results`
        DataFrame, and the values are used as the lists of lags to try. If
        `None`, the lags of the forecaster are used.

    Returns
    -------
    lags_grid : dict
        Dictionary with lags configuration for each iteration.
    lags_label : str
        Label for lags representation in the results object.

    """

    if not isinstance(lags_grid, (list, dict, type(None))):
        raise TypeError(
            f"`lags_grid` argument must be a list, dict or None. "
            f"Got {type(lags_grid)}."
        )

    lags_label = 'values'
    if isinstance(lags_grid, list):
        lags_grid = {f'{lags}': lags for lags in lags_grid}
    elif lags_grid is None:
        if forecaster.lags is None:
            lags_grid = {'None': None}
        else:
            lags = [int(lag) for lag in forecaster.lags]  # Required since numpy 2.0
            lags_grid = {f'{lags}': lags}
    else:
        lags_label = 'keys'

    if (
        any(lags is None for lags in lags_grid.values())
        and forecaster.window_features is None
    ):
        raise ValueError(
            "`lags_grid` contains `None`, a candidate without lags, but the "
            "forecaster has no `window_features` to create the predictors. Set "
            "`window_features` in the forecaster or remove `None` from `lags_grid`."
        )

    return lags_grid, lags_label