model_selection

skforecast.model_selection._validation.backtesting_forecaster

backtesting_forecaster(
    forecaster,
    y,
    cv,
    metric,
    exog=None,
    interval=None,
    interval_method="bootstrapping",
    n_boot=250,
    use_in_sample_residuals=True,
    use_binned_residuals=True,
    random_state=123,
    return_predictors=False,
    n_jobs="auto",
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
)

Backtesting of forecaster model following the folds generated by the TimeSeriesFold class and using the metric(s) provided.

If forecaster is already trained and initial_train_size is set to None in the TimeSeriesFold class, no initial train will be done and all data will be used to evaluate the model. However, the first len(forecaster.last_window) observations are needed to create the initial predictors, so no predictions are calculated for them.

A copy of the original forecaster is created so that it is not modified during the process.

Parameters:

Name	Type	Description	Default
forecaster	(ForecasterRecursive, ForecasterDirect, ForecasterEquivalentDate, ForecasterRecursiveClassifier)	Forecaster model.	required
y	pandas Series	Training time series.	required
cv	TimeSeriesFold	TimeSeriesFold object with the information needed to split the data into folds.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'} If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. If list: List containing multiple strings and/or Callables.	required
exog	pandas Series, pandas DataFrame	Exogenous variable/s included as predictor/s. Must have the same number of observations as y and should be aligned so that y[i] is regressed on exog[i].	None
interval	(float, list, tuple, str, object)	Specifies whether probabilistic predictions should be estimated and the method to use. The following options are supported: If float, represents the nominal (expected) coverage (between 0 and 1). For instance, interval=0.95 corresponds to [2.5, 97.5] percentiles. If list or tuple: Sequence of percentiles to compute, each value must be between 0 and 100 inclusive. For example, a 95% confidence interval can be specified as interval = [2.5, 97.5] or multiple percentiles (e.g. 10, 50 and 90) as interval = [10, 50, 90]. If 'bootstrapping' (str): n_boot bootstrapping predictions will be generated. If scipy.stats distribution object, the distribution parameters will be estimated for each prediction. If None, no probabilistic predictions are estimated.	None
interval_method	str	Technique used to estimate prediction intervals. Available options: 'bootstrapping': Bootstrapping is used to generate prediction intervals [1]_. 'conformal': Employs the conformal prediction split method for interval estimation [2]_.	'bootstrapping'
n_boot	int	Number of bootstrapping iterations to perform when estimating prediction intervals.	250
use_in_sample_residuals	bool	If True, residuals from the training data are used as proxy of prediction error to create predictions. If False, out of sample residuals (calibration) are used. Out-of-sample residuals must be precomputed using Forecaster's set_out_sample_residuals() method.	True
use_binned_residuals	bool	If True, residuals are selected based on the predicted values (binned selection). If False, residuals are selected randomly.	True
random_state	int	Seed for the random number generator to ensure reproducibility.	123
return_predictors	bool	If True, the predictors used to make the predictions are also returned.	False
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting.	'auto'
verbose	bool	Print number of folds and index of training and validation sets used for backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the backtesting process. See skforecast.exceptions.warn_skforecast_categories for more information.	False

Returns:

Name	Type	Description
metric_values	pandas DataFrame	Value(s) of the metric(s).
backtest_predictions	pandas DataFrame	Value of predictions. The DataFrame includes the following columns: fold: Indicates the fold number where the prediction was made. pred: Predicted values for the corresponding series and time steps. If interval is not None, additional columns are included depending on the method: For float: Columns lower_bound and upper_bound. For list or tuple of 2 elements: Columns lower_bound and upper_bound. For list or tuple with multiple percentiles: One column per percentile (e.g., p_10, p_50, p_90). For 'bootstrapping': One column per bootstrapping iteration (e.g., pred_boot_0, pred_boot_1, ..., pred_boot_n). For scipy.stats distribution objects: One column for each estimated parameter of the distribution (e.g., loc, scale). If return_predictors is True, one column per predictor is created. Depending on the relation between steps and fold_stride, the output may include repeated indexes (if fold_stride < steps) or gaps (if fold_stride > steps). See Notes below for more details.

Notes

Note on fold_stride vs. steps:

• If fold_stride == steps, test sets are placed back-to-back without overlap. Each observation appears only once in the output DataFrame, so the index is unique.
• If fold_stride < steps, test sets overlap. Multiple forecasts are generated for the same observations and, therefore, the output DataFrame contains repeated indexes.
• If fold_stride > steps, there are gaps between consecutive test sets. Some observations in the series will not have associated predictions, so the output DataFrame has non-contiguous indexes.

References

.. [1] Forecasting: Principles and Practice (3^rd ed) Rob J Hyndman and George Athanasopoulos. https://otexts.com/fpp3/prediction-intervals.html

.. [2] MAPIE - Model Agnostic Prediction Interval Estimator. https://mapie.readthedocs.io/en/stable/theoretical_description_regression.html#the-split-method

Source code in skforecast\model_selection\_validation.py

def backtesting_forecaster(
    forecaster: object,
    y: pd.Series,
    cv: TimeSeriesFold,
    metric: str | Callable | list[str | Callable],
    exog: pd.Series | pd.DataFrame | None = None,
    interval: float | list[float] | tuple[float] | str | object | None = None,
    interval_method: str = 'bootstrapping',
    n_boot: int = 250,
    use_in_sample_residuals: bool = True,
    use_binned_residuals: bool = True,
    random_state: int = 123,
    return_predictors: bool = False,
    n_jobs: int | str = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False
) -> tuple[pd.DataFrame, pd.DataFrame]:
    """
    Backtesting of forecaster model following the folds generated by the TimeSeriesFold
    class and using the metric(s) provided.

    If `forecaster` is already trained and `initial_train_size` is set to `None` in the
    TimeSeriesFold class, no initial train will be done and all data will be used
    to evaluate the model. However, the first `len(forecaster.last_window)` observations
    are needed to create the initial predictors, so no predictions are calculated for
    them.

    A copy of the original forecaster is created so that it is not modified during 
    the process.

    Parameters
    ----------
    forecaster : ForecasterRecursive, ForecasterDirect, ForecasterEquivalentDate, ForecasterRecursiveClassifier
        Forecaster model.
    y : pandas Series
        Training time series.
    cv : TimeSeriesFold
        TimeSeriesFold object with the information needed to split the data into folds.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    exog : pandas Series, pandas DataFrame, default None
        Exogenous variable/s included as predictor/s. Must have the same
        number of observations as `y` and should be aligned so that y[i] is
        regressed on exog[i].
    interval : float, list, tuple, str, object, default None
        Specifies whether probabilistic predictions should be estimated and the 
        method to use. The following options are supported:

        - If `float`, represents the nominal (expected) coverage (between 0 and 1). 
        For instance, `interval=0.95` corresponds to `[2.5, 97.5]` percentiles.
        - If `list` or `tuple`: Sequence of percentiles to compute, each value must 
        be between 0 and 100 inclusive. For example, a 95% confidence interval can 
        be specified as `interval = [2.5, 97.5]` or multiple percentiles (e.g. 10, 
        50 and 90) as `interval = [10, 50, 90]`.
        - If 'bootstrapping' (str): `n_boot` bootstrapping predictions will be generated.
        - If scipy.stats distribution object, the distribution parameters will
        be estimated for each prediction.
        - If None, no probabilistic predictions are estimated.
    interval_method : str, default 'bootstrapping'
        Technique used to estimate prediction intervals. Available options:

        - 'bootstrapping': Bootstrapping is used to generate prediction 
        intervals [1]_.
        - 'conformal': Employs the conformal prediction split method for 
        interval estimation [2]_.
    n_boot : int, default 250
        Number of bootstrapping iterations to perform when estimating prediction
        intervals.
    use_in_sample_residuals : bool, default True
        If `True`, residuals from the training data are used as proxy of
        prediction error to create predictions. 
        If `False`, out of sample residuals (calibration) are used. 
        Out-of-sample residuals must be precomputed using Forecaster's
        `set_out_sample_residuals()` method.
    use_binned_residuals : bool, default True
        If `True`, residuals are selected based on the predicted values 
        (binned selection).
        If `False`, residuals are selected randomly.
    random_state : int, default 123
        Seed for the random number generator to ensure reproducibility.
    return_predictors : bool, default False
        If `True`, the predictors used to make the predictions are also returned.
    n_jobs : int, 'auto', default 'auto'
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting.
    verbose : bool, default False
        Print number of folds and index of training and validation sets used 
        for backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings : bool, default False
        If `True`, skforecast warnings will be suppressed during the backtesting 
        process. See skforecast.exceptions.warn_skforecast_categories for more
        information.

    Returns
    -------
    metric_values : pandas DataFrame
        Value(s) of the metric(s).
    backtest_predictions : pandas DataFrame
        Value of predictions. The DataFrame includes the following columns:

        - fold: Indicates the fold number where the prediction was made.
        - pred: Predicted values for the corresponding series and time steps.

        If `interval` is not `None`, additional columns are included depending on the method:

        - For `float`: Columns `lower_bound` and `upper_bound`.
        - For `list` or `tuple` of 2 elements: Columns `lower_bound` and `upper_bound`.
        - For `list` or `tuple` with multiple percentiles: One column per percentile 
        (e.g., `p_10`, `p_50`, `p_90`).
        - For `'bootstrapping'`: One column per bootstrapping iteration 
        (e.g., `pred_boot_0`, `pred_boot_1`, ..., `pred_boot_n`).
        - For `scipy.stats` distribution objects: One column for each estimated 
        parameter of the distribution (e.g., `loc`, `scale`).

        If `return_predictors` is `True`, one column per predictor is created.

        Depending on the relation between `steps` and `fold_stride`, the output
        may include repeated indexes (if `fold_stride < steps`) or gaps
        (if `fold_stride > steps`). See Notes below for more details.

    Notes
    -----
    Note on `fold_stride` vs. `steps`:

    - If `fold_stride == steps`, test sets are placed back-to-back without overlap. 
    Each observation appears only once in the output DataFrame, so the index is unique.
    - If `fold_stride < steps`, test sets overlap. Multiple forecasts are generated 
    for the same observations and, therefore, the output DataFrame contains repeated 
    indexes.
    - If `fold_stride > steps`, there are gaps between consecutive test sets. 
    Some observations in the series will not have associated predictions, so 
    the output DataFrame has non-contiguous indexes.

    References
    ----------
    .. [1] Forecasting: Principles and Practice (3rd ed) Rob J Hyndman and George Athanasopoulos.
           https://otexts.com/fpp3/prediction-intervals.html

    .. [2] MAPIE - Model Agnostic Prediction Interval Estimator.
           https://mapie.readthedocs.io/en/stable/theoretical_description_regression.html#the-split-method

    """

    forecasters_allowed = [
        'ForecasterRecursive', 
        'ForecasterDirect',
        'ForecasterEquivalentDate',
        'ForecasterRecursiveClassifier'
    ]

    if type(forecaster).__name__ not in forecasters_allowed:
        raise TypeError(
            f"`forecaster` must be of type {forecasters_allowed}. For all other "
            f"types of forecasters use the other functions available in the "
            f"`model_selection` module."
        )

    check_backtesting_input(
        forecaster              = forecaster,
        cv                      = cv,
        y                       = y,
        metric                  = metric,
        interval                = interval,
        interval_method         = interval_method,
        n_boot                  = n_boot,
        use_in_sample_residuals = use_in_sample_residuals,
        use_binned_residuals    = use_binned_residuals,
        random_state            = random_state,
        return_predictors       = return_predictors,
        n_jobs                  = n_jobs,
        show_progress           = show_progress,
        suppress_warnings       = suppress_warnings
    )

    metric_values, backtest_predictions = _backtesting_forecaster(
        forecaster              = forecaster,
        y                       = y,
        cv                      = cv,
        metric                  = metric,
        exog                    = exog,
        interval                = interval,
        interval_method         = interval_method,
        n_boot                  = n_boot,
        use_in_sample_residuals = use_in_sample_residuals,
        use_binned_residuals    = use_binned_residuals,
        random_state            = random_state,
        return_predictors       = return_predictors,
        n_jobs                  = n_jobs,
        verbose                 = verbose,
        show_progress           = show_progress,
        suppress_warnings       = suppress_warnings
    )

    return metric_values, backtest_predictions

skforecast.model_selection._search.grid_search_forecaster

grid_search_forecaster(
    forecaster,
    y,
    cv,
    param_grid,
    metric,
    exog=None,
    lags_grid=None,
    return_best=True,
    n_jobs="auto",
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
    output_file=None,
)

Exhaustive search over specified parameter values for a Forecaster object. Validation is done using time series backtesting.

Parameters:

Name	Type	Description	Default
forecaster	(ForecasterRecursive, ForecasterDirect)	Forecaster model.	required
y	pandas Series	Training time series.	required
cv	(TimeSeriesFold, OneStepAheadFold)	TimeSeriesFold or OneStepAheadFold object with the information needed to split the data into folds.	required
param_grid	dict	Dictionary with parameters names (str) as keys and lists of parameter settings to try as values.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'} If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. If list: List containing multiple strings and/or Callables.	required
exog	pandas Series, pandas DataFrame	Exogenous variable/s included as predictor/s. Must have the same number of observations as y and should be aligned so that y[i] is regressed on exog[i].	None
lags_grid	(list, dict)	Lists of lags to try, containing int, lists, numpy ndarray, or range objects. If dict, the keys are used as labels in the results DataFrame, and the values are used as the lists of lags to try.	None
return_best	bool	Refit the forecaster using the best found parameters on the whole data.	True
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting. Ignored for OneStepAheadFold.	'auto'
verbose	bool	Print number of folds used for cv or backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information.	False
output_file	str	Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file.	None

Returns:

Name	Type	Description
results	pandas DataFrame	Results for each combination of parameters. column lags: lags configuration for each iteration. column lags_label: descriptive label or alias for the lags. column params: parameters configuration for each iteration. column metric: metric value estimated for each iteration. additional n columns with param = value.

Source code in skforecast\model_selection\_search.py

def grid_search_forecaster(
    forecaster: object,
    y: pd.Series,
    cv: TimeSeriesFold | OneStepAheadFold,
    param_grid: dict,
    metric: str | Callable | list[str | Callable],
    exog: pd.Series | pd.DataFrame | None = None,
    lags_grid: (
        list[int | list[int] | np.ndarray[int] | range[int]]
        | dict[str, list[int | list[int] | np.ndarray[int] | range[int]]]
        | None
    ) = None,
    return_best: bool = True,
    n_jobs: int | str = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: str | None = None
) -> pd.DataFrame:
    """
    Exhaustive search over specified parameter values for a Forecaster object.
    Validation is done using time series backtesting.

    Parameters
    ----------
    forecaster : ForecasterRecursive, ForecasterDirect
        Forecaster model.
    y : pandas Series
        Training time series.
    cv : TimeSeriesFold, OneStepAheadFold
        TimeSeriesFold or OneStepAheadFold object with the information needed to split
        the data into folds.
    param_grid : dict
        Dictionary with parameters names (`str`) as keys and lists of parameter
        settings to try as values.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    exog : pandas Series, pandas DataFrame, default None
        Exogenous variable/s included as predictor/s. Must have the same
        number of observations as `y` and should be aligned so that y[i] is
        regressed on exog[i].
    lags_grid : list, dict, default None
        Lists of lags to try, containing int, lists, numpy ndarray, or range 
        objects. If `dict`, the keys are used as labels in the `results` 
        DataFrame, and the values are used as the lists of lags to try.
    return_best : bool, default True
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default 'auto'
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting. Ignored for `OneStepAheadFold`.
    verbose : bool, default False
        Print number of folds used for cv or backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings : bool, default False
        If `True`, skforecast warnings will be suppressed during the hyperparameter 
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.
    output_file : str, default None
        Specifies the filename or full path where the results should be saved. 
        The results will be saved in a tab-separated values (TSV) format. If 
        `None`, the results will not be saved to a file.

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column lags: lags configuration for each iteration.
        - column lags_label: descriptive label or alias for the lags.
        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration.
        - additional n columns with param = value.

    """

    param_grid = list(ParameterGrid(param_grid))

    results = _evaluate_grid_hyperparameters(
                  forecaster        = forecaster,
                  y                 = y,
                  cv                = cv,
                  param_grid        = param_grid,
                  metric            = metric,
                  exog              = exog,
                  lags_grid         = lags_grid,
                  return_best       = return_best,
                  n_jobs            = n_jobs,
                  verbose           = verbose,
                  show_progress     = show_progress,
                  suppress_warnings = suppress_warnings,
                  output_file       = output_file
              )

    return results

skforecast.model_selection._search.random_search_forecaster

random_search_forecaster(
    forecaster,
    y,
    cv,
    param_distributions,
    metric,
    exog=None,
    lags_grid=None,
    n_iter=10,
    random_state=123,
    return_best=True,
    n_jobs="auto",
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
    output_file=None,
)

Random search over specified parameter values or distributions for a Forecaster object. Validation is done using time series backtesting.

Parameters:

Name	Type	Description	Default
forecaster	(ForecasterRecursive, ForecasterDirect)	Forecaster model.	required
y	pandas Series	Training time series.	required
cv	(TimeSeriesFold, OneStepAheadFold)	TimeSeriesFold or OneStepAheadFold object with the information needed to split the data into folds.	required
param_distributions	dict	Dictionary with parameters names (str) as keys and distributions or lists of parameters to try.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'} If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. If list: List containing multiple strings and/or Callables.	required
exog	pandas Series, pandas DataFrame	Exogenous variable/s included as predictor/s. Must have the same number of observations as y and should be aligned so that y[i] is regressed on exog[i].	None
lags_grid	(list, dict)	Lists of lags to try, containing int, lists, numpy ndarray, or range objects. If dict, the keys are used as labels in the results DataFrame, and the values are used as the lists of lags to try.	None
n_iter	int	Number of parameter settings that are sampled per lags configuration. n_iter trades off runtime vs quality of the solution.	10
random_state	int	Sets a seed to the random sampling for reproducible output.	123
return_best	bool	Refit the forecaster using the best found parameters on the whole data.	True
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting. Ignored for OneStepAheadFold.	'auto'
verbose	bool	Print number of folds used for cv or backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information.	False
output_file	str	Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file.	None

Returns:

Name	Type	Description
results	pandas DataFrame	Results for each combination of parameters. column lags: lags configuration for each iteration. column lags_label: descriptive label or alias for the lags. column params: parameters configuration for each iteration. column metric: metric value estimated for each iteration. additional n columns with param = value.

Source code in skforecast\model_selection\_search.py

def random_search_forecaster(
    forecaster: object,
    y: pd.Series,
    cv: TimeSeriesFold | OneStepAheadFold,
    param_distributions: dict,
    metric: str | Callable | list[str | Callable],
    exog: pd.Series | pd.DataFrame | None = None,
    lags_grid: (
        list[int | list[int] | np.ndarray[int] | range[int]]
        | dict[str, list[int | list[int] | np.ndarray[int] | range[int]]]
        | None
    ) = None,
    n_iter: int = 10,
    random_state: int = 123,
    return_best: bool = True,
    n_jobs: int | str = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: str | None = None
) -> pd.DataFrame:
    """
    Random search over specified parameter values or distributions for a Forecaster 
    object. Validation is done using time series backtesting.

    Parameters
    ----------
    forecaster : ForecasterRecursive, ForecasterDirect
        Forecaster model.
    y : pandas Series
        Training time series.
    cv : TimeSeriesFold, OneStepAheadFold
        TimeSeriesFold or OneStepAheadFold object with the information needed to split
        the data into folds.
    param_distributions : dict
        Dictionary with parameters names (`str`) as keys and 
        distributions or lists of parameters to try.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    exog : pandas Series, pandas DataFrame, default None
        Exogenous variable/s included as predictor/s. Must have the same
        number of observations as `y` and should be aligned so that y[i] is
        regressed on exog[i]. 
    lags_grid : list, dict, default None
        Lists of lags to try, containing int, lists, numpy ndarray, or range 
        objects. If `dict`, the keys are used as labels in the `results` 
        DataFrame, and the values are used as the lists of lags to try.
    n_iter : int, default 10
        Number of parameter settings that are sampled per lags configuration. 
        n_iter trades off runtime vs quality of the solution.
    random_state : int, default 123
        Sets a seed to the random sampling for reproducible output.
    return_best : bool, default True
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default 'auto'
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting. Ignored for `OneStepAheadFold`.
    verbose : bool, default False
        Print number of folds used for cv or backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings : bool, default False
        If `True`, skforecast warnings will be suppressed during the hyperparameter 
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.
    output_file : str, default None
        Specifies the filename or full path where the results should be saved. 
        The results will be saved in a tab-separated values (TSV) format. If 
        `None`, the results will not be saved to a file.

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column lags: lags configuration for each iteration.
        - column lags_label: descriptive label or alias for the lags.
        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration.
        - additional n columns with param = value.

    """

    param_grid = list(ParameterSampler(param_distributions, n_iter=n_iter, random_state=random_state))

    results = _evaluate_grid_hyperparameters(
                  forecaster        = forecaster,
                  y                 = y,
                  cv                = cv,
                  param_grid        = param_grid,
                  metric            = metric,
                  exog              = exog,
                  lags_grid         = lags_grid,
                  return_best       = return_best,
                  n_jobs            = n_jobs,
                  verbose           = verbose,
                  show_progress     = show_progress,
                  suppress_warnings = suppress_warnings,
                  output_file       = output_file
              )

    return results

skforecast.model_selection._search.bayesian_search_forecaster

bayesian_search_forecaster(
    forecaster,
    y,
    cv,
    search_space,
    metric,
    exog=None,
    n_trials=20,
    random_state=123,
    return_best=True,
    n_jobs="auto",
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
    output_file=None,
    kwargs_create_study=None,
    kwargs_study_optimize=None,
)

Bayesian search for hyperparameters of a Forecaster object using optuna library.

Parameters:

Name	Type	Description	Default
forecaster	(ForecasterRecursive, ForecasterDirect)	Forecaster model.	required
y	pandas Series	Training time series.	required
cv	(TimeSeriesFold, OneStepAheadFold)	TimeSeriesFold or OneStepAheadFold object with the information needed to split the data into folds.	required
search_space	Callable(optuna)	Function with argument trial which returns a dictionary with parameters names (str) as keys and Trial object from optuna (trial.suggest_float, trial.suggest_int, trial.suggest_categorical) as values.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'} If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. If list: List containing multiple strings and/or Callables.	required
exog	pandas Series, pandas DataFrame	Exogenous variable/s included as predictor/s. Must have the same number of observations as y and should be aligned so that y[i] is regressed on exog[i].	None
n_trials	int	Number of parameter settings that are sampled in each lag configuration. The first 10 trials are random (controlled by optuna's n_startup_trials); the TPE sampler only guides the search from trial 11 onward. For meaningful Bayesian optimization, n_trials should be significantly larger than 10.	20
random_state	int	Sets a seed to the sampling for reproducible output. When a new sampler is passed in kwargs_create_study, the seed must be set within the sampler. For example {'sampler': TPESampler(seed=145)}.	123
return_best	bool	Refit the forecaster using the best found parameters on the whole data.	True
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting. Ignored for OneStepAheadFold.	'auto'
verbose	bool	Print number of folds used for cv or backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information.	False
output_file	str	Specifies the filename or full path where the optuna logging output should be saved. If None, logging output will not be redirected to a file.	None
kwargs_create_study	dict	Additional keyword arguments (key, value mappings) to pass to optuna.create_study(). If default, the direction is set to 'minimize' for regression tasks or 'maximize' for classification tasks, and a TPESampler(multivariate=True, group=True, consider_endpoints=True, seed=random_state) sampler is used during optimization.	None
kwargs_study_optimize	dict	Additional keyword arguments (key, value mappings) to pass to study.optimize().	None

Returns:

Name	Type	Description
results	pandas DataFrame	Results for each combination of parameters. column trial_number: optuna trial number for each iteration. Use study.trials[trial_number] to access the full optuna trial object. column lags: lags configuration for each iteration. column params: parameters configuration for each iteration. column metric: metric value estimated for each iteration. additional n columns with param = value.
study	optuna Study	The optuna study object containing all optimization trials. Access the best trial via study.best_trial.

Source code in skforecast\model_selection\_search.py

@manage_warnings
def bayesian_search_forecaster(
    forecaster: object,
    y: pd.Series,
    cv: TimeSeriesFold | OneStepAheadFold,
    search_space: Callable,
    metric: str | Callable | list[str | Callable],
    exog: pd.Series | pd.DataFrame | None = None,
    n_trials: int = 20,
    random_state: int = 123,
    return_best: bool = True,
    n_jobs: int | str = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: str | None = None,
    kwargs_create_study: dict | None = None,
    kwargs_study_optimize: dict | None = None
) -> tuple[pd.DataFrame, object]:
    """
    Bayesian search for hyperparameters of a Forecaster object using optuna library.

    Parameters
    ----------
    forecaster : ForecasterRecursive, ForecasterDirect
        Forecaster model.
    y : pandas Series
        Training time series.
    cv : TimeSeriesFold, OneStepAheadFold
        TimeSeriesFold or OneStepAheadFold object with the information needed to split
        the data into folds.
    search_space : Callable (optuna)
        Function with argument `trial` which returns a dictionary with parameters names 
        (`str`) as keys and Trial object from optuna (trial.suggest_float, 
        trial.suggest_int, trial.suggest_categorical) as values.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    exog : pandas Series, pandas DataFrame, default None
        Exogenous variable/s included as predictor/s. Must have the same
        number of observations as `y` and should be aligned so that y[i] is
        regressed on exog[i].
    n_trials : int, default 20
        Number of parameter settings that are sampled in each lag configuration. 
        The first 10 trials are random (controlled by optuna's `n_startup_trials`); 
        the TPE sampler only guides the search from trial 11 onward. For meaningful 
        Bayesian optimization, `n_trials` should be significantly larger than 10.
    random_state : int, default 123
        Sets a seed to the sampling for reproducible output. When a new sampler 
        is passed in `kwargs_create_study`, the seed must be set within the 
        sampler. For example `{'sampler': TPESampler(seed=145)}`.
    return_best : bool, default True
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default 'auto'
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting. Ignored for `OneStepAheadFold`.
    verbose : bool, default False
        Print number of folds used for cv or backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings : bool, default False
        If `True`, skforecast warnings will be suppressed during the hyperparameter 
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.
    output_file : str, default None
        Specifies the filename or full path where the optuna logging output
        should be saved. If `None`, logging output will not be redirected to
        a file.
    kwargs_create_study : dict, default None
        Additional keyword arguments (key, value mappings) to pass to optuna.create_study().
        If default, the direction is set to 'minimize' for regression tasks or
        'maximize' for classification tasks, and a 
        `TPESampler(multivariate=True, group=True, consider_endpoints=True, seed=random_state)` 
        sampler is used during optimization.
    kwargs_study_optimize : dict, default None
        Additional keyword arguments (key, value mappings) to pass to study.optimize().

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column trial_number: optuna trial number for each iteration. Use
          `study.trials[trial_number]` to access the full optuna trial object.
        - column lags: lags configuration for each iteration.
        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration.
        - additional n columns with param = value.
    study : optuna Study
        The optuna study object containing all optimization trials. Access the
        best trial via `study.best_trial`.

    """

    if return_best and exog is not None and (len(exog) != len(y)):
        raise ValueError(
            f"`exog` must have same number of samples as `y`. "
            f"length `exog`: ({len(exog)}), length `y`: ({len(y)})"
        )

    forecaster_search = deepcopy_forecaster(forecaster)
    forecaster_name = type(forecaster_search).__name__
    is_regression = forecaster_search.__skforecast_tags__['forecaster_task'] == 'regression'
    cv_name = type(cv).__name__

    if cv_name not in ['TimeSeriesFold', 'OneStepAheadFold']:
        raise TypeError(
            f"`cv` must be an instance of `TimeSeriesFold` or `OneStepAheadFold`. "
            f"Got {type(cv)}."
        )

    if cv_name == 'OneStepAheadFold':

        check_one_step_ahead_input(
            forecaster        = forecaster_search,
            cv                = cv,
            metric            = metric,
            y                 = y,
            exog              = exog,
            show_progress     = show_progress,
            suppress_warnings = suppress_warnings
        )

        cv = deepcopy(cv)
        initial_train_size = date_to_index_position(
                                 index        = cv._extract_index(y), 
                                 date_input   = cv.initial_train_size, 
                                 method       = 'validation',
                                 date_literal = 'initial_train_size'
                             )
        cv.set_params({
            'initial_train_size': initial_train_size,
            'window_size': forecaster_search.window_size,
            'differentiation': forecaster_search.differentiation_max,
            'verbose': verbose
        })

    if not isinstance(metric, list):
        metric = [metric]
    metric = [
        _get_metric(metric=m)
        if isinstance(m, str)
        else add_y_train_argument(m) 
        for m in metric
    ]
    metric_dict = {
        (m if isinstance(m, str) else m.__name__): [] 
        for m in metric
    }

    if len(metric_dict) != len(metric):
        raise ValueError(
            "When `metric` is a `list`, each metric name must be unique."
        )

    # Objective function using backtesting_forecaster
    if cv_name == 'TimeSeriesFold':

        def _objective(
            trial,
            search_space      = search_space,
            forecaster_search = forecaster_search,
            y                 = y,
            cv                = cv,
            exog              = exog,
            metric            = metric,
            n_jobs            = n_jobs,
            verbose           = verbose,
            suppress_warnings = suppress_warnings,
        ) -> float:

            sample = search_space(trial)
            if sample.keys() != trial.params.keys():
                raise ValueError(
                    f"`search_space` dict keys must match the names passed to "
                    f"`trial.suggest_*()`.\n"
                    f"  Dict keys    : {list(sample.keys())}\n"
                    f"  Suggest names: {list(trial.params.keys())}"
                )
            sample_params = {k: v for k, v in sample.items() if k != 'lags'}
            forecaster_search.set_params(sample_params)
            if "lags" in sample:
                forecaster_search.set_lags(sample['lags'])

            metrics, _ = backtesting_forecaster(
                             forecaster        = forecaster_search,
                             y                 = y,
                             cv                = cv,
                             exog              = exog,
                             metric            = metric,
                             n_jobs            = n_jobs,
                             verbose           = verbose,
                             show_progress     = False,
                             suppress_warnings = suppress_warnings
                         )
            metrics = metrics.iloc[0, :].to_list()

            # Store all metrics in the trial using optuna's user_attrs mechanism.
            for m_name, m_val in zip(metric_dict, metrics):
                trial.set_user_attr(m_name, float(m_val))

            return metrics[0]

    else:

        _SENTINEL = object()
        _MAX_CACHE_SIZE = 10
        _cached_split = {}

        def _objective(
            trial,
            search_space      = search_space,
            forecaster_search = forecaster_search,
            y                 = y,
            cv                = cv,
            exog              = exog,
            metric            = metric
        ) -> float:

            sample = search_space(trial)
            if sample.keys() != trial.params.keys():
                raise ValueError(
                    f"`search_space` dict keys must match the names passed to "
                    f"`trial.suggest_*()`.\n"
                    f"  Dict keys    : {list(sample.keys())}\n"
                    f"  Suggest names: {list(trial.params.keys())}"
                )
            sample_params = {k: v for k, v in sample.items() if k != 'lags'}
            forecaster_search.set_params(sample_params)

            current_lags = sample.get('lags', _SENTINEL)
            if current_lags is not _SENTINEL:
                forecaster_search.set_lags(current_lags)

            lags_key = _make_lags_hashable(current_lags, sentinel=_SENTINEL)
            if lags_key not in _cached_split:

                if len(_cached_split) >= _MAX_CACHE_SIZE:
                    _cached_split.pop(next(iter(_cached_split)))

                (
                    X_train,
                    y_train,
                    X_test,
                    y_test
                ) = forecaster_search._train_test_split_one_step_ahead(
                    y=y, initial_train_size=cv.initial_train_size, exog=exog
                )
                _cached_split[lags_key] = (X_train, y_train, X_test, y_test)
            else:
                X_train, y_train, X_test, y_test = _cached_split[lags_key]

            metrics = _calculate_metrics_one_step_ahead(
                          forecaster = forecaster_search,
                          metrics    = metric,
                          X_train    = X_train,
                          y_train    = y_train,
                          X_test     = X_test,
                          y_test     = y_test
                      )

            # Store all metrics in the trial using optuna's user_attrs mechanism.
            for m_name, m_val in zip(metric_dict, metrics):
                trial.set_user_attr(m_name, float(m_val))

            return metrics[0]

    kwargs_create_study = kwargs_create_study.copy() if kwargs_create_study is not None else {}
    if 'direction' not in kwargs_create_study.keys():
        kwargs_create_study['direction'] = 'minimize' if is_regression else 'maximize'
    if 'sampler' not in kwargs_create_study:
        with warnings.catch_warnings():
            warnings.filterwarnings(
                'ignore',
                message='.*multivariate.*|.*group.*',
                module='optuna'
            )
            kwargs_create_study['sampler'] = TPESampler(
                multivariate=True, group=True, consider_endpoints=True, seed=random_state
            )

    kwargs_study_optimize = kwargs_study_optimize.copy() if kwargs_study_optimize is not None else {}
    if show_progress:
        kwargs_study_optimize['show_progress_bar'] = True
    else:
        kwargs_study_optimize.setdefault('show_progress_bar', False)

    if output_file is not None:
        # Redirect optuna logging to file
        optuna.logging.disable_default_handler()
        logger = logging.getLogger('optuna')
        logger.setLevel(logging.INFO)
        for handler in logger.handlers.copy():
            if isinstance(handler, logging.StreamHandler):
                logger.removeHandler(handler)
        handler = logging.FileHandler(output_file, mode="w")
        logger.addHandler(handler)
    else:
        logging.getLogger("optuna").setLevel(logging.WARNING)
        optuna.logging.disable_default_handler()

    study = optuna.create_study(**kwargs_create_study)

    try:
        with warnings.catch_warnings():
            warnings.filterwarnings(
                "ignore",
                category = UserWarning,
                message  = "Choices for a categorical distribution should be*"
            )
            study.optimize(_objective, n_trials=n_trials, **kwargs_study_optimize)
    finally:
        if output_file is not None:
            handler.close()
            logger.removeHandler(handler)

    lags_list = []
    params_list = []
    trial_number_list = []
    for trial in study.get_trials(states=[TrialState.COMPLETE]):
        estimator_params = {k: v for k, v in trial.params.items() if k != 'lags'}
        lags = trial.params.get(
            'lags',
            forecaster_search.lags if hasattr(forecaster_search, 'lags') else None
        )
        params_list.append(estimator_params)
        lags_list.append(initialize_lags(forecaster_name=forecaster_name, lags=lags)[0])
        trial_number_list.append(trial.number)
        for m_name in metric_dict:
            metric_dict[m_name].append(trial.user_attrs[m_name])

    results = pd.DataFrame({
                  'trial_number': trial_number_list,
                  'lags': lags_list,
                  'params': params_list,
                  **metric_dict
              })

    results = (
        results
        .sort_values(by=list(metric_dict.keys())[0], ascending=True if is_regression else False)
        .reset_index(drop=True)
    )
    results = pd.concat([results, results['params'].apply(pd.Series)], axis=1)

    if return_best:

        best_lags = results.loc[0, 'lags']
        best_params = results.loc[0, 'params']
        best_metric = results.loc[0, list(metric_dict.keys())[0]]

        # NOTE: Here we use the actual forecaster passed by the user
        forecaster.set_lags(best_lags)
        forecaster.set_params(best_params)

        forecaster.fit(
            y                         = y,
            exog                      = exog,
            store_in_sample_residuals = True,
            suppress_warnings         = suppress_warnings
        )

        if verbose:
            print(
                f"`Forecaster` refitted using the best-found lags and parameters, "
                f"and the whole data set: \n"
                f"  Lags: {best_lags} \n"
                f"  Parameters: {best_params}\n"
                f"  {'Backtesting' if cv_name == 'TimeSeriesFold' else 'One-step-ahead'} "
                f"metric: {best_metric}"
            )

    return results, study

skforecast.model_selection._validation.backtesting_forecaster_multiseries

backtesting_forecaster_multiseries(
    forecaster,
    series,
    cv,
    metric,
    levels=None,
    add_aggregated_metric=True,
    exog=None,
    interval=None,
    interval_method="conformal",
    n_boot=250,
    use_in_sample_residuals=True,
    use_binned_residuals=True,
    random_state=123,
    return_predictors=False,
    n_jobs="auto",
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
)

Backtesting of forecaster model following the folds generated by the TimeSeriesFold class and using the metric(s) provided.

If forecaster is already trained and initial_train_size is set to None in the TimeSeriesFold class, no initial train will be done and all data will be used to evaluate the model. However, the first len(forecaster.last_window) observations are needed to create the initial predictors, so no predictions are calculated for them.

A copy of the original forecaster is created so that it is not modified during the process.

Parameters:

Name	Type	Description	Default
forecaster	(ForecasterRecursiveMultiSeries, ForecasterDirectMultiVariate, ForecasterRnn)	Forecaster model.	required
series	pandas DataFrame, dict	Training time series.	required
cv	TimeSeriesFold	TimeSeriesFold object with the information needed to split the data into folds.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'} If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. If list: List containing multiple strings and/or Callables.	required
levels	(str, list)	Time series to be predicted. If None all levels will be predicted.	None
add_aggregated_metric	bool	If True, and multiple series (levels) are predicted, the aggregated metrics (average, weighted average and pooled) are also returned. 'average': the average (arithmetic mean) of all levels. 'weighted_average': the average of the metrics weighted by the number of predicted values of each level. 'pooling': the values of all levels are pooled and then the metric is calculated.	True
exog	pandas Series, pandas DataFrame, dict	Exogenous variables.	None
interval	(float, list, tuple, str, object)	Specifies whether probabilistic predictions should be estimated and the method to use. The following options are supported: If float, represents the nominal (expected) coverage (between 0 and 1). For instance, interval=0.95 corresponds to [2.5, 97.5] percentiles. If list or tuple: Sequence of percentiles to compute, each value must be between 0 and 100 inclusive. For example, a 95% confidence interval can be specified as interval = [2.5, 97.5] or multiple percentiles (e.g. 10, 50 and 90) as interval = [10, 50, 90]. If 'bootstrapping' (str): n_boot bootstrapping predictions will be generated. If scipy.stats distribution object, the distribution parameters will be estimated for each prediction. If None, no probabilistic predictions are estimated.	None
interval_method	str	Technique used to estimate prediction intervals. Available options: 'bootstrapping': Bootstrapping is used to generate prediction intervals [1]_. 'conformal': Employs the conformal prediction split method for interval estimation [2]_.	'conformal'
n_boot	int	Number of bootstrapping iterations to perform when estimating prediction intervals.	250
use_in_sample_residuals	bool	If True, residuals from the training data are used as proxy of prediction error to create predictions. If False, out of sample residuals (calibration) are used. Out-of-sample residuals must be precomputed using Forecaster's set_out_sample_residuals() method.	True
use_binned_residuals	bool	If True, residuals are selected based on the predicted values (binned selection). If False, residuals are selected randomly.	True
random_state	int	Seed for the random number generator to ensure reproducibility.	123
return_predictors	bool	If True, the predictors used to make the predictions are also returned.	False
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting.	'auto'
verbose	bool	Print number of folds and index of training and validation sets used for backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the backtesting process. See skforecast.exceptions.warn_skforecast_categories for more information.	False

Returns:

Name	Type	Description
metrics_levels	pandas DataFrame	Value(s) of the metric(s). Index are the levels and columns the metrics.
backtest_predictions	pandas DataFrame	Long-format DataFrame containing the predicted values for each series. The DataFrame includes the following columns: level: Identifier for the time series or level being predicted. fold: Indicates the fold number where the prediction was made. pred: Predicted values for the corresponding series and time steps. If interval is not None, additional columns are included depending on the method: For float: Columns lower_bound and upper_bound. For list or tuple of 2 elements: Columns lower_bound and upper_bound. For list or tuple with multiple percentiles: One column per percentile (e.g., p_10, p_50, p_90). For 'bootstrapping': One column per bootstrapping iteration (e.g., pred_boot_0, pred_boot_1, ..., pred_boot_n). For scipy.stats distribution objects: One column for each estimated parameter of the distribution (e.g., loc, scale). If return_predictors is True, one column per predictor is created. Depending on the relation between steps and fold_stride, the output may include repeated indexes (if fold_stride < steps) or gaps (if fold_stride > steps). See Notes below for more details.

Notes

Note on fold_stride vs. steps:

• If fold_stride == steps, test sets are placed back-to-back without overlap. Each observation appears only once in the output DataFrame, so the index is unique.
• If fold_stride < steps, test sets overlap. Multiple forecasts are generated for the same observations and, therefore, the output DataFrame contains repeated indexes.
• If fold_stride > steps, there are gaps between consecutive test sets. Some observations in the series will not have associated predictions, so the output DataFrame has non-contiguous indexes.

References

.. [1] Forecasting: Principles and Practice (3^rd ed) Rob J Hyndman and George Athanasopoulos. https://otexts.com/fpp3/prediction-intervals.html

.. [2] MAPIE - Model Agnostic Prediction Interval Estimator. https://mapie.readthedocs.io/en/stable/theoretical_description_regression.html#the-split-method

Source code in skforecast\model_selection\_validation.py

@manage_warnings
def backtesting_forecaster_multiseries(
    forecaster: object,
    series: pd.DataFrame | dict[str, pd.Series | pd.DataFrame],
    cv: TimeSeriesFold,
    metric: str | Callable | list[str | Callable],
    levels: str | list[str] | None = None,
    add_aggregated_metric: bool = True,
    exog: pd.Series | pd.DataFrame | dict[str, pd.Series | pd.DataFrame] | None = None,
    interval: float | list[float] | tuple[float] | str | object | None = None,
    interval_method: str = 'conformal',
    n_boot: int = 250,
    use_in_sample_residuals: bool = True,
    use_binned_residuals: bool = True,
    random_state: int = 123,
    return_predictors: bool = False,
    n_jobs: int | str = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False
) -> tuple[pd.DataFrame, pd.DataFrame]:
    """
    Backtesting of forecaster model following the folds generated by the TimeSeriesFold
    class and using the metric(s) provided.

    If `forecaster` is already trained and `initial_train_size` is set to `None` in the
    TimeSeriesFold class, no initial train will be done and all data will be used
    to evaluate the model. However, the first `len(forecaster.last_window)` observations
    are needed to create the initial predictors, so no predictions are calculated for
    them.

    A copy of the original forecaster is created so that it is not modified during 
    the process.

    Parameters
    ----------
    forecaster : ForecasterRecursiveMultiSeries, ForecasterDirectMultiVariate, ForecasterRnn
        Forecaster model.
    series : pandas DataFrame, dict
        Training time series.
    cv : TimeSeriesFold
        TimeSeriesFold object with the information needed to split the data into folds.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    levels : str, list, default None
        Time series to be predicted. If `None` all levels will be predicted.
    add_aggregated_metric : bool, default True
        If `True`, and multiple series (`levels`) are predicted, the aggregated
        metrics (average, weighted average and pooled) are also returned.

        - 'average': the average (arithmetic mean) of all levels.
        - 'weighted_average': the average of the metrics weighted by the number of
        predicted values of each level.
        - 'pooling': the values of all levels are pooled and then the metric is
        calculated.
    exog : pandas Series, pandas DataFrame, dict, default None
        Exogenous variables.
    interval : float, list, tuple, str, object, default None
        Specifies whether probabilistic predictions should be estimated and the 
        method to use. The following options are supported:

        - If `float`, represents the nominal (expected) coverage (between 0 and 1). 
        For instance, `interval=0.95` corresponds to `[2.5, 97.5]` percentiles.
        - If `list` or `tuple`: Sequence of percentiles to compute, each value must 
        be between 0 and 100 inclusive. For example, a 95% confidence interval can 
        be specified as `interval = [2.5, 97.5]` or multiple percentiles (e.g. 10, 
        50 and 90) as `interval = [10, 50, 90]`.
        - If 'bootstrapping' (str): `n_boot` bootstrapping predictions will be generated.
        - If scipy.stats distribution object, the distribution parameters will
        be estimated for each prediction.
        - If None, no probabilistic predictions are estimated.
    interval_method : str, default 'conformal'
        Technique used to estimate prediction intervals. Available options:

        - 'bootstrapping': Bootstrapping is used to generate prediction 
        intervals [1]_.
        - 'conformal': Employs the conformal prediction split method for 
        interval estimation [2]_.
    n_boot : int, default 250
        Number of bootstrapping iterations to perform when estimating prediction 
        intervals.
    use_in_sample_residuals : bool, default True
        If `True`, residuals from the training data are used as proxy of
        prediction error to create predictions. 
        If `False`, out of sample residuals (calibration) are used. 
        Out-of-sample residuals must be precomputed using Forecaster's
        `set_out_sample_residuals()` method.
    use_binned_residuals : bool, default True
        If `True`, residuals are selected based on the predicted values 
        (binned selection).
        If `False`, residuals are selected randomly.
    random_state : int, default 123
        Seed for the random number generator to ensure reproducibility.
    return_predictors : bool, default False
        If `True`, the predictors used to make the predictions are also returned.
    n_jobs : int, 'auto', default 'auto'
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting.
    verbose : bool, default False
        Print number of folds and index of training and validation sets used 
        for backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings: bool, default False
        If `True`, skforecast warnings will be suppressed during the backtesting 
        process. See skforecast.exceptions.warn_skforecast_categories for more
        information.

    Returns
    -------
    metrics_levels : pandas DataFrame
        Value(s) of the metric(s). Index are the levels and columns the metrics.
    backtest_predictions : pandas DataFrame
        Long-format DataFrame containing the predicted values for each series. The 
        DataFrame includes the following columns:

        - `level`: Identifier for the time series or level being predicted.
        - fold: Indicates the fold number where the prediction was made.
        - `pred`: Predicted values for the corresponding series and time steps.

        If `interval` is not `None`, additional columns are included depending on the method:

        - For `float`: Columns `lower_bound` and `upper_bound`.
        - For `list` or `tuple` of 2 elements: Columns `lower_bound` and `upper_bound`.
        - For `list` or `tuple` with multiple percentiles: One column per percentile 
        (e.g., `p_10`, `p_50`, `p_90`).
        - For `'bootstrapping'`: One column per bootstrapping iteration 
        (e.g., `pred_boot_0`, `pred_boot_1`, ..., `pred_boot_n`).
        - For `scipy.stats` distribution objects: One column for each estimated 
        parameter of the distribution (e.g., `loc`, `scale`).

        If `return_predictors` is `True`, one column per predictor is created.

        Depending on the relation between `steps` and `fold_stride`, the output
        may include repeated indexes (if `fold_stride < steps`) or gaps
        (if `fold_stride > steps`). See Notes below for more details.

    Notes
    -----
    Note on `fold_stride` vs. `steps`:

    - If `fold_stride == steps`, test sets are placed back-to-back without overlap. 
    Each observation appears only once in the output DataFrame, so the index is unique.
    - If `fold_stride < steps`, test sets overlap. Multiple forecasts are generated 
    for the same observations and, therefore, the output DataFrame contains repeated 
    indexes.
    - If `fold_stride > steps`, there are gaps between consecutive test sets. 
    Some observations in the series will not have associated predictions, so 
    the output DataFrame has non-contiguous indexes.

    References
    ----------
    .. [1] Forecasting: Principles and Practice (3rd ed) Rob J Hyndman and George Athanasopoulos.
           https://otexts.com/fpp3/prediction-intervals.html

    .. [2] MAPIE - Model Agnostic Prediction Interval Estimator.
           https://mapie.readthedocs.io/en/stable/theoretical_description_regression.html#the-split-method

    """

    multi_series_forecasters = [
        'ForecasterRecursiveMultiSeries', 
        'ForecasterDirectMultiVariate',
        'ForecasterRnn'
    ]

    forecaster_name = type(forecaster).__name__

    if forecaster_name not in multi_series_forecasters:
        raise TypeError(
            f"`forecaster` must be of type {multi_series_forecasters}, "
            f"for all other types of forecasters use the functions available in "
            f"the `model_selection` module. Got {forecaster_name}"
        )

    if forecaster_name == 'ForecasterRecursiveMultiSeries':
        series, series_indexes = check_preprocess_series(series)
        if exog is not None:
            series_names_in_ = list(series.keys())
            exog_dict = {serie: None for serie in series_names_in_}
            exog, _ = check_preprocess_exog_multiseries(
                          series_names_in_  = series_names_in_,
                          series_index_type = type(series_indexes[series_names_in_[0]]),
                          exog              = exog,
                          exog_dict         = exog_dict
                      )

    check_backtesting_input(
        forecaster              = forecaster,
        cv                      = cv,
        metric                  = metric,
        add_aggregated_metric   = add_aggregated_metric,
        series                  = series,
        exog                    = exog,
        interval                = interval,
        interval_method         = interval_method,
        n_boot                  = n_boot,
        use_in_sample_residuals = use_in_sample_residuals,
        use_binned_residuals    = use_binned_residuals,
        random_state            = random_state,
        return_predictors       = return_predictors,
        n_jobs                  = n_jobs,
        show_progress           = show_progress,
        suppress_warnings       = suppress_warnings
    )

    metrics_levels, backtest_predictions = _backtesting_forecaster_multiseries(
        forecaster              = forecaster,
        series                  = series,
        cv                      = cv,
        levels                  = levels,
        metric                  = metric,
        add_aggregated_metric   = add_aggregated_metric,
        exog                    = exog,
        interval                = interval,
        interval_method         = interval_method,
        n_boot                  = n_boot,
        use_in_sample_residuals = use_in_sample_residuals,
        use_binned_residuals    = use_binned_residuals,
        random_state            = random_state,
        return_predictors       = return_predictors,
        n_jobs                  = n_jobs,
        verbose                 = verbose,
        show_progress           = show_progress,
        suppress_warnings       = suppress_warnings
    )

    return metrics_levels, backtest_predictions

skforecast.model_selection._search.grid_search_forecaster_multiseries

grid_search_forecaster_multiseries(
    forecaster,
    series,
    cv,
    param_grid,
    metric,
    aggregate_metric=None,
    levels=None,
    exog=None,
    lags_grid=None,
    return_best=True,
    n_jobs="auto",
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
    output_file=None,
)

Exhaustive search over specified parameter values for a Forecaster object. Validation is done using multi-series backtesting.

Parameters:

Name	Type	Description	Default
forecaster	(ForecasterRecursiveMultiSeries, ForecasterDirectMultiVariate)	Forecaster model.	required
series	pandas DataFrame, dict	Training time series.	required
cv	(TimeSeriesFold, OneStepAheadFold)	TimeSeriesFold or OneStepAheadFold object with the information needed to split the data into folds.	required
param_grid	dict	Dictionary with parameters names (str) as keys and lists of parameter settings to try as values.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'} If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. If list: List containing multiple strings and/or Callables.	required
aggregate_metric	(str, list)	Aggregation method/s used to combine the metric/s of all levels (series) when multiple levels are predicted. If list, the first aggregation method is used to select the best parameters. If None, ['weighted_average', 'average', 'pooling'] is used. 'average': the average (arithmetic mean) of all levels. 'weighted_average': the average of the metrics weighted by the number of predicted values of each level. 'pooling': the values of all levels are pooled and then the metric is calculated.	None
levels	(str, list)	level (str) or levels (list) at which the forecaster is optimized. If None, all levels are taken into account.	None
exog	pandas Series, pandas DataFrame, dict	Exogenous variables.	None
lags_grid	(list, dict)	Lists of lags to try, containing int, lists, numpy ndarray, or range objects. If dict, the keys are used as labels in the results DataFrame, and the values are used as the lists of lags to try.	None
return_best	bool	Refit the forecaster using the best found parameters on the whole data.	True
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting. Ignored for OneStepAheadFold.	'auto'
verbose	bool	Print number of folds used for cv or backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information.	False
output_file	str	Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file.	None

Returns:

Name	Type	Description
results	pandas DataFrame	Results for each combination of parameters. column levels: levels configuration for each iteration. column lags: lags configuration for each iteration. column lags_label: descriptive label or alias for the lags. column params: parameters configuration for each iteration. column metric: metric value estimated for each iteration. The resulting metric will be the average of the optimization of all levels. additional n columns with param = value.

Source code in skforecast\model_selection\_search.py

def grid_search_forecaster_multiseries(
    forecaster: object,
    series: pd.DataFrame | dict[str, pd.Series | pd.DataFrame],
    cv: TimeSeriesFold | OneStepAheadFold,
    param_grid: dict,
    metric: str | Callable | list[str | Callable],
    aggregate_metric: str | list[str] | None = None,
    levels: str | list[str] | None = None,
    exog: pd.Series | pd.DataFrame | dict[str, pd.Series | pd.DataFrame] | None = None,
    lags_grid: (
        list[int | list[int] | np.ndarray[int] | range[int]]
        | dict[str, list[int | list[int] | np.ndarray[int] | range[int]]]
        | None
    ) = None,
    return_best: bool = True,
    n_jobs: int | str = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: str | None = None
) -> pd.DataFrame:
    """
    Exhaustive search over specified parameter values for a Forecaster object.
    Validation is done using multi-series backtesting.

    Parameters
    ----------
    forecaster : ForecasterRecursiveMultiSeries, ForecasterDirectMultiVariate
        Forecaster model.
    series : pandas DataFrame, dict
        Training time series.
    cv : TimeSeriesFold, OneStepAheadFold
        TimeSeriesFold or OneStepAheadFold object with the information needed to split
        the data into folds.
    param_grid : dict
        Dictionary with parameters names (`str`) as keys and lists of parameter
        settings to try as values.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    aggregate_metric : str, list, default None
        Aggregation method/s used to combine the metric/s of all levels (series)
        when multiple levels are predicted. If list, the first aggregation method
        is used to select the best parameters. If `None`,
        `['weighted_average', 'average', 'pooling']` is used.

        - 'average': the average (arithmetic mean) of all levels.
        - 'weighted_average': the average of the metrics weighted by the number of
        predicted values of each level.
        - 'pooling': the values of all levels are pooled and then the metric is
        calculated.
    levels : str, list, default None
        level (`str`) or levels (`list`) at which the forecaster is optimized. 
        If `None`, all levels are taken into account.
    exog : pandas Series, pandas DataFrame, dict, default None
        Exogenous variables.
    lags_grid : list, dict, default None
        Lists of lags to try, containing int, lists, numpy ndarray, or range 
        objects. If `dict`, the keys are used as labels in the `results` 
        DataFrame, and the values are used as the lists of lags to try.
    return_best : bool, default True
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default 'auto'
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting. Ignored for `OneStepAheadFold`.
    verbose : bool, default False
        Print number of folds used for cv or backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings: bool, default False
        If `True`, skforecast warnings will be suppressed during the hyperparameter 
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.
    output_file : str, default None
        Specifies the filename or full path where the results should be saved. 
        The results will be saved in a tab-separated values (TSV) format. If 
        `None`, the results will not be saved to a file.

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column levels: levels configuration for each iteration.
        - column lags: lags configuration for each iteration.
        - column lags_label: descriptive label or alias for the lags.
        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration. The resulting 
        metric will be the average of the optimization of all levels.
        - additional n columns with param = value.

    """

    param_grid = list(ParameterGrid(param_grid))

    results = _evaluate_grid_hyperparameters_multiseries(
                  forecaster        = forecaster,
                  series            = series,
                  cv                = cv,
                  param_grid        = param_grid,
                  metric            = metric,
                  aggregate_metric  = aggregate_metric,
                  levels            = levels,
                  exog              = exog,
                  lags_grid         = lags_grid,
                  n_jobs            = n_jobs,
                  return_best       = return_best,
                  verbose           = verbose,
                  show_progress     = show_progress,
                  suppress_warnings = suppress_warnings,
                  output_file       = output_file
              )

    return results

skforecast.model_selection._search.random_search_forecaster_multiseries

random_search_forecaster_multiseries(
    forecaster,
    series,
    cv,
    param_distributions,
    metric,
    aggregate_metric=None,
    levels=None,
    exog=None,
    lags_grid=None,
    n_iter=10,
    random_state=123,
    return_best=True,
    n_jobs="auto",
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
    output_file=None,
)

Random search over specified parameter values or distributions for a Forecaster object. Validation is done using multi-series backtesting.

Parameters:

Name	Type	Description	Default
forecaster	(ForecasterRecursiveMultiSeries, ForecasterDirectMultiVariate)	Forecaster model.	required
series	pandas DataFrame, dict	Training time series.	required
cv	(TimeSeriesFold, OneStepAheadFold)	TimeSeriesFold or OneStepAheadFold object with the information needed to split the data into folds.	required
param_distributions	dict	Dictionary with parameters names (str) as keys and distributions or lists of parameters to try.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'} If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. If list: List containing multiple strings and/or Callables.	required
aggregate_metric	(str, list)	Aggregation method/s used to combine the metric/s of all levels (series) when multiple levels are predicted. If list, the first aggregation method is used to select the best parameters. If None, ['weighted_average', 'average', 'pooling'] is used. 'average': the average (arithmetic mean) of all levels. 'weighted_average': the average of the metrics weighted by the number of predicted values of each level. 'pooling': the values of all levels are pooled and then the metric is calculated.	None
levels	(str, list)	level (str) or levels (list) at which the forecaster is optimized. If None, all levels are taken into account.	None
exog	pandas Series, pandas DataFrame, dict	Exogenous variables.	None
lags_grid	(list, dict)	Lists of lags to try, containing int, lists, numpy ndarray, or range objects. If dict, the keys are used as labels in the results DataFrame, and the values are used as the lists of lags to try.	None
n_iter	int	Number of parameter settings that are sampled per lags configuration. n_iter trades off runtime vs quality of the solution.	10
random_state	int	Sets a seed to the random sampling for reproducible output.	123
return_best	bool	Refit the forecaster using the best found parameters on the whole data.	True
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting. Ignored for OneStepAheadFold.	'auto'
verbose	bool	Print number of folds used for cv or backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information.	False
output_file	str	Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file.	None

Returns:

Name	Type	Description
results	pandas DataFrame	Results for each combination of parameters. column levels: levels configuration for each iteration. column lags: lags configuration for each iteration. column lags_label: descriptive label or alias for the lags. column params: parameters configuration for each iteration. column metric: metric value estimated for each iteration. The resulting metric will be the average of the optimization of all levels. additional n columns with param = value.

Source code in skforecast\model_selection\_search.py

def random_search_forecaster_multiseries(
    forecaster: object,
    series: pd.DataFrame | dict[str, pd.Series | pd.DataFrame],
    cv: TimeSeriesFold | OneStepAheadFold,
    param_distributions: dict,
    metric: str | Callable | list[str | Callable],
    aggregate_metric: str | list[str] | None = None,
    levels: str | list[str] | None = None,
    exog: pd.Series | pd.DataFrame | dict[str, pd.Series | pd.DataFrame] | None = None,
    lags_grid: (
        list[int | list[int] | np.ndarray[int] | range[int]]
        | dict[str, list[int | list[int] | np.ndarray[int] | range[int]]]
        | None
    ) = None,
    n_iter: int = 10,
    random_state: int = 123,
    return_best: bool = True,
    n_jobs: int | str = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: str | None = None
) -> pd.DataFrame:
    """
    Random search over specified parameter values or distributions for a Forecaster 
    object. Validation is done using multi-series backtesting.

    Parameters
    ----------
    forecaster : ForecasterRecursiveMultiSeries, ForecasterDirectMultiVariate
        Forecaster model.
    series : pandas DataFrame, dict
        Training time series.
    cv : TimeSeriesFold, OneStepAheadFold
        TimeSeriesFold or OneStepAheadFold object with the information needed to split
        the data into folds.
    param_distributions : dict
        Dictionary with parameters names (`str`) as keys and distributions or 
        lists of parameters to try.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    aggregate_metric : str, list, default None
        Aggregation method/s used to combine the metric/s of all levels (series)
        when multiple levels are predicted. If list, the first aggregation method
        is used to select the best parameters. If `None`,
        `['weighted_average', 'average', 'pooling']` is used.

        - 'average': the average (arithmetic mean) of all levels.
        - 'weighted_average': the average of the metrics weighted by the number of
        predicted values of each level.
        - 'pooling': the values of all levels are pooled and then the metric is
        calculated.
    levels : str, list, default None
        level (`str`) or levels (`list`) at which the forecaster is optimized. 
        If `None`, all levels are taken into account.
    exog : pandas Series, pandas DataFrame, dict, default None
        Exogenous variables.
    lags_grid : list, dict, default None
        Lists of lags to try, containing int, lists, numpy ndarray, or range 
        objects. If `dict`, the keys are used as labels in the `results` 
        DataFrame, and the values are used as the lists of lags to try.
    n_iter : int, default 10
        Number of parameter settings that are sampled per lags configuration. 
        n_iter trades off runtime vs quality of the solution.
    random_state : int, default 123
        Sets a seed to the random sampling for reproducible output.
    return_best : bool, default True
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default 'auto'
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting. Ignored for `OneStepAheadFold`.
    verbose : bool, default False
        Print number of folds used for cv or backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings: bool, default False
        If `True`, skforecast warnings will be suppressed during the hyperparameter 
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.
    output_file : str, default None
        Specifies the filename or full path where the results should be saved. 
        The results will be saved in a tab-separated values (TSV) format. If 
        `None`, the results will not be saved to a file.

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column levels: levels configuration for each iteration.
        - column lags: lags configuration for each iteration.
        - column lags_label: descriptive label or alias for the lags.
        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration. The resulting 
        metric will be the average of the optimization of all levels.
        - additional n columns with param = value.

    """

    param_grid = list(
        ParameterSampler(param_distributions, n_iter=n_iter, random_state=random_state)
    )

    results = _evaluate_grid_hyperparameters_multiseries(
                  forecaster        = forecaster,
                  series            = series,
                  cv                = cv,
                  param_grid        = param_grid,
                  metric            = metric,
                  aggregate_metric  = aggregate_metric,
                  levels            = levels,
                  exog              = exog,
                  lags_grid         = lags_grid,
                  return_best       = return_best,
                  n_jobs            = n_jobs,
                  verbose           = verbose,
                  show_progress     = show_progress,
                  suppress_warnings = suppress_warnings,
                  output_file       = output_file
              )

    return results

skforecast.model_selection._search.bayesian_search_forecaster_multiseries

bayesian_search_forecaster_multiseries(
    forecaster,
    series,
    cv,
    search_space,
    metric,
    aggregate_metric=None,
    levels=None,
    exog=None,
    n_trials=20,
    random_state=123,
    return_best=True,
    n_jobs="auto",
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
    output_file=None,
    kwargs_create_study=None,
    kwargs_study_optimize=None,
)

Bayesian search for hyperparameters of a Forecaster object using optuna library.

Parameters:

Name	Type	Description	Default
forecaster	(ForecasterRecursiveMultiSeries, ForecasterDirectMultiVariate)	Forecaster model.	required
series	pandas DataFrame, dict	Training time series.	required
cv	(TimeSeriesFold, OneStepAheadFold)	TimeSeriesFold or OneStepAheadFold object with the information needed to split the data into folds.	required
search_space	Callable	Function with argument trial which returns a dictionary with parameters names (str) as keys and Trial object from optuna (trial.suggest_float, trial.suggest_int, trial.suggest_categorical) as values.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'} If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. If list: List containing multiple strings and/or Callables.	required
aggregate_metric	(str, list)	Aggregation method/s used to combine the metric/s of all levels (series) when multiple levels are predicted. If list, the first aggregation method is used to select the best parameters. If None, ['weighted_average', 'average', 'pooling'] is used. 'average': the average (arithmetic mean) of all levels. 'weighted_average': the average of the metrics weighted by the number of predicted values of each level. 'pooling': the values of all levels are pooled and then the metric is calculated.	None
levels	(str, list)	level (str) or levels (list) at which the forecaster is optimized. If None, all levels are taken into account.	None
exog	pandas Series, pandas DataFrame, dict	Exogenous variables.	None
n_trials	int	Number of parameter settings that are sampled in each lag configuration. The first 10 trials are random (controlled by optuna's n_startup_trials); the TPE sampler only guides the search from trial 11 onward. For meaningful Bayesian optimization, n_trials should be significantly larger than 10.	20
random_state	int	Sets a seed to the sampling for reproducible output. When a new sampler is passed in kwargs_create_study, the seed must be set within the sampler. For example {'sampler': TPESampler(seed=145)}.	123
return_best	bool	Refit the forecaster using the best found parameters on the whole data.	True
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting. Ignored for OneStepAheadFold.	'auto'
verbose	bool	Print number of folds used for cv or backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information.	False
output_file	str	Specifies the filename or full path where the optuna logging output should be saved. If None, logging output will not be redirected to a file.	None
kwargs_create_study	dict	Additional keyword arguments (key, value mappings) to pass to optuna.create_study(). If default, the direction is set to 'minimize' for regression tasks or 'maximize' for classification tasks, and a TPESampler(multivariate=True, group=True, consider_endpoints=True, seed=random_state) sampler is used during optimization.	None
kwargs_study_optimize	dict	Additional keyword arguments (key, value mappings) to pass to study.optimize().	None

Returns:

Name	Type	Description
results	pandas DataFrame	Results for each combination of parameters. column trial_number: optuna trial number for each iteration. Use study.trials[trial_number] to access the full optuna trial object. column levels: levels configuration for each iteration. column lags: lags configuration for each iteration. column params: parameters configuration for each iteration. column metric: metric value estimated for each iteration. The resulting metric will be the average of the optimization of all levels. additional n columns with param = value.
study	optuna Study	The optuna study object containing all optimization trials. Access the best trial via study.best_trial.

Source code in skforecast\model_selection\_search.py

@manage_warnings
def bayesian_search_forecaster_multiseries(
    forecaster: object,
    series: pd.DataFrame | dict[str, pd.Series | pd.DataFrame],
    cv: TimeSeriesFold | OneStepAheadFold,
    search_space: Callable,
    metric: str | Callable | list[str | Callable],
    aggregate_metric: str | list[str] | None = None,
    levels: str | list[str] | None = None,
    exog: pd.Series | pd.DataFrame | dict[str, pd.Series | pd.DataFrame] | None = None,
    n_trials: int = 20,
    random_state: int = 123,
    return_best: bool = True,
    n_jobs: int | str = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: str | None = None,
    kwargs_create_study: dict | None = None,
    kwargs_study_optimize: dict | None = None
) -> tuple[pd.DataFrame, object]:
    """
    Bayesian search for hyperparameters of a Forecaster object using optuna library.

    Parameters
    ----------
    forecaster : ForecasterRecursiveMultiSeries, ForecasterDirectMultiVariate
        Forecaster model.
    series : pandas DataFrame, dict
        Training time series.
    cv : TimeSeriesFold, OneStepAheadFold
        TimeSeriesFold or OneStepAheadFold object with the information needed to split
        the data into folds.
    search_space : Callable
        Function with argument `trial` which returns a dictionary with parameters names 
        (`str`) as keys and Trial object from optuna (trial.suggest_float, 
        trial.suggest_int, trial.suggest_categorical) as values.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    aggregate_metric : str, list, default None
        Aggregation method/s used to combine the metric/s of all levels (series)
        when multiple levels are predicted. If list, the first aggregation method
        is used to select the best parameters. If `None`,
        `['weighted_average', 'average', 'pooling']` is used.

        - 'average': the average (arithmetic mean) of all levels.
        - 'weighted_average': the average of the metrics weighted by the number of
        predicted values of each level.
        - 'pooling': the values of all levels are pooled and then the metric is
        calculated.
    levels : str, list, default None
        level (`str`) or levels (`list`) at which the forecaster is optimized. 
        If `None`, all levels are taken into account.
    exog : pandas Series, pandas DataFrame, dict, default None
        Exogenous variables.
    n_trials : int, default 20
        Number of parameter settings that are sampled in each lag configuration. 
        The first 10 trials are random (controlled by optuna's `n_startup_trials`); 
        the TPE sampler only guides the search from trial 11 onward. For meaningful 
        Bayesian optimization, `n_trials` should be significantly larger than 10.
    random_state : int, default 123
        Sets a seed to the sampling for reproducible output. When a new sampler 
        is passed in `kwargs_create_study`, the seed must be set within the 
        sampler. For example `{'sampler': TPESampler(seed=145)}`.
    return_best : bool, default True
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default 'auto'
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting. Ignored for `OneStepAheadFold`.
    verbose : bool, default False
        Print number of folds used for cv or backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings: bool, default False
        If `True`, skforecast warnings will be suppressed during the hyperparameter
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.
    output_file : str, default None
        Specifies the filename or full path where the optuna logging output
        should be saved. If `None`, logging output will not be redirected to
        a file.
    kwargs_create_study : dict, default None
        Additional keyword arguments (key, value mappings) to pass to optuna.create_study().
        If default, the direction is set to 'minimize' for regression tasks or
        'maximize' for classification tasks, and a 
        `TPESampler(multivariate=True, group=True, consider_endpoints=True, seed=random_state)` 
        sampler is used during optimization.
    kwargs_study_optimize : dict, default None
        Additional keyword arguments (key, value mappings) to pass to study.optimize().

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column trial_number: optuna trial number for each iteration. Use
          `study.trials[trial_number]` to access the full optuna trial object.
        - column levels: levels configuration for each iteration.
        - column lags: lags configuration for each iteration.
        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration. The resulting 
        metric will be the average of the optimization of all levels.
        - additional n columns with param = value.
    study : optuna Study
        The optuna study object containing all optimization trials. Access the
        best trial via `study.best_trial`.

    """

    forecaster_search = deepcopy_forecaster(forecaster)
    forecaster_name = forecaster_search.__skforecast_tags__['forecaster_name']
    cv_name = type(cv).__name__

    if forecaster_name == 'ForecasterRecursiveMultiSeries':
        series, series_indexes = check_preprocess_series(series)
        if exog is not None:
            series_names_in_ = list(series.keys())
            exog_dict = {serie: None for serie in series_names_in_}
            exog, _ = check_preprocess_exog_multiseries(
                          series_names_in_  = series_names_in_,
                          series_index_type = type(series_indexes[series_names_in_[0]]),
                          exog              = exog,
                          exog_dict         = exog_dict
                      )
    else:
        # TODO: This only applies to wide DataFrames. Delete when input is always dict
        # in all forecasters.
        if return_best and exog is not None and (len(exog) != len(series)):
            raise ValueError(
                f"`exog` must have same number of samples as `series`. "
                f"length `exog`: ({len(exog)}), length `series`: ({len(series)})"
            )

    if cv_name not in ['TimeSeriesFold', 'OneStepAheadFold']:
        raise TypeError(
            f"`cv` must be an instance of `TimeSeriesFold` or `OneStepAheadFold`. "
            f"Got {type(cv)}."
        )

    if cv_name == 'OneStepAheadFold':

        check_one_step_ahead_input(
            forecaster        = forecaster_search,
            cv                = cv,
            metric            = metric,
            series            = series,
            exog              = exog,
            show_progress     = show_progress,
            suppress_warnings = suppress_warnings
        )

        cv = deepcopy(cv)
        initial_train_size = date_to_index_position(
                                 index        = cv._extract_index(series), 
                                 date_input   = cv.initial_train_size, 
                                 method       = 'validation',
                                 date_literal = 'initial_train_size'
                             )
        cv.set_params({
            'initial_train_size': initial_train_size,
            'window_size': forecaster_search.window_size,
            'differentiation': forecaster_search.differentiation_max,
            'verbose': verbose
        })

    if aggregate_metric is None:
        aggregate_metric = ['weighted_average', 'average', 'pooling']
    if isinstance(aggregate_metric, str):
        aggregate_metric = [aggregate_metric]
    allowed_aggregate_metrics = ['average', 'weighted_average', 'pooling']
    if not set(aggregate_metric).issubset(allowed_aggregate_metrics):
        raise ValueError(
            f"Allowed `aggregate_metric` are: {allowed_aggregate_metrics}. "
            f"Got: {aggregate_metric}."
        )

    if not isinstance(metric, list):
        metric = [metric]
    metric = [
        _get_metric(metric=m)
        if isinstance(m, str)
        else add_y_train_argument(m) 
        for m in metric
    ]
    metric_names = [(m if isinstance(m, str) else m.__name__) for m in metric]
    if len(metric_names) != len(set(metric_names)):
        raise ValueError(
            "When `metric` is a `list`, each metric name must be unique."
        )

    levels = _initialize_levels_model_selection_multiseries(
                 forecaster = forecaster_search,
                 series     = series,
                 levels     = levels
             )
    add_aggregated_metric = True if len(levels) > 1 else False
    if add_aggregated_metric:
        metric_names = [
            f"{metric_name}__{aggregation}"
            for metric_name in metric_names
            for aggregation in aggregate_metric
        ]

    # Objective function using backtesting_forecaster_multiseries
    if cv_name == 'TimeSeriesFold':

        def _objective(
            trial,
            search_space          = search_space,
            forecaster_search     = forecaster_search,
            series                = series,
            cv                    = cv,
            exog                  = exog,
            levels                = levels,
            metric                = metric,
            add_aggregated_metric = add_aggregated_metric,
            aggregate_metric      = aggregate_metric,
            metric_names          = metric_names,
            n_jobs                = n_jobs,
            verbose               = verbose,
            suppress_warnings     = suppress_warnings
        ) -> float:

            sample = search_space(trial)
            if sample.keys() != trial.params.keys():
                raise ValueError(
                    f"`search_space` dict keys must match the names passed to "
                    f"`trial.suggest_*()`.\n"
                    f"  Dict keys    : {list(sample.keys())}\n"
                    f"  Suggest names: {list(trial.params.keys())}"
                )
            sample_params = {k: v for k, v in sample.items() if k != 'lags'}
            forecaster_search.set_params(sample_params)
            if "lags" in sample:
                forecaster_search.set_lags(sample['lags'])

            metrics, _ = backtesting_forecaster_multiseries(
                             forecaster            = forecaster_search,
                             series                = series,
                             cv                    = cv,
                             exog                  = exog,
                             levels                = levels,
                             metric                = metric,
                             add_aggregated_metric = add_aggregated_metric,
                             n_jobs                = n_jobs,
                             verbose               = verbose,
                             show_progress         = False,
                             suppress_warnings     = suppress_warnings
                         )

            if add_aggregated_metric:
                metrics = metrics.loc[metrics['levels'].isin(aggregate_metric), :]
            else:
                metrics = metrics.loc[metrics['levels'] == levels[0], :]
            metrics = pd.DataFrame(
                          data    = [metrics.iloc[:, 1:].transpose().stack().to_numpy()],
                          columns = metric_names
                      )

            # Store all metrics in the trial using optuna's user_attrs mechanism.
            for m_name in metric_names:
                trial.set_user_attr(m_name, float(metrics.loc[0, m_name]))

            return metrics.loc[0, metric_names[0]]

    else:

        _SENTINEL = object()
        _MAX_CACHE_SIZE = 10
        _cached_split = {}

        def _objective(
            trial,
            search_space          = search_space,
            forecaster_search     = forecaster_search,
            series                = series,
            cv                    = cv,
            exog                  = exog,
            levels                = levels,
            metric                = metric,
            add_aggregated_metric = add_aggregated_metric,
            aggregate_metric      = aggregate_metric,
            metric_names          = metric_names
        ) -> float:

            sample = search_space(trial)
            if sample.keys() != trial.params.keys():
                raise ValueError(
                    f"`search_space` dict keys must match the names passed to "
                    f"`trial.suggest_*()`.\n"
                    f"  Dict keys    : {list(sample.keys())}\n"
                    f"  Suggest names: {list(trial.params.keys())}"
                )
            sample_params = {k: v for k, v in sample.items() if k != 'lags'}
            forecaster_search.set_params(sample_params)

            current_lags = sample.get('lags', _SENTINEL)
            if current_lags is not _SENTINEL:
                forecaster_search.set_lags(current_lags)

            lags_key = _make_lags_hashable(current_lags, sentinel=_SENTINEL)
            if lags_key not in _cached_split:

                if len(_cached_split) >= _MAX_CACHE_SIZE:
                    _cached_split.pop(next(iter(_cached_split)))

                (
                    X_train,
                    y_train,
                    X_test,
                    y_test,
                    X_train_encoding,
                    X_test_encoding
                ) = forecaster_search._train_test_split_one_step_ahead(
                    series=series, exog=exog, initial_train_size=cv.initial_train_size,
                )
                _cached_split[lags_key] = (
                    X_train, y_train, X_test, y_test,
                    X_train_encoding, X_test_encoding
                )
            else:
                (
                    X_train, y_train, X_test, y_test, X_train_encoding, X_test_encoding
                ) = _cached_split[lags_key]

            metrics, _ = _predict_and_calculate_metrics_one_step_ahead_multiseries(
                             forecaster            = forecaster_search,
                             series                = series,
                             X_train               = X_train,
                             y_train               = y_train,
                             X_test                = X_test,
                             y_test                = y_test,
                             X_train_encoding      = X_train_encoding,
                             X_test_encoding       = X_test_encoding,
                             levels                = levels,
                             metrics               = metric,
                             add_aggregated_metric = add_aggregated_metric
                         )

            if add_aggregated_metric:
                metrics = metrics.loc[metrics['levels'].isin(aggregate_metric), :]
            else:
                metrics = metrics.loc[metrics['levels'] == levels[0], :]
            metrics = pd.DataFrame(
                          data    = [metrics.iloc[:, 1:].transpose().stack().to_numpy()],
                          columns = metric_names
                      )

            # Store all metrics in the trial using optuna's user_attrs mechanism.
            for m_name in metric_names:
                trial.set_user_attr(m_name, float(metrics.loc[0, m_name]))

            return metrics.loc[0, metric_names[0]]

    is_regression = forecaster_search.__skforecast_tags__['forecaster_task'] == 'regression'
    kwargs_create_study = kwargs_create_study.copy() if kwargs_create_study is not None else {}
    if 'direction' not in kwargs_create_study:
        kwargs_create_study['direction'] = 'minimize' if is_regression else 'maximize'
    if 'sampler' not in kwargs_create_study:
        with warnings.catch_warnings():
            warnings.filterwarnings(
                'ignore',
                message='.*multivariate.*|.*group.*',
                module='optuna'
            )
            kwargs_create_study['sampler'] = TPESampler(
                multivariate=True, group=True, consider_endpoints=True, seed=random_state
            )

    kwargs_study_optimize = kwargs_study_optimize.copy() if kwargs_study_optimize is not None else {}
    if show_progress:
        kwargs_study_optimize['show_progress_bar'] = True
    else:
        kwargs_study_optimize.setdefault('show_progress_bar', False)

    if output_file is not None:
        # Redirect optuna logging to file
        optuna.logging.disable_default_handler()
        logger = logging.getLogger('optuna')
        logger.setLevel(logging.INFO)
        for handler in logger.handlers.copy():
            if isinstance(handler, logging.StreamHandler):
                logger.removeHandler(handler)
        handler = logging.FileHandler(output_file, mode="w")
        logger.addHandler(handler)
    else:
        logging.getLogger("optuna").setLevel(logging.WARNING)
        optuna.logging.disable_default_handler()

    study = optuna.create_study(**kwargs_create_study)

    try:
        with warnings.catch_warnings():
            warnings.filterwarnings(
                "ignore",
                category=UserWarning,
                message="Choices for a categorical distribution should be*"
            )
            study.optimize(_objective, n_trials=n_trials, **kwargs_study_optimize)
    finally:
        if output_file is not None:
            handler.close()
            logger.removeHandler(handler)

    lags_list = []
    params_list = []
    metrics_data = []
    trial_number_list = []
    for trial in study.get_trials(states=[TrialState.COMPLETE]):
        estimator_params = {k: v for k, v in trial.params.items() if k != 'lags'}
        lags = trial.params.get(
            'lags',
            forecaster_search.lags if hasattr(forecaster_search, 'lags') else None
        )
        params_list.append(estimator_params)
        lags_list.append(lags)
        trial_number_list.append(trial.number)
        metrics_data.append(
            {m_name: trial.user_attrs[m_name] for m_name in metric_names}
        )

    if forecaster_name not in ['ForecasterDirectMultiVariate']:
        lags_list = [
            initialize_lags(forecaster_name=forecaster_name, lags=lag)[0]
            for lag in lags_list
        ]
    else:
        lags_list_initialized = []
        for lags in lags_list:
            if isinstance(lags, dict):
                for key in lags:
                    if lags[key] is None:
                        lags[key] = None
                    else:
                        lags[key] = initialize_lags(
                                        forecaster_name = forecaster_name,
                                        lags            = lags[key]
                                    )[0]
            else:
                lags = initialize_lags(
                           forecaster_name = forecaster_name,
                           lags            = lags
                       )[0]
            lags_list_initialized.append(lags)

        lags_list = lags_list_initialized

    results = pd.DataFrame(metrics_data)
    results.insert(0, 'trial_number', trial_number_list)
    results.insert(1, 'levels', [levels] * len(results))
    results.insert(2, 'lags', lags_list)
    results.insert(3, 'params', params_list)
    results = (
        results
        .sort_values(by=metric_names[0], ascending=True if is_regression else False)
        .reset_index(drop=True)
    )
    results = pd.concat([results, results['params'].apply(pd.Series)], axis=1)

    if return_best:

        best_lags = results.loc[0, 'lags']
        best_params = results.loc[0, 'params']
        best_metric = results.loc[0, metric_names[0]]

        # NOTE: Here we use the actual forecaster passed by the user
        forecaster.set_lags(best_lags)
        forecaster.set_params(best_params)

        forecaster.fit(
            series                    = series, 
            exog                      = exog, 
            store_in_sample_residuals = True, 
            suppress_warnings         = suppress_warnings
        )

        if len(levels) > 20:
            levels_print = levels[:10] + ["..."] + levels[-10:]
        else:
            levels_print = levels

        if verbose:
            print(
                f"`Forecaster` refitted using the best-found lags and parameters, "
                f"and the whole data set: \n"
                f"  Lags: {best_lags} \n"
                f"  Parameters: {best_params}\n"
                f"  {'Backtesting' if cv_name == 'TimeSeriesFold' else 'One-step-ahead'} "
                f"metric: {best_metric}\n"
                f"  Levels: {levels_print}"
            )

    return results, study

skforecast.model_selection._validation.backtesting_stats

backtesting_stats(
    forecaster,
    y,
    cv,
    metric,
    exog=None,
    alpha=None,
    interval=None,
    freeze_params=True,
    n_jobs="auto",
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
)

Backtesting of ForecasterStats.

A copy of the original forecaster is created so that it is not modified during the process.

Parameters:

Name	Type	Description	Default
forecaster	ForecasterStats	Forecaster model.	required
y	pandas Series	Training time series.	required
cv	TimeSeriesFold	TimeSeriesFold object with the information needed to split the data into folds.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'} If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. If list: List containing multiple strings and/or Callables.	required
exog	pandas Series, pandas DataFrame	Exogenous variable/s included as predictor/s. Must have the same number of observations as y and should be aligned so that y[i] is regressed on exog[i].	None
alpha	float	The confidence intervals for the forecasts are (1 - alpha) %. If both, alpha and interval are provided, alpha will be used.	None
interval	(list, tuple)	Confidence of the prediction interval estimated. The values must be symmetric. Sequence of percentiles to compute, which must be between 0 and 100 inclusive. For example, interval of 95% should be as interval = [2.5, 97.5]. If both, alpha and interval are provided, alpha will be used.	None
freeze_params	bool	Determines whether to freeze the model parameters after the first fit for estimators that perform automatic model selection. If True, the model parameters found during the first fit (e.g., order and seasonal_order for Arima, or smoothing parameters for Ets) are reused in all subsequent refits. This avoids re-running the automatic selection procedure in each fold and reduces runtime. If False, automatic model selection is performed independently in each refit, allowing parameters to adapt across folds. This increases runtime and adds a params column to the output with the parameters selected per fold.	True
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting.	'auto'
verbose	bool	Print number of folds and index of training and validation sets used for backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the backtesting process. See skforecast.exceptions.warn_skforecast_categories for more information.	False

Returns:

Name	Type	Description
metric_values	pandas DataFrame	Value(s) of the metric(s).
backtest_predictions	pandas DataFrame	Value of predictions. The DataFrame includes the following columns: fold: Indicates the fold number where the prediction was made. pred: Predicted values for the corresponding series and time steps. If interval is not None, additional columns are included: lower_bound: lower bound of the interval. upper_bound: upper bound of the interval. If freeze_params is False, an additional column is included: estimator_params: parameters used in the estimator for each fold. Depending on the relation between steps and fold_stride, the output may include repeated indexes (if fold_stride < steps) or gaps (if fold_stride > steps). See Notes below for more details.

Notes

Note on fold_stride vs. steps:

• If fold_stride == steps, test sets are placed back-to-back without overlap. Each observation appears only once in the output DataFrame, so the index is unique.
• If fold_stride < steps, test sets overlap. Multiple forecasts are generated for the same observations and, therefore, the output DataFrame contains repeated indexes.
• If fold_stride > steps, there are gaps between consecutive test sets. Some observations in the series will not have associated predictions, so the output DataFrame has non-contiguous indexes.

Source code in skforecast\model_selection\_validation.py

def backtesting_stats(
    forecaster: object,
    y: pd.Series,
    cv: TimeSeriesFold,
    metric: str | Callable | list[str | Callable],
    exog: pd.Series | pd.DataFrame | None = None,
    alpha: float | None = None,
    interval: list[float] | tuple[float] | None = None,
    freeze_params: bool = True,
    n_jobs: int | str = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False,
) -> tuple[pd.DataFrame, pd.DataFrame]:
    """
    Backtesting of ForecasterStats.

    A copy of the original forecaster is created so that it is not modified during 
    the process.

    Parameters
    ----------
    forecaster : ForecasterStats
        Forecaster model.
    y : pandas Series
        Training time series.
    cv : TimeSeriesFold
        TimeSeriesFold object with the information needed to split the data into folds.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    exog : pandas Series, pandas DataFrame, default None
        Exogenous variable/s included as predictor/s. Must have the same
        number of observations as `y` and should be aligned so that y[i] is
        regressed on exog[i].
    alpha : float, default None
        The confidence intervals for the forecasts are (1 - alpha) %.
        If both, `alpha` and `interval` are provided, `alpha` will be used.
    interval : list, tuple, default None
        Confidence of the prediction interval estimated. The values must be
        symmetric. Sequence of percentiles to compute, which must be between 
        0 and 100 inclusive. For example, interval of 95% should be as 
        `interval = [2.5, 97.5]`. If both, `alpha` and `interval` are 
        provided, `alpha` will be used.
    freeze_params : bool, default True
        Determines whether to freeze the model parameters after the first fit
        for estimators that perform automatic model selection.

        - If `True`, the model parameters found during the first fit (e.g., order 
        and seasonal_order for Arima, or smoothing parameters for Ets) are reused
        in all subsequent refits. This avoids re-running the automatic selection
        procedure in each fold and reduces runtime.
        - If `False`, automatic model selection is performed independently in each
        refit, allowing parameters to adapt across folds. This increases runtime
        and adds a `params` column to the output with the parameters selected per
        fold.
    n_jobs : int, 'auto', default 'auto'
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting. 
    verbose : bool, default False
        Print number of folds and index of training and validation sets used 
        for backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings: bool, default False
        If `True`, skforecast warnings will be suppressed during the backtesting 
        process. See skforecast.exceptions.warn_skforecast_categories for more
        information.

    Returns
    -------
    metric_values : pandas DataFrame
        Value(s) of the metric(s).
    backtest_predictions : pandas DataFrame
        Value of predictions. The DataFrame includes the following columns:

        - fold: Indicates the fold number where the prediction was made.
        - pred: Predicted values for the corresponding series and time steps.

        If `interval` is not `None`, additional columns are included:

        - lower_bound: lower bound of the interval.
        - upper_bound: upper bound of the interval.

        If `freeze_params` is `False`, an additional column is included:

        - estimator_params: parameters used in the estimator for each fold.

        Depending on the relation between `steps` and `fold_stride`, the output
        may include repeated indexes (if `fold_stride < steps`) or gaps
        (if `fold_stride > steps`). See Notes below for more details.

    Notes
    -----
    Note on `fold_stride` vs. `steps`:

    - If `fold_stride == steps`, test sets are placed back-to-back without overlap. 
    Each observation appears only once in the output DataFrame, so the index is unique.
    - If `fold_stride < steps`, test sets overlap. Multiple forecasts are generated 
    for the same observations and, therefore, the output DataFrame contains repeated 
    indexes.
    - If `fold_stride > steps`, there are gaps between consecutive test sets. 
    Some observations in the series will not have associated predictions, so 
    the output DataFrame has non-contiguous indexes.

    """

    if type(forecaster).__name__  != 'ForecasterStats':
        raise TypeError(
            "`forecaster` must be of type `ForecasterStats`. For all other "
            "types of forecasters use the other functions available in the "
            "`model_selection` module."
        )

    check_backtesting_input(
        forecaster        = forecaster,
        cv                = cv,
        y                 = y,
        metric            = metric,
        interval          = interval,
        alpha             = alpha,
        freeze_params     = freeze_params,
        n_jobs            = n_jobs,
        show_progress     = show_progress,
        suppress_warnings = suppress_warnings
    )

    metric_values, backtest_predictions = _backtesting_stats(
        forecaster        = forecaster,
        y                 = y,
        cv                = cv,
        metric            = metric,
        exog              = exog,
        alpha             = alpha,
        interval          = interval,
        freeze_params     = freeze_params,
        n_jobs            = n_jobs,
        verbose           = verbose,
        show_progress     = show_progress,
        suppress_warnings = suppress_warnings
    )

    return metric_values, backtest_predictions

skforecast.model_selection._search.grid_search_stats

grid_search_stats(
    forecaster,
    y,
    cv,
    param_grid,
    metric,
    exog=None,
    return_best=True,
    n_jobs="auto",
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
    output_file=None,
)

Exhaustive search over specified parameter values for a ForecasterStats object. Validation is done using time series backtesting.

Parameters:

Name	Type	Description	Default
forecaster	ForecasterStats	Forecaster model.	required
y	pandas Series	Training time series.	required
cv	TimeSeriesFold	TimeSeriesFold object with the information needed to split the data into folds.	required
param_grid	dict	Dictionary with parameters names (str) as keys and lists of parameter settings to try as values.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'} If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. If list: List containing multiple strings and/or Callables.	required
exog	pandas Series, pandas DataFrame	Exogenous variable/s included as predictor/s. Must have the same number of observations as y and should be aligned so that y[i] is regressed on exog[i].	None
return_best	bool	Refit the forecaster using the best found parameters on the whole data.	True
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting. Ignored for OneStepAheadFold.	'auto'
verbose	bool	Print number of folds used for cv or backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information.	False
output_file	str	Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file.	None

Returns:

Name	Type	Description
results	pandas DataFrame	Results for each combination of parameters. column params: parameters configuration for each iteration. column metric: metric value estimated for each iteration. additional n columns with param = value.

Source code in skforecast\model_selection\_search.py

def grid_search_stats(
    forecaster: object,
    y: pd.Series,
    cv: TimeSeriesFold,
    param_grid: dict,
    metric: str | Callable | list[str | Callable],
    exog: pd.Series | pd.DataFrame | None = None,
    return_best: bool = True,
    n_jobs: int | str = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: str | None = None
) -> pd.DataFrame:
    """
    Exhaustive search over specified parameter values for a ForecasterStats object.
    Validation is done using time series backtesting.

    Parameters
    ----------
    forecaster : ForecasterStats
        Forecaster model.
    y : pandas Series
        Training time series. 
    cv : TimeSeriesFold
        TimeSeriesFold object with the information needed to split the data into folds.
    param_grid : dict
        Dictionary with parameters names (`str`) as keys and lists of parameter
        settings to try as values.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    exog : pandas Series, pandas DataFrame, default None
        Exogenous variable/s included as predictor/s. Must have the same
        number of observations as `y` and should be aligned so that y[i] is
        regressed on exog[i].
    return_best : bool, default True
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default 'auto'
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting. Ignored for `OneStepAheadFold`.
    verbose : bool, default False
        Print number of folds used for cv or backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings : bool, default False
        If `True`, skforecast warnings will be suppressed during the hyperparameter 
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.
    output_file : str, default None
        Specifies the filename or full path where the results should be saved. 
        The results will be saved in a tab-separated values (TSV) format. If 
        `None`, the results will not be saved to a file.

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration.
        - additional n columns with param = value.

    """

    param_grid = list(ParameterGrid(param_grid))

    results = _evaluate_grid_hyperparameters_stats(
        forecaster        = forecaster,
        y                 = y,
        cv                = cv,
        param_grid        = param_grid,
        metric            = metric,
        exog              = exog,
        return_best       = return_best,
        n_jobs            = n_jobs,
        verbose           = verbose,
        suppress_warnings = suppress_warnings,
        show_progress     = show_progress,
        output_file       = output_file
    )

    return results

skforecast.model_selection._search.random_search_stats

random_search_stats(
    forecaster,
    y,
    cv,
    param_distributions,
    metric,
    exog=None,
    n_iter=10,
    random_state=123,
    return_best=True,
    n_jobs="auto",
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
    output_file=None,
)

Random search over specified parameter values or distributions for a ForecasterStats object. Validation is done using time series backtesting.

Parameters:

Name	Type	Description	Default
forecaster	ForecasterStats	Forecaster model.	required
y	pandas Series	Training time series.	required
cv	TimeSeriesFold	TimeSeriesFold object with the information needed to split the data into folds.	required
param_distributions	dict	Dictionary with parameters names (str) as keys and distributions or lists of parameters to try.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'} If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. If list: List containing multiple strings and/or Callables.	required
exog	pandas Series, pandas DataFrame	Exogenous variable/s included as predictor/s. Must have the same number of observations as y and should be aligned so that y[i] is regressed on exog[i].	None
n_iter	int	Number of parameter settings that are sampled. n_iter trades off runtime vs quality of the solution.	10
random_state	int	Sets a seed to the random sampling for reproducible output.	123
return_best	bool	Refit the forecaster using the best found parameters on the whole data.	True
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting. Ignored for OneStepAheadFold.	'auto'
verbose	bool	Print number of folds used for cv or backtesting.	False
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information.	False
output_file	str	Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file.	None

Returns:

Name	Type	Description
results	pandas DataFrame	Results for each combination of parameters. column params: parameters configuration for each iteration. column metric: metric value estimated for each iteration. additional n columns with param = value.

Source code in skforecast\model_selection\_search.py

def random_search_stats(
    forecaster: object,
    y: pd.Series,
    cv: TimeSeriesFold,
    param_distributions: dict,
    metric: str | Callable | list[str | Callable],
    exog: pd.Series | pd.DataFrame | None = None,
    n_iter: int = 10,
    random_state: int = 123,
    return_best: bool = True,
    n_jobs: int | str = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: str | None = None
) -> pd.DataFrame:
    """
    Random search over specified parameter values or distributions for a ForecasterStats 
    object. Validation is done using time series backtesting.

    Parameters
    ----------
    forecaster : ForecasterStats
        Forecaster model.
    y : pandas Series
        Training time series. 
    cv : TimeSeriesFold
        TimeSeriesFold object with the information needed to split the data into folds.
    param_distributions : dict
        Dictionary with parameters names (`str`) as keys and 
        distributions or lists of parameters to try.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    exog : pandas Series, pandas DataFrame, default None
        Exogenous variable/s included as predictor/s. Must have the same
        number of observations as `y` and should be aligned so that y[i] is
        regressed on exog[i].
    n_iter : int, default 10
        Number of parameter settings that are sampled. 
        n_iter trades off runtime vs quality of the solution.
    random_state : int, default 123
        Sets a seed to the random sampling for reproducible output.
    return_best : bool, default True
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default 'auto'
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting. Ignored for `OneStepAheadFold`.
    verbose : bool, default False
        Print number of folds used for cv or backtesting.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings : bool, default False
        If `True`, skforecast warnings will be suppressed during the hyperparameter 
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.
    output_file : str, default None
        Specifies the filename or full path where the results should be saved. 
        The results will be saved in a tab-separated values (TSV) format. If 
        `None`, the results will not be saved to a file.

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration.
        - additional n columns with param = value.

    """

    param_grid = list(ParameterSampler(param_distributions, n_iter=n_iter, random_state=random_state))

    results = _evaluate_grid_hyperparameters_stats(
        forecaster        = forecaster,
        y                 = y,
        cv                = cv,
        param_grid        = param_grid,
        metric            = metric,
        exog              = exog,
        return_best       = return_best,
        n_jobs            = n_jobs,
        verbose           = verbose,
        suppress_warnings = suppress_warnings,
        show_progress     = show_progress,
        output_file       = output_file
    )

    return results

skforecast.model_selection._split.BaseFold

BaseFold(
    steps=None,
    initial_train_size=None,
    fold_stride=None,
    window_size=None,
    differentiation=None,
    refit=False,
    fixed_train_size=True,
    gap=0,
    skip_folds=None,
    allow_incomplete_fold=True,
    return_all_indexes=False,
    verbose=True,
)

Base class for all Fold classes in skforecast. All fold classes should specify all the parameters that can be set at the class level in their __init__.

Parameters:

Name	Type	Description	Default
steps	int	Number of observations used to be predicted in each fold. This is also commonly referred to as the forecast horizon or test size.	None
initial_train_size	int, str, pandas Timestamp	Number of observations used for initial training. If an integer, the number of observations used for initial training. If a date string or pandas Timestamp, it is the last date included in the initial training set.	None
fold_stride	int	Number of observations that the start of the test set advances between consecutive folds. If None, it defaults to the same value as steps, meaning that folds are placed back-to-back without overlap. If fold_stride < steps, test sets overlap and multiple forecasts will be generated for the same observations. If fold_stride > steps, gaps are left between consecutive test sets.	None
window_size	int	Number of observations needed to generate the autoregressive predictors.	None
differentiation	int	Number of observations to use for differentiation. This is used to extend the last_window as many observations as the differentiation order.	None
refit	(bool, int)	Whether to refit the forecaster in each fold. If True, the forecaster is refitted in each fold. If False, the forecaster is trained only in the first fold. If an integer, the forecaster is trained in the first fold and then refitted every refit folds.	False
fixed_train_size	bool	Whether the training size is fixed or increases in each fold.	True
gap	int	Number of observations between the end of the training set and the start of the test set.	0
skip_folds	(int, list)	Number of folds to skip. If an integer, every 'skip_folds'-th is returned. If a list, the indexes of the folds to skip. For example, if skip_folds=3 and there are 10 folds, the returned folds are 0, 3, 6, and 9. If skip_folds=[1, 2, 3], the returned folds are 0, 4, 5, 6, 7, 8, and 9.	None
allow_incomplete_fold	bool	Whether to allow the last fold to include fewer observations than steps. If False, the last fold is excluded if it is incomplete.	True
return_all_indexes	bool	Whether to return all indexes or only the start and end indexes of each fold.	False
verbose	bool	Whether to print information about generated folds.	True

Attributes:

Name	Type	Description
initial_train_size	int	Number of observations used for initial training.
window_size	int	Number of observations needed to generate the autoregressive predictors.
differentiation	int	Number of observations to use for differentiation. This is used to extend the last_window as many observations as the differentiation order.
return_all_indexes	bool	Whether to return all indexes or only the start and end indexes of each fold.
verbose	bool	Whether to print information about generated folds.

Methods:

Name	Description
set_params	Set the parameters of the Fold object. Before overwriting the current

Source code in skforecast\model_selection\_split.py

def __init__(
    self,
    steps: int | None = None,
    initial_train_size: int | str | pd.Timestamp | None = None,
    fold_stride: int | None = None,
    window_size: int | None = None,
    differentiation: int | None = None,
    refit: bool | int = False,
    fixed_train_size: bool = True,
    gap: int = 0,
    skip_folds: int | list[int] | None = None,
    allow_incomplete_fold: bool = True,
    return_all_indexes: bool = False,
    verbose: bool = True
) -> None:

    self._validate_params(
        cv_name               = type(self).__name__,
        steps                 = steps,
        initial_train_size    = initial_train_size,
        fold_stride           = fold_stride,
        window_size           = window_size,
        differentiation       = differentiation,
        refit                 = refit,
        fixed_train_size      = fixed_train_size,
        gap                   = gap,
        skip_folds            = skip_folds,
        allow_incomplete_fold = allow_incomplete_fold,
        return_all_indexes    = return_all_indexes,
        verbose               = verbose
    )

    self.initial_train_size = initial_train_size
    self.window_size        = window_size
    self.differentiation    = differentiation
    self.return_all_indexes = return_all_indexes
    self.verbose            = verbose

Attributes

initial_train_size instance-attribute

initial_train_size = initial_train_size

window_size instance-attribute

window_size = window_size

differentiation instance-attribute

differentiation = differentiation

return_all_indexes instance-attribute

return_all_indexes = return_all_indexes

verbose instance-attribute

verbose = verbose

Functions

_validate_params

_validate_params(
    cv_name,
    steps=None,
    initial_train_size=None,
    fold_stride=None,
    window_size=None,
    differentiation=None,
    refit=False,
    fixed_train_size=True,
    gap=0,
    skip_folds=None,
    allow_incomplete_fold=True,
    return_all_indexes=False,
    verbose=True,
    **kwargs
)

Validate all input parameters to ensure correctness.

Source code in skforecast\model_selection\_split.py

def _validate_params(
    self,
    cv_name: str,
    steps: int | None = None,
    initial_train_size: int | str | pd.Timestamp | None = None,
    fold_stride: int | None = None,
    window_size: int | None = None,
    differentiation: int | None = None,
    refit: bool | int = False,
    fixed_train_size: bool = True,
    gap: int = 0,
    skip_folds: int | list[int] | None = None,
    allow_incomplete_fold: bool = True,
    return_all_indexes: bool = False,
    verbose: bool = True,
    **kwargs
) -> None: 
    """
    Validate all input parameters to ensure correctness.
    """

    if cv_name == "TimeSeriesFold":
        if not isinstance(steps, (int, np.integer)) or steps < 1:
            raise ValueError(
                f"`steps` must be an integer greater than 0. Got {steps}."
            )
        if not isinstance(initial_train_size, (int, np.integer, str, pd.Timestamp, type(None))):
            raise ValueError(
                f"`initial_train_size` must be an integer greater than 0, a date "
                f"string, a pandas Timestamp, or None. Got {initial_train_size}."
            )
        if isinstance(initial_train_size, (int, np.integer)) and initial_train_size < 1:
            raise ValueError(
                f"`initial_train_size` must be an integer greater than 0, "
                f"a date string, a pandas Timestamp, or None. Got {initial_train_size}."
            )
        if fold_stride is not None:
            if not isinstance(fold_stride, (int, np.integer)) or fold_stride < 1:
                raise ValueError(
                    f"`fold_stride` must be an integer greater than 0. Got {fold_stride}."
                )
        if not isinstance(refit, (bool, int, np.integer)):
            raise TypeError(
                f"`refit` must be a boolean or an integer equal or greater than 0. "
                f"Got {refit}."
            )
        if isinstance(refit, (int, np.integer)) and not isinstance(refit, bool) and refit < 0:
            raise TypeError(
                f"`refit` must be a boolean or an integer equal or greater than 0. "
                f"Got {refit}."
            )
        if not isinstance(fixed_train_size, bool):
            raise TypeError(
                f"`fixed_train_size` must be a boolean: `True`, `False`. "
                f"Got {fixed_train_size}."
            )
        if not isinstance(gap, (int, np.integer)) or gap < 0:
            raise ValueError(
                f"`gap` must be an integer greater than or equal to 0. Got {gap}."
            )
        if skip_folds is not None:
            if not isinstance(skip_folds, (int, np.integer, list, type(None))):
                raise TypeError(
                    f"`skip_folds` must be an integer greater than 0, a list of "
                    f"integers or `None`. Got {skip_folds}."
                )
            if isinstance(skip_folds, (int, np.integer)) and skip_folds < 1:
                raise ValueError(
                    f"`skip_folds` must be an integer greater than 0, a list of "
                    f"integers or `None`. Got {skip_folds}."
                )
            if isinstance(skip_folds, list) and any([x < 1 for x in skip_folds]):
                raise ValueError(
                    f"`skip_folds` list must contain integers greater than or "
                    f"equal to 1. The first fold is always needed to train the "
                    f"forecaster. Got {skip_folds}."
                ) 
        if not isinstance(allow_incomplete_fold, bool):
            raise TypeError(
                f"`allow_incomplete_fold` must be a boolean: `True`, `False`. "
                f"Got {allow_incomplete_fold}."
            )

    if cv_name == "OneStepAheadFold":
        if not isinstance(initial_train_size, (int, np.integer, str, pd.Timestamp)):
            raise ValueError(
                f"`initial_train_size` must be an integer greater than 0, a date "
                f"string, or a pandas Timestamp. Got {initial_train_size}."
            )
        if isinstance(initial_train_size, (int, np.integer)) and initial_train_size < 1:
            raise ValueError(
                f"`initial_train_size` must be an integer greater than 0, "
                f"a date string, or a pandas Timestamp. Got {initial_train_size}."
            )

    if (
        not isinstance(window_size, (int, np.integer, pd.DateOffset, type(None)))
        or isinstance(window_size, (int, np.integer))
        and window_size < 1
    ):
        raise ValueError(
            f"`window_size` must be an integer greater than 0. Got {window_size}."
        )

    if differentiation is not None:
        if not isinstance(differentiation, (int, np.integer)) or differentiation < 0:
            raise ValueError(
                f"`differentiation` must be None or an integer greater than or "
                f"equal to 0. Got {differentiation}."
            )

    if not isinstance(return_all_indexes, bool):
        raise TypeError(
            f"`return_all_indexes` must be a boolean: `True`, `False`. "
            f"Got {return_all_indexes}."
        )

    if not isinstance(verbose, bool):
        raise TypeError(
            f"`verbose` must be a boolean: `True`, `False`. "
            f"Got {verbose}."
        )

_extract_index

_extract_index(X)

Extracts and returns the index from the input data X.

Parameters:

Name	Type	Description	Default
X	pandas Series, pandas DataFrame, pandas Index, dict	Time series data or index to split.	required

Returns:

Name	Type	Description
idx	pandas Index	Index extracted from the input data.

Source code in skforecast\model_selection\_split.py

def _extract_index(
    self,
    X: pd.Series | pd.DataFrame | pd.Index | dict[str, pd.Series | pd.DataFrame]
) -> pd.Index:
    """
    Extracts and returns the index from the input data X.

    Parameters
    ----------
    X : pandas Series, pandas DataFrame, pandas Index, dict
        Time series data or index to split.

    Returns
    -------
    idx : pandas Index
        Index extracted from the input data.

    """

    if isinstance(X, (pd.Series, pd.DataFrame)):
        idx = X.index
    elif isinstance(X, dict):
        indexes_freq = set()
        not_valid_index = []
        min_index = []
        max_index = []
        for k, v in X.items():
            if v is None:
                continue

            idx = v.index
            if isinstance(idx, pd.DatetimeIndex):
                indexes_freq.add(idx.freq)
            elif isinstance(idx, pd.RangeIndex):
                indexes_freq.add(idx.step)
            else:
                not_valid_index.append(k)

            min_index.append(idx[0])
            max_index.append(idx[-1])

        if not_valid_index:
            raise TypeError(
                f"If `X` is a dictionary, all series must have a Pandas "
                f"RangeIndex or DatetimeIndex with the same step/frequency. "
                f"Review series: {not_valid_index}"
            )

        if None in indexes_freq:
            raise ValueError(
                "If `X` is a dictionary, all series must have a Pandas "
                "RangeIndex or DatetimeIndex with the same step/frequency. "
                "Found series with no frequency or step."
            )
        if not len(indexes_freq) == 1:
            raise ValueError(
                f"If `X` is a dictionary, all series must have a Pandas "
                f"RangeIndex or DatetimeIndex with the same step/frequency. "
                f"Found frequencies: {sorted(indexes_freq)}"
            )

        if isinstance(idx, pd.DatetimeIndex):
            idx = pd.date_range(
                start=min(min_index), end=max(max_index), freq=indexes_freq.pop()
            )
        else:
            idx = pd.RangeIndex(
                start=min(min_index), stop=max(max_index) + 1, step=indexes_freq.pop()
            )
    else:
        idx = X

    return idx

set_params

set_params(params)

Set the parameters of the Fold object. Before overwriting the current parameters, the input parameters are validated to ensure correctness.

Parameters:

Name	Type	Description	Default
params	dict	Dictionary with the parameters to set.	required

Returns:

Type	Description
None

Source code in skforecast\model_selection\_split.py

def set_params(
    self, 
    params: dict
) -> None:
    """
    Set the parameters of the Fold object. Before overwriting the current 
    parameters, the input parameters are validated to ensure correctness.

    Parameters
    ----------
    params : dict
        Dictionary with the parameters to set.

    Returns
    -------
    None

    """

    if not isinstance(params, dict):
        raise TypeError(
            f"`params` must be a dictionary. Got {type(params)}."
        )

    current_params = dict(vars(self))
    unknown_params = set(params.keys()) - set(current_params.keys())
    if unknown_params:
        warnings.warn(
            f"Unknown parameters: {unknown_params}. They have been ignored.",
            IgnoredArgumentWarning
        )

    filtered_params = {k: v for k, v in params.items() if k in current_params}
    updated_params = {'cv_name': type(self).__name__, **current_params, **filtered_params}

    self._validate_params(**updated_params)
    for key, value in updated_params.items():
        setattr(self, key, value)

skforecast.model_selection._split.TimeSeriesFold

TimeSeriesFold(
    steps,
    initial_train_size=None,
    fold_stride=None,
    window_size=None,
    differentiation=None,
    refit=False,
    fixed_train_size=True,
    gap=0,
    skip_folds=None,
    allow_incomplete_fold=True,
    return_all_indexes=False,
    verbose=True,
)

Bases: BaseFold

Class to split time series data into train and test folds. When used within a backtesting or hyperparameter search, the arguments 'initial_train_size', 'window_size' and 'differentiation' are not required as they are automatically set by the backtesting or hyperparameter search functions.

Parameters:

Name	Type	Description	Default
steps	int	Number of observations used to be predicted in each fold. This is also commonly referred to as the forecast horizon or test size.	required
initial_train_size	int, str, pandas Timestamp	Number of observations used for initial training. If None or 0, the initial forecaster is not trained in the first fold. If an integer, the number of observations used for initial training. If a date string or pandas Timestamp, it is the last date included in the initial training set.	None
fold_stride	int	Number of observations that the start of the test set advances between consecutive folds. If None, it defaults to the same value as steps, meaning that folds are placed back-to-back without overlap. If fold_stride < steps, test sets overlap and multiple forecasts will be generated for the same observations. If fold_stride > steps, gaps are left between consecutive test sets. New in version 0.18.0	None
window_size	int	Number of observations needed to generate the autoregressive predictors.	None
differentiation	int	Number of observations to use for differentiation. This is used to extend the last_window as many observations as the differentiation order.	None
refit	(bool, int)	Whether to refit the forecaster in each fold. If True, the forecaster is refitted in each fold. If False, the forecaster is trained only in the first fold. If an integer, the forecaster is trained in the first fold and then refitted every refit folds.	False
fixed_train_size	bool	Whether the training size is fixed or increases in each fold.	True
gap	int	Number of observations between the end of the training set and the start of the test set.	0
skip_folds	(int, list)	Number of folds to skip. If an integer, every 'skip_folds'-th is returned. If a list, the indexes of the folds to skip. For example, if skip_folds=3 and there are 10 folds, the returned folds are 0, 3, 6, and 9. If skip_folds=[1, 2, 3], the returned folds are 0, 4, 5, 6, 7, 8, and 9.	None
allow_incomplete_fold	bool	Whether to allow the last fold to include fewer observations than steps. If False, the last fold is excluded if it is incomplete.	True
return_all_indexes	bool	Whether to return all indexes or only the start and end indexes of each fold.	False
verbose	bool	Whether to print information about generated folds.	True

Attributes:

Name	Type	Description
steps	int	Number of observations used to be predicted in each fold. This is also commonly referred to as the forecast horizon or test size.
initial_train_size	int	Number of observations used for initial training. If None or 0, the initial forecaster is not trained in the first fold.
fold_stride	int	Number of observations that the start of the test set advances between consecutive folds.
overlapping_folds	bool	Whether the folds overlap.
window_size	int	Number of observations needed to generate the autoregressive predictors.
differentiation	int	Number of observations to use for differentiation. This is used to extend the last_window as many observations as the differentiation order.
refit	(bool, int)	Whether to refit the forecaster in each fold.
fixed_train_size	bool	Whether the training size is fixed or increases in each fold.
gap	int	Number of observations between the end of the training set and the start of the test set.
skip_folds	(int, list)	Number of folds to skip.
allow_incomplete_fold	bool	Whether to allow the last fold to include fewer observations than steps.
return_all_indexes	bool	Whether to return all indexes or only the start and end indexes of each fold.
verbose	bool	Whether to print information about generated folds.

Notes

Returned values are the positions of the observations and not the actual values of the index, so they can be used to slice the data directly using iloc. For example, if the input series is X = [10, 11, 12, 13, 14, 15, 16, 17, 18, 19], the initial_train_size = 3, window_size = 2, steps = 4, and gap = 1, the output of the first fold will be: [0, [0, 3], [1, 3], [3, 8], [4, 8], True].

The first element is the fold number, the first list [0, 3] indicates that the training set goes from the first to the third observation. The second list [1, 3] indicates that the last window seen by the forecaster during training goes from the second to the third observation. The third list [3, 8] indicates that the test set goes from the fourth to the eighth observation. The fourth list [4, 8] indicates that the test set including the gap goes from the fifth to the eighth observation. The boolean True indicates that the forecaster will be trained in this fold.

Following the python convention, the start index is inclusive and the end index is exclusive. This means that the last index is not included in the slice.

As an example, with initial_train_size=50, steps=30, and fold_stride=7, the first test fold will cover observations [50, 80), the second fold [57, 87), and the third fold [64, 94). This configuration produces multiple forecasts for the same observations, which is often desirable in rolling-origin evaluation.

Methods:

Name	Description
split	Split the time series data into train and test folds.

Source code in skforecast\model_selection\_split.py

def __init__(
    self,
    steps: int,
    initial_train_size: int | str | pd.Timestamp | None = None,
    fold_stride: int | None = None,
    window_size: int | None = None,
    differentiation: int | None = None,
    refit: bool | int = False,
    fixed_train_size: bool = True,
    gap: int = 0,
    skip_folds: int | list[int] | None = None,
    allow_incomplete_fold: bool = True,
    return_all_indexes: bool = False,
    verbose: bool = True
) -> None:

    super().__init__(
        steps                 = steps,
        initial_train_size    = initial_train_size,
        fold_stride           = fold_stride,
        window_size           = window_size,
        differentiation       = differentiation,
        refit                 = refit,
        fixed_train_size      = fixed_train_size,
        gap                   = gap,
        skip_folds            = skip_folds,
        allow_incomplete_fold = allow_incomplete_fold,
        return_all_indexes    = return_all_indexes,
        verbose               = verbose
    )

    self.steps                 = steps
    self.fold_stride           = fold_stride if fold_stride is not None else steps
    self.overlapping_folds     = self.fold_stride < self.steps
    self.refit                 = refit
    self.fixed_train_size      = fixed_train_size
    self.gap                   = gap
    self.skip_folds            = skip_folds
    self.allow_incomplete_fold = allow_incomplete_fold

Attributes

steps instance-attribute

steps = steps

fold_stride instance-attribute

fold_stride = (
    fold_stride if fold_stride is not None else steps
)

overlapping_folds instance-attribute

overlapping_folds = fold_stride < steps

refit instance-attribute

refit = refit

fixed_train_size instance-attribute

fixed_train_size = fixed_train_size

gap instance-attribute

gap = gap

skip_folds instance-attribute

skip_folds = skip_folds

allow_incomplete_fold instance-attribute

allow_incomplete_fold = allow_incomplete_fold

Functions

_repr_html_

_repr_html_()

HTML representation of the object. The "General Information" section is expanded by default.

Source code in skforecast\model_selection\_split.py

def _repr_html_(self) -> str:
    """
    HTML representation of the object.
    The "General Information" section is expanded by default.
    """

    style, unique_id = get_style_repr_html()
    content = f"""
    <div class="container-{unique_id}">
        <p style="font-size: 1.5em; font-weight: bold; margin-block-start: 0.83em; margin-block-end: 0.83em;">{type(self).__name__}</p>
        <details open>
            <summary>General Information</summary>
            <ul>
                <li><strong>Initial train size:</strong> {self.initial_train_size}</li>
                <li><strong>Steps:</strong> {self.steps}</li>
                <li><strong>Fold stride:</strong> {self.fold_stride}</li>
                <li><strong>Overlapping folds:</strong> {self.overlapping_folds}</li>
                <li><strong>Window size:</strong> {self.window_size}</li>
                <li><strong>Differentiation:</strong> {self.differentiation}</li>
                <li><strong>Refit:</strong> {self.refit}</li>
                <li><strong>Fixed train size:</strong> {self.fixed_train_size}</li>
                <li><strong>Gap:</strong> {self.gap}</li>
                <li><strong>Skip folds:</strong> {self.skip_folds}</li>
                <li><strong>Allow incomplete fold:</strong> {self.allow_incomplete_fold}</li>
                <li><strong>Return all indexes:</strong> {self.return_all_indexes}</li>
            </ul>
        </details>
        <p>
            <a href="https://skforecast.org/{__version__}/api/model_selection.html#skforecast.model_selection._split.TimeSeriesFold">&#128712 <strong>API Reference</strong></a>
            &nbsp;&nbsp;
            <a href="https://skforecast.org/{__version__}/user_guides/backtesting.html#timeseriesfold">&#128462 <strong>User Guide</strong></a>
        </p>
    </div>
    """

    return style + content

split

split(X, as_pandas=False)

Split the time series data into train and test folds.

Parameters:

Name	Type	Description	Default
X	pandas Series, pandas DataFrame, pandas Index, dict	Time series data or index to split.	required
as_pandas	bool	If True, the folds are returned as a DataFrame. This is useful to visualize the folds in a more interpretable way.	False

Returns:

Name	Type	Description
folds	list, pandas DataFrame	A list of lists containing the indices (position) for each fold. Each list contains 4 lists and a boolean with the following information: fold: fold number. [train_start, train_end]: list with the start and end positions of the training set. [last_window_start, last_window_end]: list with the start and end positions of the last window seen by the forecaster during training. The last window is used to generate the lags used as predictors. If differentiation is included, the interval is extended as many observations as the differentiation order. If the argument window_size is None, this list is empty. [test_start, test_end]: list with the start and end positions of the test set. These are the observations used to evaluate the forecaster. [test_start_with_gap, test_end_with_gap]: list with the start and end positions of the test set including the gap. The gap is the number of observations between the end of the training set and the start of the test set. fit_forecaster: boolean indicating whether the forecaster should be fitted in this fold. It is important to note that the returned values are the positions of the observations and not the actual values of the index, so they can be used to slice the data directly using iloc. If as_pandas is True, the folds are returned as a DataFrame with the following columns: 'fold', 'train_start', 'train_end', 'last_window_start', 'last_window_end', 'test_start', 'test_end', 'test_start_with_gap', 'test_end_with_gap', 'fit_forecaster'. Following the python convention, the start index is inclusive and the end index is exclusive. This means that the last index is not included in the slice.

Source code in skforecast\model_selection\_split.py

def split(
    self,
    X: pd.Series | pd.DataFrame | pd.Index | dict[str, pd.Series | pd.DataFrame],
    as_pandas: bool = False
) -> list | pd.DataFrame:
    """
    Split the time series data into train and test folds.

    Parameters
    ----------
    X : pandas Series, pandas DataFrame, pandas Index, dict
        Time series data or index to split.
    as_pandas : bool, default False
        If True, the folds are returned as a DataFrame. This is useful to visualize
        the folds in a more interpretable way.

    Returns
    -------
    folds : list, pandas DataFrame
        A list of lists containing the indices (position) for each fold. Each list
        contains 4 lists and a boolean with the following information:

        - fold: fold number.
        - [train_start, train_end]: list with the start and end positions of the
        training set.
        - [last_window_start, last_window_end]: list with the start and end positions
        of the last window seen by the forecaster during training. The last window
        is used to generate the lags used as predictors. If `differentiation` is
        included, the interval is extended as many observations as the
        differentiation order. If the argument `window_size` is `None`, this list is
        empty.
        - [test_start, test_end]: list with the start and end positions of the test
        set. These are the observations used to evaluate the forecaster.
        - [test_start_with_gap, test_end_with_gap]: list with the start and end
        positions of the test set including the gap. The gap is the number of
        observations between the end of the training set and the start of the test
        set.
        - fit_forecaster: boolean indicating whether the forecaster should be fitted
        in this fold.

        It is important to note that the returned values are the positions of the
        observations and not the actual values of the index, so they can be used to
        slice the data directly using iloc.

        If `as_pandas` is `True`, the folds are returned as a DataFrame with the
        following columns: 'fold', 'train_start', 'train_end', 'last_window_start',
        'last_window_end', 'test_start', 'test_end', 'test_start_with_gap',
        'test_end_with_gap', 'fit_forecaster'.

        Following the python convention, the start index is inclusive and the end
        index is exclusive. This means that the last index is not included in the
        slice.

    """

    if not isinstance(X, (pd.Series, pd.DataFrame, pd.Index, dict)):
        raise TypeError(
            f"X must be a pandas Series, DataFrame, Index or a dictionary. "
            f"Got {type(X)}."
        )

    window_size_as_date_offset = isinstance(self.window_size, pd.tseries.offsets.DateOffset)
    if window_size_as_date_offset:
        # Calculate the window_size in steps. This is not a exact calculation
        # because the offset follows the calendar rules and the distance between
        # two dates may not be constant.
        first_valid_index = X.index[-1] - self.window_size
        try:
            window_size_idx_start = X.index.get_loc(first_valid_index)
            window_size_idx_end = X.index.get_loc(X.index[-1])
            self.window_size = window_size_idx_end - window_size_idx_start
        except KeyError:
            raise ValueError(
                f"The length of `y` ({len(X)}), must be greater than or equal "
                f"to the window size ({self.window_size}). This is because "
                f"the offset (forecaster.offset) is larger than the available "
                f"data. Try to decrease the size of the offset (forecaster.offset), "
                f"the number of `n_offsets` (forecaster.n_offsets) or increase the "
                f"size of `y`."
            )

    if self.initial_train_size is None:
        if self.window_size is None:
            raise ValueError(
                "To use split method when `initial_train_size` is None, "
                "`window_size` must be an integer greater than 0. "
                "Although no initial training is done and all data is used to "
                "evaluate the model, the first `window_size` observations are "
                "needed to create the initial predictors. Got `window_size` = None."
            )
        if self.refit:
            raise ValueError(
                "`refit` is only allowed when `initial_train_size` is not `None`. "
                "Set `refit` to `False` if you want to use `initial_train_size = None`."
            )
        externally_fitted = True
        self.initial_train_size = self.window_size  # Reset to None later
    else:
        if self.window_size is None:
            warnings.warn(
                "Last window cannot be calculated because `window_size` is None.",
                IgnoredArgumentWarning
            )
        externally_fitted = False

    index = self._extract_index(X)
    idx = range(len(index))
    folds = []
    i = 0

    self.initial_train_size = date_to_index_position(
                                  index        = index, 
                                  date_input   = self.initial_train_size, 
                                  method       = 'validation',
                                  date_literal = 'initial_train_size'
                              )

    if window_size_as_date_offset:
        if self.initial_train_size is not None:
            if self.initial_train_size < self.window_size:
                raise ValueError(
                    f"If `initial_train_size` is an integer, it must be greater than "
                    f"the `window_size` of the forecaster ({self.window_size}) "
                    f"and smaller than the length of the series ({len(X)}). If "
                    f"it is a date, it must be within this range of the index."
                )

    if self.allow_incomplete_fold:
        # At least one observation after the gap to allow incomplete fold
        if len(index) <= self.initial_train_size + self.gap:
            raise ValueError(
                f"The time series must have more than `initial_train_size + gap` "
                f"observations to create at least one fold.\n"
                f"    Time series length: {len(index)}\n"
                f"    Required > {self.initial_train_size + self.gap}\n"
                f"    initial_train_size: {self.initial_train_size}\n"
                f"    gap: {self.gap}\n"
            )
    else:
        # At least one complete fold
        if len(index) < self.initial_train_size + self.gap + self.steps:
            raise ValueError(
                f"The time series must have at least `initial_train_size + gap + steps` "
                f"observations to create a minimum of one complete fold "
                f"(allow_incomplete_fold=False).\n"
                f"    Time series length: {len(index)}\n"
                f"    Required >= {self.initial_train_size + self.gap + self.steps}\n"
                f"    initial_train_size: {self.initial_train_size}\n"
                f"    gap: {self.gap}\n"
                f"    steps: {self.steps}\n"
            )

    while self.initial_train_size + (i * self.fold_stride) + self.gap < len(index):

        if self.refit:
            # NOTE: If `fixed_train_size` the train size doesn't increase but 
            # moves by `fold_stride` positions in each iteration. If `False`, 
            # the train size increases by `fold_stride` in each iteration.
            train_iloc_start = i * (self.fold_stride) if self.fixed_train_size else 0
            train_iloc_end = self.initial_train_size + i * (self.fold_stride)
            test_iloc_start = train_iloc_end
        else:
            # NOTE: The train size doesn't increase and doesn't move.
            train_iloc_start = 0
            train_iloc_end = self.initial_train_size
            test_iloc_start = self.initial_train_size + i * (self.fold_stride)

        if self.window_size is not None:
            last_window_iloc_start = test_iloc_start - self.window_size

        test_iloc_end = test_iloc_start + self.gap + self.steps

        partitions = [
            idx[train_iloc_start : train_iloc_end],
            idx[last_window_iloc_start : test_iloc_start] if self.window_size is not None else [],
            idx[test_iloc_start : test_iloc_end],
            idx[test_iloc_start + self.gap : test_iloc_end]
        ]
        folds.append(partitions)
        i += 1

    # NOTE: Delete all incomplete folds at the end if not allowed
    n_removed_folds = 0
    if not self.allow_incomplete_fold:
        # NOTE: While folds and the last "test_index_with_gap" is incomplete,
        # calculating len of range objects
        while folds and len(folds[-1][3]) < self.steps:
            folds.pop()
            n_removed_folds += 1

    # Replace partitions inside folds with length 0 with `None`
    folds = [
        [partition if len(partition) > 0 else None for partition in fold] 
        for fold in folds
    ]

    # Create a flag to know whether to train the forecaster
    if self.refit == 0:
        self.refit = False

    if isinstance(self.refit, bool):
        fit_forecaster = [self.refit] * len(folds)
        fit_forecaster[0] = True
    else:
        fit_forecaster = [False] * len(folds)
        for i in range(0, len(fit_forecaster), self.refit): 
            fit_forecaster[i] = True

    for i in range(len(folds)): 
        folds[i].insert(0, i)
        folds[i].append(fit_forecaster[i])
        if fit_forecaster[i] is False:
            folds[i][1] = folds[i - 1][1]

    index_to_skip = []
    if self.skip_folds is not None:
        if isinstance(self.skip_folds, (int, np.integer)) and self.skip_folds > 0:
            index_to_keep = np.arange(0, len(folds), self.skip_folds)
            index_to_skip = np.setdiff1d(np.arange(0, len(folds)), index_to_keep, assume_unique=True)
            index_to_skip = [int(x) for x in index_to_skip]  # Required since numpy 2.0
        if isinstance(self.skip_folds, list):
            index_to_skip = [i for i in self.skip_folds if i < len(folds)]

    if self.verbose:
        self._print_info(
            index              = index,
            folds              = folds,
            externally_fitted  = externally_fitted,
            n_removed_folds    = n_removed_folds,
            index_to_skip      = index_to_skip
        )

    folds = [fold for i, fold in enumerate(folds) if i not in index_to_skip]
    if not self.return_all_indexes:
        # NOTE: +1 to prevent iloc pandas from deleting the last observation
        folds = [
            [
                fold[0],
                [fold[1][0], fold[1][-1] + 1],
                (
                    [fold[2][0], fold[2][-1] + 1]
                    if self.window_size is not None
                    else []
                ),
                [fold[3][0], fold[3][-1] + 1],
                [fold[4][0], fold[4][-1] + 1],
                fold[5],
            ]
            for fold in folds
        ]

    if externally_fitted:
        self.initial_train_size = None
        folds[0][5] = False

    if as_pandas:
        if self.window_size is None:
            for fold in folds:
                fold[2] = [None, None]

        if not self.return_all_indexes:
            folds = pd.DataFrame(
                data = [[fold[0]] + list(itertools.chain(*fold[1:-1])) + [fold[-1]] for fold in folds],
                columns = [
                    'fold',
                    'train_start',
                    'train_end',
                    'last_window_start',
                    'last_window_end',
                    'test_start',
                    'test_end',
                    'test_start_with_gap',
                    'test_end_with_gap',
                    'fit_forecaster'
                ],
            )
        else:
            folds = pd.DataFrame(
                data = folds,
                columns = [
                    'fold',
                    'train_index',
                    'last_window_index',
                    'test_index',
                    'test_index_with_gap',
                    'fit_forecaster'
                ],
            )

    return folds

_print_info

_print_info(
    index,
    folds,
    externally_fitted,
    n_removed_folds,
    index_to_skip,
)

Print information about folds.

Parameters:

Name	Type	Description	Default
index	pandas Index	Index of the time series data.	required
folds	list	A list of lists containing the indices (position) for each fold.	required
externally_fitted	bool	Whether an already trained forecaster is to be used.	required
n_removed_folds	int	Number of folds removed.	required
index_to_skip	list	Indexes of folds to skip.	required

Returns:

Type	Description
None

Source code in skforecast\model_selection\_split.py

def _print_info(
    self,
    index: pd.Index,
    folds: list[list[int]],
    externally_fitted: bool,
    n_removed_folds: int,
    index_to_skip: list[int]
) -> None:
    """
    Print information about folds.

    Parameters
    ----------
    index : pandas Index
        Index of the time series data.
    folds : list
        A list of lists containing the indices (position) for each fold.
    externally_fitted : bool
        Whether an already trained forecaster is to be used.
    n_removed_folds : int
        Number of folds removed.
    index_to_skip : list
        Indexes of folds to skip.

    Returns
    -------
    None

    """

    print("Information of folds")
    print("--------------------")
    if externally_fitted:
        print(
            f"An already trained forecaster is to be used. Window size: "
            f"{self.window_size}"
        )
    else:
        if self.differentiation is None:
            print(
                f"Number of observations used for initial training: "
                f"{self.initial_train_size}"
            )
        else:
            print(
                f"Number of observations used for initial training: "
                f"{self.initial_train_size - self.differentiation}"
            )
            print(
                f"    First {self.differentiation} observation/s in training sets "
                f"are used for differentiation"
            )
    print(
        f"Number of observations used for backtesting: "
        f"{len(index) - self.initial_train_size}"
    )
    print(f"    Number of folds: {len(folds)}")
    print(
        f"    Number skipped folds: "
        f"{len(index_to_skip)} {index_to_skip if index_to_skip else ''}"
    )
    print(f"    Number of steps per fold: {self.steps}")
    if self.steps != self.fold_stride:
        print(f"    Number of steps to the next fold (fold stride): {self.fold_stride}")
    print(
        f"    Number of steps to exclude between last observed data "
        f"(last window) and predictions (gap): {self.gap}"
    )
    if n_removed_folds > 0:
        print(
            f"    The last {n_removed_folds} fold(s) have been excluded "
            f"because they were incomplete."
        )

    if len(folds[-1][4]) < self.steps:
        print(f"    Last fold only includes {len(folds[-1][4])} observations.")

    print("")

    if self.differentiation is None:
        differentiation = 0
    else:
        differentiation = self.differentiation

    for i, fold in enumerate(folds):
        is_fold_skipped   = i in index_to_skip
        has_training      = fold[-1] if i != 0 else True
        training_start    = (
            index[fold[1][0] + differentiation] if fold[1] is not None else None
        )
        training_end      = index[fold[1][-1]] if fold[1] is not None else None
        training_length   = (
            len(fold[1]) - differentiation if fold[1] is not None else 0
        )
        validation_start  = index[fold[4][0]]
        validation_end    = index[fold[4][-1]]
        validation_length = len(fold[4])

        print(f"Fold: {i}")
        if is_fold_skipped:
            print("    Fold skipped")
        elif not externally_fitted and has_training:
            print(
                f"    Training:   {training_start} -- {training_end}  "
                f"(n={training_length})"
            )
            print(
                f"    Validation: {validation_start} -- {validation_end}  "
                f"(n={validation_length})"
            )
        else:
            print("    Training:   No training in this fold")
            print(
                f"    Validation: {validation_start} -- {validation_end}  "
                f"(n={validation_length})"
            )

    print("")

skforecast.model_selection._split.OneStepAheadFold

OneStepAheadFold(
    initial_train_size,
    window_size=None,
    differentiation=None,
    return_all_indexes=False,
    verbose=True,
)

Bases: BaseFold

Class to split time series data into train and test folds for one-step-ahead forecasting.

Parameters:

Name	Type	Description	Default
initial_train_size	int, str, pandas Timestamp	Number of observations used for initial training. If an integer, the number of observations used for initial training. If a date string or pandas Timestamp, it is the last date included in the initial training set.	required
window_size	int	Number of observations needed to generate the autoregressive predictors.	None
differentiation	int	Number of observations to use for differentiation. This is used to extend the last_window as many observations as the differentiation order.	None
return_all_indexes	bool	Whether to return all indexes or only the start and end indexes of each fold.	False
verbose	bool	Whether to print information about generated folds.	True

Attributes:

Name	Type	Description
initial_train_size	int	Number of observations used for initial training.
window_size	int	Number of observations needed to generate the autoregressive predictors.
differentiation	int	Number of observations to use for differentiation. This is used to extend the last_window as many observations as the differentiation order.
return_all_indexes	bool	Whether to return all indexes or only the start and end indexes of each fold.
verbose	bool	Whether to print information about generated folds.

Methods:

Name	Description
split	Split the time series data into train and test folds.

Source code in skforecast\model_selection\_split.py

def __init__(
    self,
    initial_train_size: int | str | pd.Timestamp,
    window_size: int | None = None,
    differentiation: int | None = None,
    return_all_indexes: bool = False,
    verbose: bool = True
) -> None:

    super().__init__(
        initial_train_size = initial_train_size,
        window_size        = window_size,
        differentiation    = differentiation,
        return_all_indexes = return_all_indexes,
        verbose            = verbose
    )

Functions

_repr_html_

_repr_html_()

HTML representation of the object. The "General Information" section is expanded by default.

Source code in skforecast\model_selection\_split.py

def _repr_html_(self) -> str:
    """
    HTML representation of the object.
    The "General Information" section is expanded by default.
    """

    style, unique_id = get_style_repr_html()
    content = f"""
    <div class="container-{unique_id}">
        <p style="font-size: 1.5em; font-weight: bold; margin-block-start: 0.83em; margin-block-end: 0.83em;">{type(self).__name__}</p>
        <details open>
            <summary>General Information</summary>
            <ul>
                <li><strong>Initial train size:</strong> {self.initial_train_size}</li>
                <li><strong>Window size:</strong> {self.window_size}</li>
                <li><strong>Differentiation:</strong> {self.differentiation}</li>
                <li><strong>Return all indexes:</strong> {self.return_all_indexes}</li>
            </ul>
        </details>
        <p>
            <a href="https://skforecast.org/{__version__}/api/model_selection.html#skforecast.model_selection._split.OneStepAheadFold">&#128712 <strong>API Reference</strong></a>
            &nbsp;&nbsp;
            <a href="https://skforecast.org/{__version__}/faq/parameters-search-backtesting-vs-one-step-ahead.html">&#128462 <strong>User Guide</strong></a>
        </p>
    </div>
    """

    return style + content

split

split(X, as_pandas=False, externally_fitted=None)

Split the time series data into train and test folds.

Parameters:

Name	Type	Description	Default
X	pandas Series, DataFrame, Index, or dictionary	Time series data or index to split.	required
as_pandas	bool	If True, the folds are returned as a DataFrame. This is useful to visualize the folds in a more interpretable way.	False
externally_fitted	Any	This argument is not used in this class. It is included for API consistency.	None

Returns:

Name	Type	Description
fold	list, pandas DataFrame	A list of lists containing the indices (position) of the fold. The list contains 2 lists with the following information: fold: fold number. [train_start, train_end]: list with the start and end positions of the training set. [test_start, test_end]: list with the start and end positions of the test set. These are the observations used to evaluate the forecaster. fit_forecaster: boolean indicating whether the forecaster should be fitted in this fold. It is important to note that the returned values are the positions of the observations and not the actual values of the index, so they can be used to slice the data directly using iloc. If as_pandas is True, the folds are returned as a DataFrame with the following columns: 'fold', 'train_start', 'train_end', 'test_start', 'test_end', 'fit_forecaster'. Following the python convention, the start index is inclusive and the end index is exclusive. This means that the last index is not included in the slice.

Source code in skforecast\model_selection\_split.py

def split(
    self,
    X: pd.Series | pd.DataFrame | pd.Index | dict[str, pd.Series | pd.DataFrame],
    as_pandas: bool = False,
    externally_fitted: Any = None
) -> list | pd.DataFrame:
    """
    Split the time series data into train and test folds.

    Parameters
    ----------
    X : pandas Series, DataFrame, Index, or dictionary
        Time series data or index to split.
    as_pandas : bool, default False
        If True, the folds are returned as a DataFrame. This is useful to visualize
        the folds in a more interpretable way.
    externally_fitted : Any
        This argument is not used in this class. It is included for API consistency.

    Returns
    -------
    fold : list, pandas DataFrame
        A list of lists containing the indices (position) of the fold. The list
        contains 2 lists with the following information:

        - fold: fold number.
        - [train_start, train_end]: list with the start and end positions of the
        training set.
        - [test_start, test_end]: list with the start and end positions of the test
        set. These are the observations used to evaluate the forecaster.
        - fit_forecaster: boolean indicating whether the forecaster should be fitted
        in this fold.

        It is important to note that the returned values are the positions of the
        observations and not the actual values of the index, so they can be used to
        slice the data directly using iloc.

        If `as_pandas` is `True`, the folds are returned as a DataFrame with the
        following columns: 'fold', 'train_start', 'train_end', 'test_start', 
        'test_end', 'fit_forecaster'.

        Following the python convention, the start index is inclusive and the end
        index is exclusive. This means that the last index is not included in the
        slice.

    """

    if not isinstance(X, (pd.Series, pd.DataFrame, pd.Index, dict)):
        raise TypeError(
            f"X must be a pandas Series, DataFrame, Index or a dictionary. "
            f"Got {type(X)}."
        )

    index = self._extract_index(X)

    self.initial_train_size = date_to_index_position(
                                  index        = index, 
                                  date_input   = self.initial_train_size, 
                                  method       = 'validation',
                                  date_literal = 'initial_train_size'
                              )

    fold = [
        0,
        [0, self.initial_train_size - 1],
        [self.initial_train_size, len(X)],
        True
    ]

    if self.verbose:
        self._print_info(index=index, fold=fold)

    # NOTE: +1 to prevent iloc pandas from deleting the last observation
    if self.return_all_indexes:
        fold = [
            fold[0], 
            [range(fold[1][0], fold[1][1] + 1)],
            [range(fold[2][0], fold[2][1])],
            fold[3]
        ]
    else:
        fold = [
            fold[0], 
            [fold[1][0], fold[1][1] + 1],
            [fold[2][0], fold[2][1]],
            fold[3]
        ]

    if as_pandas:
        if not self.return_all_indexes:
            fold = pd.DataFrame(
                data = [[fold[0]] + list(itertools.chain(*fold[1:-1])) + [fold[-1]]],
                columns = [
                    'fold',
                    'train_start',
                    'train_end',
                    'test_start',
                    'test_end',
                    'fit_forecaster'
                ],
            )
        else:
            fold = pd.DataFrame(
                data = [fold],
                columns = [
                    'fold',
                    'train_index',
                    'test_index',
                    'fit_forecaster'
                ],
            )

    return fold

_print_info

_print_info(index, fold)

Print information about folds.

Parameters:

Name	Type	Description	Default
index	pandas Index	Index of the time series data.	required
fold	list	A list of lists containing the indices (position) of the fold.	required

Returns:

Type	Description
None

Source code in skforecast\model_selection\_split.py

def _print_info(
    self,
    index: pd.Index,
    fold: list[list[int]]
) -> None:
    """
    Print information about folds.

    Parameters
    ----------
    index : pandas Index
        Index of the time series data.
    fold : list
        A list of lists containing the indices (position) of the fold.

    Returns
    -------
    None

    """

    if self.differentiation is None:
        differentiation = 0
    else:
        differentiation = self.differentiation

    initial_train_size = self.initial_train_size - differentiation
    test_length = len(index) - (initial_train_size + differentiation)

    print("Information of folds")
    print("--------------------")
    print(
        f"Number of observations in train: {initial_train_size}"
    )
    if self.differentiation is not None:
        print(
            f"    First {differentiation} observation/s in training set "
            f"are used for differentiation"
        )
    print(
        f"Number of observations in test: {test_length}"
    )

    training_start = index[fold[1][0] + differentiation]
    training_end = index[fold[1][-1]]
    test_start  = index[fold[2][0]]
    test_end    = index[fold[2][-1] - 1]

    print(
        f"Training : {training_start} -- {training_end} (n={initial_train_size})"
    )
    print(
        f"Test     : {test_start} -- {test_end} (n={test_length})"
    )
    print("")

skforecast.model_selection._utils.initialize_lags_grid

initialize_lags_grid(forecaster, lags_grid=None)

Initialize lags grid and lags label for model selection.

Parameters:

Name	Type	Description	Default
forecaster	Forecaster	Forecaster model. ForecasterRecursive, ForecasterDirect, ForecasterRecursiveMultiSeries, ForecasterDirectMultiVariate.	required
lags_grid	(list, dict)	Lists of lags to try, containing int, lists, numpy ndarray, or range objects. If dict, the keys are used as labels in the results DataFrame, and the values are used as the lists of lags to try.	None

Returns:

Name	Type	Description
lags_grid	dict	Dictionary with lags configuration for each iteration.
lags_label	str	Label for lags representation in the results object.

Source code in skforecast\model_selection\_utils.py

def initialize_lags_grid(
    forecaster: object, 
    lags_grid: (
        list[int | list[int] | np.ndarray[int] | range[int]]
        | dict[str, list[int | list[int] | np.ndarray[int] | range[int]]]
        | None
    ) = None,
) -> tuple[dict[str, int], str]:
    """
    Initialize lags grid and lags label for model selection. 

    Parameters
    ----------
    forecaster : Forecaster
        Forecaster model. ForecasterRecursive, ForecasterDirect, 
        ForecasterRecursiveMultiSeries, ForecasterDirectMultiVariate.
    lags_grid : list, dict, default None
        Lists of lags to try, containing int, lists, numpy ndarray, or range 
        objects. If `dict`, the keys are used as labels in the `results` 
        DataFrame, and the values are used as the lists of lags to try.

    Returns
    -------
    lags_grid : dict
        Dictionary with lags configuration for each iteration.
    lags_label : str
        Label for lags representation in the results object.

    """

    if not isinstance(lags_grid, (list, dict, type(None))):
        raise TypeError(
            f"`lags_grid` argument must be a list, dict or None. "
            f"Got {type(lags_grid)}."
        )

    lags_label = 'values'
    if isinstance(lags_grid, list):
        lags_grid = {f'{lags}': lags for lags in lags_grid}
    elif lags_grid is None:
        lags = [int(lag) for lag in forecaster.lags]  # Required since numpy 2.0
        lags_grid = {f'{lags}': lags}
    else:
        lags_label = 'keys'

    return lags_grid, lags_label

skforecast.model_selection._utils.check_backtesting_input

check_backtesting_input(
    forecaster,
    cv,
    metric,
    add_aggregated_metric=True,
    y=None,
    series=None,
    exog=None,
    interval=None,
    interval_method="bootstrapping",
    alpha=None,
    n_boot=250,
    use_in_sample_residuals=True,
    use_binned_residuals=True,
    random_state=123,
    return_predictors=False,
    freeze_params=True,
    n_jobs="auto",
    show_progress=True,
    suppress_warnings=False,
)

This is a helper function to check most inputs of backtesting functions in modules model_selection.

Parameters:

Name	Type	Description	Default
forecaster	Forecaster	Forecaster model.	required
cv	TimeSeriesFold	TimeSeriesFold object with the information needed to split the data into folds.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model.	required
add_aggregated_metric	bool	If True, the aggregated metrics (average, weighted average and pooling) over all levels are also returned (only multiseries).	True
y	pandas Series	Training time series for uni-series forecasters.	None
series	pandas DataFrame, dict	Training time series for multi-series forecasters.	None
exog	pandas Series, pandas DataFrame, dict	Exogenous variables.	None
interval	(float, list, tuple, str, object)	Specifies whether probabilistic predictions should be estimated and the method to use. The following options are supported: If float, represents the nominal (expected) coverage (between 0 and 1). For instance, interval=0.95 corresponds to [2.5, 97.5] percentiles. If list or tuple: Sequence of percentiles to compute, each value must be between 0 and 100 inclusive. For example, a 95% confidence interval can be specified as interval = [2.5, 97.5] or multiple percentiles (e.g. 10, 50 and 90) as interval = [10, 50, 90]. If 'bootstrapping' (str): n_boot bootstrapping predictions will be generated. If scipy.stats distribution object, the distribution parameters will be estimated for each prediction. If None, no probabilistic predictions are estimated.	None
interval_method	str	Technique used to estimate prediction intervals. Available options: 'bootstrapping': Bootstrapping is used to generate prediction intervals. 'conformal': Employs the conformal prediction split method for interval estimation.	'bootstrapping'
alpha	float	The confidence intervals used in ForecasterStats are (1 - alpha) %.	None
n_boot	int	Number of bootstrapping iterations to perform when estimating prediction intervals.	`250`
use_in_sample_residuals	bool	If True, residuals from the training data are used as proxy of prediction error to create prediction intervals. If False, out_sample_residuals are used if they are already stored inside the forecaster.	True
use_binned_residuals	bool	If True, residuals are selected based on the predicted values (binned selection). If False, residuals are selected randomly.	True
random_state	int	Seed for the random number generator to ensure reproducibility.	`123`
return_predictors	bool	If True, the predictors used to make the predictions are also returned.	False
n_jobs	(int, 'auto')	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_fit_forecaster.	`'auto'`
freeze_params	bool	Determines whether to freeze the model parameters after the first fit for estimators that perform automatic model selection. If True, the model parameters found during the first fit (e.g., order and seasonal_order for Arima, or smoothing parameters for Ets) are reused in all subsequent refits. This avoids re-running the automatic selection procedure in each fold and reduces runtime. If False, automatic model selection is performed independently in each refit, allowing parameters to adapt across folds. This increases runtime and adds a params column to the output with the parameters selected per fold.	True
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the backtesting process. See skforecast.exceptions.warn_skforecast_categories for more information.	False

Returns:

Type	Description
None

Source code in skforecast\model_selection\_utils.py

def check_backtesting_input(
    forecaster: object,
    cv: object,
    metric: str | Callable | list[str | Callable],
    add_aggregated_metric: bool = True,
    y: pd.Series | None = None,
    series: pd.DataFrame | dict[str, pd.Series | pd.DataFrame] = None,
    exog: pd.Series | pd.DataFrame | dict[str, pd.Series | pd.DataFrame] | None = None,
    interval: float | list[float] | tuple[float] | str | object | None = None,
    interval_method: str = 'bootstrapping',    
    alpha: float | None = None,
    n_boot: int = 250,
    use_in_sample_residuals: bool = True,
    use_binned_residuals: bool = True,
    random_state: int = 123,
    return_predictors: bool = False,
    freeze_params: bool = True,
    n_jobs: int | str = 'auto',
    show_progress: bool = True,
    suppress_warnings: bool = False
) -> None:
    """
    This is a helper function to check most inputs of backtesting functions in 
    modules `model_selection`.

    Parameters
    ----------
    forecaster : Forecaster
        Forecaster model.
    cv : TimeSeriesFold
        TimeSeriesFold object with the information needed to split the data into folds.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.
    add_aggregated_metric : bool, default True
        If `True`, the aggregated metrics (average, weighted average and pooling)
        over all levels are also returned (only multiseries).
    y : pandas Series, default None
        Training time series for uni-series forecasters.
    series : pandas DataFrame, dict, default None
        Training time series for multi-series forecasters.
    exog : pandas Series, pandas DataFrame, dict, default None
        Exogenous variables.
    interval : float, list, tuple, str, object, default None
        Specifies whether probabilistic predictions should be estimated and the 
        method to use. The following options are supported:

        - If `float`, represents the nominal (expected) coverage (between 0 and 1). 
        For instance, `interval=0.95` corresponds to `[2.5, 97.5]` percentiles.
        - If `list` or `tuple`: Sequence of percentiles to compute, each value must 
        be between 0 and 100 inclusive. For example, a 95% confidence interval can 
        be specified as `interval = [2.5, 97.5]` or multiple percentiles (e.g. 10, 
        50 and 90) as `interval = [10, 50, 90]`.
        - If 'bootstrapping' (str): `n_boot` bootstrapping predictions will be generated.
        - If scipy.stats distribution object, the distribution parameters will
        be estimated for each prediction.
        - If None, no probabilistic predictions are estimated.
    interval_method : str, default 'bootstrapping'
        Technique used to estimate prediction intervals. Available options:

        + 'bootstrapping': Bootstrapping is used to generate prediction 
        intervals.
        + 'conformal': Employs the conformal prediction split method for 
        interval estimation.
    alpha : float, default None
        The confidence intervals used in ForecasterStats are (1 - alpha) %. 
    n_boot : int, default `250`
        Number of bootstrapping iterations to perform when estimating prediction
            intervals.
    use_in_sample_residuals : bool, default True
        If `True`, residuals from the training data are used as proxy of prediction 
        error to create prediction intervals.  If `False`, out_sample_residuals 
        are used if they are already stored inside the forecaster.
    use_binned_residuals : bool, default True
        If `True`, residuals are selected based on the predicted values 
        (binned selection).
        If `False`, residuals are selected randomly.
    random_state : int, default `123`
        Seed for the random number generator to ensure reproducibility.
    return_predictors : bool, default False
        If `True`, the predictors used to make the predictions are also returned.
    n_jobs : int, 'auto', default `'auto'`
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_fit_forecaster.
    freeze_params : bool, default True
        Determines whether to freeze the model parameters after the first fit
        for estimators that perform automatic model selection.

        - If `True`, the model parameters found during the first fit (e.g., order 
        and seasonal_order for Arima, or smoothing parameters for Ets) are reused
        in all subsequent refits. This avoids re-running the automatic selection
        procedure in each fold and reduces runtime.
        - If `False`, automatic model selection is performed independently in each
        refit, allowing parameters to adapt across folds. This increases runtime
        and adds a `params` column to the output with the parameters selected per
        fold.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings: bool, default False
        If `True`, skforecast warnings will be suppressed during the backtesting 
        process. See skforecast.exceptions.warn_skforecast_categories for more
        information.

    Returns
    -------
    None

    """

    forecaster_name = type(forecaster).__name__
    cv_name = type(cv).__name__

    if cv_name != "TimeSeriesFold":
        raise TypeError(f"`cv` must be a 'TimeSeriesFold' object. Got '{cv_name}'.")

    steps = cv.steps
    initial_train_size = cv.initial_train_size
    gap = cv.gap
    allow_incomplete_fold = cv.allow_incomplete_fold
    refit = cv.refit

    forecasters_uni = [
        "ForecasterRecursive",
        "ForecasterDirect",
        "ForecasterStats",
        "ForecasterEquivalentDate",
        "ForecasterRecursiveClassifier"
    ]
    forecasters_direct = [
        "ForecasterDirect",
        "ForecasterDirectMultiVariate",
        "ForecasterRnn"
    ]
    forecasters_multi_no_dict = [
        "ForecasterDirectMultiVariate",
        "ForecasterRnn",
    ]
    forecasters_multi_dict = [
        "ForecasterRecursiveMultiSeries"
    ]
    # NOTE: ForecasterStats has interval but not with bootstrapping or conformal
    # NOTE: ForecasterRnn has interval with conformal but not bootstrapping
    forecasters_boot_conformal = [
        "ForecasterRecursive",
        "ForecasterDirect",
        "ForecasterRecursiveMultiSeries",
        "ForecasterDirectMultiVariate",
        "ForecasterEquivalentDate",
    ]
    forecasters_return_predictors = [
        "ForecasterRecursive",
        "ForecasterDirect",
        "ForecasterRecursiveMultiSeries",
        "ForecasterDirectMultiVariate",
        "ForecasterRecursiveClassifier"
    ]

    if forecaster_name in forecasters_uni:
        if not isinstance(y, pd.Series):
            raise TypeError("`y` must be a pandas Series.")
        data_name = 'y'
        data_length = len(y)

    elif forecaster_name in forecasters_multi_no_dict:
        if not isinstance(series, pd.DataFrame):
            raise TypeError("`series` must be a pandas DataFrame.")
        data_name = 'series'
        data_length = len(series)

    elif forecaster_name in forecasters_multi_dict:

        # NOTE: Checks are not need as they are done in the function 
        # `check_preprocess_series` that is used before `check_backtesting_input`
        # in the backtesting function.

        data_name = 'series'
        data_length = max([len(series[serie]) for serie in series])

    if exog is not None:
        if forecaster_name in forecasters_multi_dict:
            # NOTE: Checks are not need as they are done in the function 
            # `check_preprocess_exog_multiseries` that is used before 
            # `check_backtesting_input` in the backtesting function.
            pass
        else:
            if not isinstance(exog, (pd.Series, pd.DataFrame)):
                raise TypeError(
                    f"`exog` must be a pandas Series, DataFrame or None. Got {type(exog)}."
                )

    if hasattr(forecaster, 'differentiation'):
        if forecaster.differentiation_max != cv.differentiation:
            if forecaster_name == "ForecasterRecursiveMultiSeries" and isinstance(
                forecaster.differentiation, dict
            ):
                raise ValueError(
                    f"When using a dict as `differentiation` in ForecasterRecursiveMultiSeries, "
                    f"the `differentiation` included in the cv ({cv.differentiation}) must be "
                    f"the same as the maximum `differentiation` included in the forecaster "
                    f"({forecaster.differentiation_max}). Set the same value "
                    f"for both using the `differentiation` argument."
                )
            else:
                raise ValueError(
                    f"The differentiation included in the forecaster "
                    f"({forecaster.differentiation_max}) differs from the differentiation "
                    f"included in the cv ({cv.differentiation}). Set the same value "
                    f"for both using the `differentiation` argument."
                )

    if not isinstance(metric, (str, Callable, list)):
        raise TypeError(
            f"`metric` must be a string, a callable function, or a list containing "
            f"multiple strings and/or callables. Got {type(metric)}."
        )

    if forecaster_name == "ForecasterEquivalentDate" and isinstance(
        forecaster.offset, pd.tseries.offsets.DateOffset
    ):
        # NOTE: Checks when initial_train_size is not None cannot be done here
        # because the forecaster is not fitted yet and we don't know the
        # window_size since pd.DateOffset is not a fixed window size.
        if initial_train_size is None:
            raise ValueError(
                f"`initial_train_size` must be an integer greater than "
                f"the `window_size` of the forecaster ({forecaster.window_size}) "
                f"and smaller than the length of `{data_name}` ({data_length}) or "
                f"a date within this range of the index."
            )
    elif initial_train_size is not None:
        if forecaster_name in forecasters_uni:
            index = cv._extract_index(y)
        else:
            index = cv._extract_index(series)

        initial_train_size = date_to_index_position(
                                 index        = index, 
                                 date_input   = initial_train_size, 
                                 method       = 'validation',
                                 date_literal = 'initial_train_size'
                             )
        if initial_train_size < forecaster.window_size or initial_train_size >= data_length:
            raise ValueError(
                f"If `initial_train_size` is an integer, it must be greater than "
                f"the `window_size` of the forecaster ({forecaster.window_size}) "
                f"and smaller than the length of `{data_name}` ({data_length}). If "
                f"it is a date, it must be within this range of the index."
            )
        if allow_incomplete_fold:
            # At least one observation after the gap to allow incomplete fold
            if data_length <= initial_train_size + gap:
                raise ValueError(
                    f"`{data_name}` must have more than `initial_train_size + gap` "
                    f"observations to create at least one fold.\n"
                    f"    Time series length: {data_length}\n"
                    f"    Required > {initial_train_size + gap}\n"
                    f"    initial_train_size: {initial_train_size}\n"
                    f"    gap: {gap}\n"
                )
        else:
            # At least one complete fold
            if data_length < initial_train_size + gap + steps:
                raise ValueError(
                    f"`{data_name}` must have at least `initial_train_size + gap + steps` "
                    f"observations to create a minimum of one complete fold "
                    f"(allow_incomplete_fold=False).\n"
                    f"    Time series length: {data_length}\n"
                    f"    Required >= {initial_train_size + gap + steps}\n"
                    f"    initial_train_size: {initial_train_size}\n"
                    f"    gap: {gap}\n"
                    f"    steps: {steps}\n"
                )
    else:
        if forecaster_name in ['ForecasterStats', 'ForecasterEquivalentDate']:
            raise ValueError(
                f"When using {forecaster_name}, `initial_train_size` must be an "
                f"integer smaller than the length of `{data_name}` ({data_length})."
            )
        else:
            if not forecaster.is_fitted:
                raise NotFittedError(
                    "`forecaster` must be already trained if no `initial_train_size` "
                    "is provided."
                )
            if refit:
                raise ValueError(
                    "`refit` is only allowed when `initial_train_size` is not `None`."
                )

    if forecaster_name == 'ForecasterStats' and cv.skip_folds is not None:
        raise ValueError(
            "`skip_folds` is not allowed for ForecasterStats. Set it to `None`."
        )

    if not isinstance(add_aggregated_metric, bool):
        raise TypeError("`add_aggregated_metric` must be a boolean: `True`, `False`.")
    if not isinstance(n_boot, (int, np.integer)) or n_boot < 0:
        raise TypeError(f"`n_boot` must be an integer greater than 0. Got {n_boot}.")
    if not isinstance(use_in_sample_residuals, bool):
        raise TypeError("`use_in_sample_residuals` must be a boolean: `True`, `False`.")
    if not isinstance(use_binned_residuals, bool):
        raise TypeError("`use_binned_residuals` must be a boolean: `True`, `False`.")
    if not isinstance(random_state, (int, np.integer)) or random_state < 0:
        raise TypeError(f"`random_state` must be an integer greater than 0. Got {random_state}.")
    if not isinstance(return_predictors, bool):
        raise TypeError("`return_predictors` must be a boolean: `True`, `False`.")
    if not isinstance(freeze_params, bool):
        raise TypeError("`freeze_params` must be a boolean: `True`, `False`.")
    if not isinstance(n_jobs, int) and n_jobs != 'auto':
        raise TypeError(f"`n_jobs` must be an integer or `'auto'`. Got {n_jobs}.")
    if not isinstance(show_progress, bool):
        raise TypeError("`show_progress` must be a boolean: `True`, `False`.")
    if not isinstance(suppress_warnings, bool):
        raise TypeError("`suppress_warnings` must be a boolean: `True`, `False`.")

    if interval is not None or alpha is not None:

        if forecaster_name in forecasters_boot_conformal:

            if interval_method == 'conformal':
                if not isinstance(interval, (float, list, tuple)):
                    raise TypeError(
                        f"When `interval_method` is 'conformal', `interval` must "
                        f"be a float or a list/tuple defining a symmetric interval. "
                        f"Got {type(interval)}."
                    )
            elif interval_method == 'bootstrapping':
                if (
                    not isinstance(interval, (float, list, tuple, str))
                    and (not hasattr(interval, "_pdf") or not callable(getattr(interval, "fit", None)))
                ):                
                    raise TypeError(
                        f"When `interval_method` is 'bootstrapping', `interval` "
                        f"must be a float, a list or tuple of floats, a "
                        f"scipy.stats distribution object (with methods `_pdf` and "
                        f"`fit`) or the string 'bootstrapping'. Got {type(interval)}."
                    )
                if isinstance(interval, (list, tuple)):
                    for i in interval:
                        if not isinstance(i, (int, float)):
                            raise TypeError(
                                f"`interval` must be a list or tuple of floats. "
                                f"Got {type(i)} in {interval}."
                            )
                    if len(interval) == 2:
                        check_interval(interval=interval)
                    else:
                        for q in interval:
                            if (q < 0.) or (q > 100.):
                                raise ValueError(
                                    f"When `interval` is a list or tuple, all values must be "
                                    f"between 0 and 100 inclusive. Got {q} in {interval}."
                                )
                elif isinstance(interval, float):
                    if (interval <= 0.) or (interval >= 1.):
                        raise ValueError(
                            f"When `interval` is a float, it must be between 0 and 1 "
                            f"exclusive. Got {interval}."
                        )
                elif isinstance(interval, str):
                    if interval != 'bootstrapping':
                        raise ValueError(
                            f"When `interval` is a string, it must be 'bootstrapping'."
                            f"Got {interval}."
                        )
            else:
                raise ValueError(
                    f"`interval_method` must be 'bootstrapping' or 'conformal'. "
                    f"Got {interval_method}."
                )
        elif forecaster_name == 'ForecasterRnn':
            if use_binned_residuals:
                raise ValueError(
                    "`use_binned_residuals` is not supported for ForecasterRnn. "
                    "Set `use_binned_residuals=False`."
                )
        else:
            if forecaster_name == 'ForecasterRecursiveClassifier':
                raise ValueError(
                    f"`interval` is not supported for {forecaster_name}. Class "
                    f"probabilities are returned by default during backtesting, "
                    f"set `interval=None`."
                )
            check_interval(interval=interval, alpha=alpha)

    if return_predictors and forecaster_name not in forecasters_return_predictors:
        raise ValueError(
            f"`return_predictors` is only allowed for forecasters of type "
            f"{forecasters_return_predictors}. Got {forecaster_name}."
        )

    if forecaster_name in forecasters_direct and forecaster.max_step < steps + gap:
        raise ValueError(
            f"When using a {forecaster_name}, the combination of steps "
            f"+ gap ({steps + gap}) cannot be greater than the `steps` parameter "
            f"declared when the forecaster is initialized ({forecaster.max_step})."
        )

skforecast.model_selection._utils.check_one_step_ahead_input

check_one_step_ahead_input(
    forecaster,
    cv,
    metric,
    y=None,
    series=None,
    exog=None,
    show_progress=True,
    suppress_warnings=False,
)

This is a helper function to check most inputs of hyperparameter tuning functions in modules model_selection when using a OneStepAheadFold.

Parameters:

Name	Type	Description	Default
forecaster	Forecaster	Forecaster model.	required
cv	OneStepAheadFold	OneStepAheadFold object with the information needed to split the data into folds.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model.	required
y	pandas Series	Training time series for uni-series forecasters.	None
series	pandas DataFrame, dict	Training time series for multi-series forecasters.	None
exog	pandas Series, pandas DataFrame, dict	Exogenous variables.	None
show_progress	bool	Whether to show a progress bar.	True
suppress_warnings	bool	If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information.	False

Returns:

Type	Description
None

Source code in skforecast\model_selection\_utils.py

def check_one_step_ahead_input(
    forecaster: object,
    cv: object,
    metric: str | Callable | list[str | Callable],
    y: pd.Series | None = None,
    series: pd.DataFrame | dict[str, pd.Series | pd.DataFrame] = None,
    exog: pd.Series | pd.DataFrame | dict[str, pd.Series | pd.DataFrame] | None = None,
    show_progress: bool = True,
    suppress_warnings: bool = False
) -> None:
    """
    This is a helper function to check most inputs of hyperparameter tuning
    functions in modules `model_selection` when using a `OneStepAheadFold`.

    Parameters
    ----------
    forecaster : Forecaster
        Forecaster model.
    cv : OneStepAheadFold
        OneStepAheadFold object with the information needed to split the data into folds.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.
    y : pandas Series, default None
        Training time series for uni-series forecasters.
    series : pandas DataFrame, dict, default None
        Training time series for multi-series forecasters.
    exog : pandas Series, pandas DataFrame, dict, default None
        Exogenous variables.
    show_progress : bool, default True
        Whether to show a progress bar.
    suppress_warnings: bool, default False
        If `True`, skforecast warnings will be suppressed during the hyperparameter 
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.

    Returns
    -------
    None

    """

    forecaster_name = type(forecaster).__name__
    cv_name = type(cv).__name__

    if cv_name != "OneStepAheadFold":
        raise TypeError(f"`cv` must be a 'OneStepAheadFold' object. Got '{cv_name}'.")

    initial_train_size = cv.initial_train_size

    forecasters_one_step_ahead = [
        "ForecasterRecursive",
        "ForecasterDirect",
        "ForecasterRecursiveClassifier",
        'ForecasterRecursiveMultiSeries',
        'ForecasterDirectMultiVariate'
    ]
    if forecaster_name not in forecasters_one_step_ahead:
        raise TypeError(
            f"Only forecasters of type {forecasters_one_step_ahead} are allowed "
            f"when using `cv` of type `OneStepAheadFold`. Got {forecaster_name}."
        )

    forecasters_uni = [
        "ForecasterRecursive",
        "ForecasterDirect",
        "ForecasterRecursiveClassifier"
    ]
    forecasters_multi_no_dict = [
        "ForecasterDirectMultiVariate",
    ]
    forecasters_multi_dict = [
        "ForecasterRecursiveMultiSeries"
    ]

    if forecaster_name in forecasters_uni:
        if not isinstance(y, pd.Series):
            raise TypeError(f"`y` must be a pandas Series. Got {type(y)}")
        data_name = 'y'
        data_length = len(y)

    elif forecaster_name in forecasters_multi_no_dict:
        if not isinstance(series, pd.DataFrame):
            raise TypeError(f"`series` must be a pandas DataFrame. Got {type(series)}")
        data_name = 'series'
        data_length = len(series)

    elif forecaster_name in forecasters_multi_dict:

        # NOTE: Checks are not need as they are done in the function 
        # `check_preprocess_series` that is used before `check_one_step_ahead_input`
        # in the backtesting function.

        data_name = 'series'
        data_length = max([len(series[serie]) for serie in series])

    if exog is not None:
        if forecaster_name in forecasters_multi_dict:
            # NOTE: Checks are not need as they are done in the function 
            # `check_preprocess_exog_multiseries` that is used before 
            # `check_backtesting_input` in the backtesting function.
            pass
        else:
            if not isinstance(exog, (pd.Series, pd.DataFrame)):
                raise TypeError(
                    f"`exog` must be a pandas Series, DataFrame or None. Got {type(exog)}."
                )

    if hasattr(forecaster, 'differentiation'):
        if forecaster.differentiation_max != cv.differentiation:
            if forecaster_name == "ForecasterRecursiveMultiSeries" and isinstance(
                forecaster.differentiation, dict
            ):
                raise ValueError(
                    f"When using a dict as `differentiation` in ForecasterRecursiveMultiSeries, "
                    f"the `differentiation` included in the cv ({cv.differentiation}) must be "
                    f"the same as the maximum `differentiation` included in the forecaster "
                    f"({forecaster.differentiation_max}). Set the same value "
                    f"for both using the `differentiation` argument."
                )
            else:
                raise ValueError(
                    f"The differentiation included in the forecaster "
                    f"({forecaster.differentiation_max}) differs from the differentiation "
                    f"included in the cv ({cv.differentiation}). Set the same value "
                    f"for both using the `differentiation` argument."
                )

    if not isinstance(metric, (str, Callable, list)):
        raise TypeError(
            f"`metric` must be a string, a callable function, or a list containing "
            f"multiple strings and/or callables. Got {type(metric)}."
        )

    if forecaster_name in forecasters_uni:
        index = cv._extract_index(y)
    else:
        index = cv._extract_index(series)

    initial_train_size = date_to_index_position(
                             index        = index, 
                             date_input   = initial_train_size, 
                             method       = 'validation',
                             date_literal = 'initial_train_size'
                         )
    if initial_train_size < forecaster.window_size or initial_train_size >= data_length:
        raise ValueError(
            f"If `initial_train_size` is an integer, it must be greater than "
            f"the `window_size` of the forecaster ({forecaster.window_size}) "
            f"and smaller than the length of `{data_name}` ({data_length}). If "
            f"it is a date, it must be within this range of the index."
        )

    if not isinstance(show_progress, bool):
        raise TypeError("`show_progress` must be a boolean: `True`, `False`.")
    if not isinstance(suppress_warnings, bool):
        raise TypeError("`suppress_warnings` must be a boolean: `True`, `False`.")

    if not suppress_warnings:
        warnings.warn(
            "One-step-ahead predictions are used for faster model comparison, but they "
            "may not fully represent multi-step prediction performance. It is recommended "
            "to backtest the final model for a more accurate multi-step performance "
            "estimate.", OneStepAheadValidationWarning
        )

skforecast.model_selection._utils.select_n_jobs_backtesting

select_n_jobs_backtesting(forecaster, refit)

Select the optimal number of jobs to use in the backtesting process. This selection is based on heuristics and is not guaranteed to be optimal.

The number of jobs is chosen as follows:

• If refit is an integer, then n_jobs = 1. This is because parallelization doesn't work with intermittent refit.
• If forecaster is 'ForecasterRecursive' and estimator is a linear estimator, then n_jobs = 1.
• If forecaster is 'ForecasterRecursive' and estimator is not a linear estimator then n_jobs = cpu_count() - 1.
• If forecaster is 'ForecasterDirect' or 'ForecasterDirectMultiVariate' and refit = True, then n_jobs = cpu_count() - 1.
• If forecaster is 'ForecasterDirect' or 'ForecasterDirectMultiVariate' and refit = False, then n_jobs = 1.
• If forecaster is 'ForecasterRecursiveMultiSeries', then n_jobs = cpu_count() - 1.
• If forecaster is 'ForecasterStats' or 'ForecasterEquivalentDate', then n_jobs = 1.
• If estimator is a LGBMRegressor(n_jobs=1), then n_jobs = cpu_count() - 1.
• If estimator is a LGBMRegressor with internal n_jobs != 1, then n_jobs = 1. This is because lightgbm is highly optimized for gradient boosting and parallelizes operations at a very fine-grained level, making additional parallelization unnecessary and potentially harmful due to resource contention.

Parameters:

Name	Type	Description	Default
forecaster	Forecaster	Forecaster model.	required
refit	(bool, int)	If the forecaster is refitted during the backtesting process.	required

Returns:

Name	Type	Description
n_jobs	int	The number of jobs to run in parallel.

Source code in skforecast\model_selection\_utils.py

def select_n_jobs_backtesting(
    forecaster: object,
    refit: bool | int
) -> int:
    """
    Select the optimal number of jobs to use in the backtesting process. This
    selection is based on heuristics and is not guaranteed to be optimal.

    The number of jobs is chosen as follows:

    - If `refit` is an integer, then `n_jobs = 1`. This is because parallelization doesn't 
    work with intermittent refit.
    - If forecaster is 'ForecasterRecursive' and estimator is a linear estimator, 
    then `n_jobs = 1`.
    - If forecaster is 'ForecasterRecursive' and estimator is not a linear 
    estimator then `n_jobs = cpu_count() - 1`.
    - If forecaster is 'ForecasterDirect' or 'ForecasterDirectMultiVariate'
    and `refit = True`, then `n_jobs = cpu_count() - 1`.
    - If forecaster is 'ForecasterDirect' or 'ForecasterDirectMultiVariate'
    and `refit = False`, then `n_jobs = 1`.
    - If forecaster is 'ForecasterRecursiveMultiSeries', then `n_jobs = cpu_count() - 1`.
    - If forecaster is 'ForecasterStats' or 'ForecasterEquivalentDate', 
    then `n_jobs = 1`.
    - If estimator is a `LGBMRegressor(n_jobs=1)`, then `n_jobs = cpu_count() - 1`.
    - If estimator is a `LGBMRegressor` with internal n_jobs != 1, then `n_jobs = 1`.
    This is because `lightgbm` is highly optimized for gradient boosting and
    parallelizes operations at a very fine-grained level, making additional
    parallelization unnecessary and potentially harmful due to resource contention.

    Parameters
    ----------
    forecaster : Forecaster
        Forecaster model.
    refit : bool, int
        If the forecaster is refitted during the backtesting process.

    Returns
    -------
    n_jobs : int
        The number of jobs to run in parallel.

    """

    forecaster_name = type(forecaster).__name__

    if forecaster_name == 'ForecasterStats':
        n_jobs = 1
        return n_jobs

    if isinstance(forecaster.estimator, Pipeline):
        estimator = forecaster.estimator[-1]
    else:
        estimator = forecaster.estimator

    refit = False if refit == 0 else refit
    if not isinstance(refit, bool) and refit != 1:
        n_jobs = 1
    else:
        if forecaster_name in {'ForecasterRecursive', 'ForecasterRecursiveClassifier'}:
            if isinstance(estimator, (LinearModel, LinearClassifierMixin)):
                n_jobs = 1
            elif type(estimator).__name__ in {'LGBMRegressor', 'LGBMClassifier'}:
                n_jobs = cpu_count() - 1 if estimator.n_jobs == 1 else 1
            else:
                n_jobs = cpu_count() - 1
        elif forecaster_name in {'ForecasterDirect', 'ForecasterDirectMultiVariate'}:
            # Parallelization is applied during the fitting process.
            n_jobs = 1
        elif forecaster_name in {'ForecasterRecursiveMultiSeries'}:
            if type(estimator).__name__ == 'LGBMRegressor':
                n_jobs = cpu_count() - 1 if estimator.n_jobs == 1 else 1
            else:
                n_jobs = cpu_count() - 1
        elif forecaster_name in {'ForecasterEquivalentDate'}:
            n_jobs = 1
        else:
            n_jobs = 1

    return n_jobs