model_selection

backtesting_forecaster(forecaster, y, steps, metric, initial_train_size=None, fixed_train_size=True, gap=0, skip_folds=None, allow_incomplete_fold=True, exog=None, refit=False, interval=None, n_boot=250, random_state=123, in_sample_residuals=True, binned_residuals=False, n_jobs='auto', verbose=False, show_progress=True)

Backtesting of forecaster model.

• If refit is False, the model will be trained only once using the initial_train_size first observations.
• If refit is True, the model is trained on each iteration, increasing the training set.
• If refit is an integer, the model will be trained every that number of iterations.
• If forecaster is already trained and initial_train_size is None, no initial train will be done and all data will be used to evaluate the model. However, the first len(forecaster.last_window) observations are needed to create the initial predictors, so no predictions are calculated for them.

A copy of the original forecaster is created so that it is not modified during the process.

Parameters:

Name	Type	Description	Default
forecaster	(ForecasterAutoreg, ForecasterAutoregCustom, ForecasterAutoregDirect)	Forecaster model.	required
y	pandas Series	Training time series.	required
steps	int	Number of steps to predict.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'} If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. If list: List containing multiple strings and/or Callables.	required
initial_train_size	int	Number of samples in the initial train split. If None and forecaster is already trained, no initial train is done and all data is used to evaluate the model. However, the first len(forecaster.last_window) observations are needed to create the initial predictors, so no predictions are calculated for them. This useful to backtest the model on the same data used to train it. None is only allowed when refit is False and forecaster is already trained.	`None`
fixed_train_size	bool	If True, train size doesn't increase but moves by steps in each iteration.	`True`
gap	int	Number of samples to be excluded after the end of each training set and before the test set.	`0`
skip_folds	(int, list)	If skip_folds is an integer, every 'skip_folds'-th is returned. If skip_folds is a list, the folds in the list are skipped. For example, if skip_folds = 3, and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If skip_folds is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].	`None`
allow_incomplete_fold	bool	Last fold is allowed to have a smaller number of samples than the test_size. If False, the last fold is excluded.	`True`
exog	pandas Series, pandas DataFrame	Exogenous variable/s included as predictor/s. Must have the same number of observations as y and should be aligned so that y[i] is regressed on exog[i].	`None`
refit	(bool, int)	Whether to re-fit the forecaster in each iteration. If refit is an integer, the Forecaster will be trained every that number of iterations.	`False`
interval	list	Confidence of the prediction interval estimated. Sequence of percentiles to compute, which must be between 0 and 100 inclusive. For example, interval of 95% should be as interval = [2.5, 97.5]. If None, no intervals are estimated.	`None`
n_boot	int	Number of bootstrapping iterations used to estimate prediction intervals.	`500`
random_state	int	Sets a seed to the random generator, so that boot intervals are always deterministic.	`123`
in_sample_residuals	bool	If True, residuals from the training data are used as proxy of prediction error to create prediction intervals. If False, out_sample_residuals are used if they are already stored inside the forecaster.	`True`
binned_residuals	bool	If `True`, residuals used in each bootstrapping iteration are selected conditioning on the predicted values. If `False`, residuals are selected randomly without conditioning on the predicted values.	`False`
n_jobs	(int, auto)	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting. New in version 0.9.0	`'auto'`
verbose	bool	Print number of folds and index of training and validation sets used for backtesting.	`False`
show_progress	bool	Whether to show a progress bar.	`True`

Returns:

Name	Type	Description
metric_values	pandas DataFrame	Value(s) of the metric(s).
backtest_predictions	pandas DataFrame	Value of predictions and their estimated interval if interval is not None. column pred: predictions. column lower_bound: lower bound of the interval. column upper_bound: upper bound of the interval.

Source code in skforecast\model_selection\model_selection.py

def backtesting_forecaster(
    forecaster: object,
    y: pd.Series,
    steps: int,
    metric: Union[str, Callable, list],
    initial_train_size: Optional[int] = None,
    fixed_train_size: bool = True,
    gap: int = 0,
    skip_folds: Optional[Union[int, list]] = None,
    allow_incomplete_fold: bool = True,
    exog: Optional[Union[pd.Series, pd.DataFrame]] = None,
    refit: Union[bool, int] = False,
    interval: Optional[list] = None,
    n_boot: int = 250,
    random_state: int = 123,
    in_sample_residuals: bool = True,
    binned_residuals: bool = False,
    n_jobs: Union[int, str] = 'auto',
    verbose: bool = False,
    show_progress: bool = True
) -> Tuple[pd.DataFrame, pd.DataFrame]:
    """
    Backtesting of forecaster model.

    - If `refit` is `False`, the model will be trained only once using the 
    `initial_train_size` first observations. 
    - If `refit` is `True`, the model is trained on each iteration, increasing
    the training set. 
    - If `refit` is an `integer`, the model will be trained every that number 
    of iterations.
    - If `forecaster` is already trained and `initial_train_size` is `None`,
    no initial train will be done and all data will be used to evaluate the model.
    However, the first `len(forecaster.last_window)` observations are needed
    to create the initial predictors, so no predictions are calculated for them.

    A copy of the original forecaster is created so that it is not modified during 
    the process.

    Parameters
    ----------
    forecaster : ForecasterAutoreg, ForecasterAutoregCustom, ForecasterAutoregDirect
        Forecaster model.
    y : pandas Series
        Training time series.
    steps : int
        Number of steps to predict.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    initial_train_size : int, default `None`
        Number of samples in the initial train split. If `None` and `forecaster` is 
        already trained, no initial train is done and all data is used to evaluate the 
        model. However, the first `len(forecaster.last_window)` observations are needed 
        to create the initial predictors, so no predictions are calculated for them. 
        This useful to backtest the model on the same data used to train it.
        `None` is only allowed when `refit` is `False` and `forecaster` is already
        trained.
    fixed_train_size : bool, default `True`
        If True, train size doesn't increase but moves by `steps` in each iteration.
    gap : int, default `0`
        Number of samples to be excluded after the end of each training set and 
        before the test set.
    skip_folds : int, list, default `None`
        If `skip_folds` is an integer, every 'skip_folds'-th is returned. If `skip_folds`
        is a list, the folds in the list are skipped. For example, if `skip_folds = 3`,
        and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If `skip_folds`
        is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].
    allow_incomplete_fold : bool, default `True`
        Last fold is allowed to have a smaller number of samples than the 
        `test_size`. If `False`, the last fold is excluded.
    exog : pandas Series, pandas DataFrame, default `None`
        Exogenous variable/s included as predictor/s. Must have the same
        number of observations as `y` and should be aligned so that y[i] is
        regressed on exog[i].
    refit : bool, int, default `False`
        Whether to re-fit the forecaster in each iteration. If `refit` is an integer, 
        the Forecaster will be trained every that number of iterations.
    interval : list, default `None`
        Confidence of the prediction interval estimated. Sequence of percentiles
        to compute, which must be between 0 and 100 inclusive. For example, 
        interval of 95% should be as `interval = [2.5, 97.5]`. If `None`, no
        intervals are estimated.
    n_boot : int, default `500`
        Number of bootstrapping iterations used to estimate prediction
        intervals.
    random_state : int, default `123`
        Sets a seed to the random generator, so that boot intervals are always 
        deterministic.
    in_sample_residuals : bool, default `True`
        If `True`, residuals from the training data are used as proxy of prediction 
        error to create prediction intervals. If `False`, out_sample_residuals 
        are used if they are already stored inside the forecaster.
    binned_residuals : bool, default `False`
            If `True`, residuals used in each bootstrapping iteration are selected
            conditioning on the predicted values. If `False`, residuals are selected
            randomly without conditioning on the predicted values.
    n_jobs : int, 'auto', default `'auto'`
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting.
        **New in version 0.9.0**
    verbose : bool, default `False`
        Print number of folds and index of training and validation sets used 
        for backtesting.
    show_progress : bool, default `True`
        Whether to show a progress bar.

    Returns
    -------
    metric_values : pandas DataFrame
        Value(s) of the metric(s).
    backtest_predictions : pandas DataFrame
        Value of predictions and their estimated interval if `interval` is not `None`.

        - column pred: predictions.
        - column lower_bound: lower bound of the interval.
        - column upper_bound: upper bound of the interval.

    """

    forecaters_allowed = [
        'ForecasterAutoreg', 
        'ForecasterAutoregCustom', 
        'ForecasterAutoregDirect',
        'ForecasterEquivalentDate'
    ]

    if type(forecaster).__name__ not in forecaters_allowed:
        raise TypeError(
            (f"`forecaster` must be of type {forecaters_allowed}, for all other types of "
             f" forecasters use the functions available in the other `model_selection` "
             f"modules.")
        )

    check_backtesting_input(
        forecaster            = forecaster,
        steps                 = steps,
        metric                = metric,
        y                     = y,
        initial_train_size    = initial_train_size,
        fixed_train_size      = fixed_train_size,
        gap                   = gap,
        skip_folds            = skip_folds,
        allow_incomplete_fold = allow_incomplete_fold,
        refit                 = refit,
        interval              = interval,
        n_boot                = n_boot,
        random_state          = random_state,
        in_sample_residuals   = in_sample_residuals,
        n_jobs                = n_jobs,
        verbose               = verbose,
        show_progress         = show_progress
    )

    if type(forecaster).__name__ == 'ForecasterAutoregDirect' and \
       forecaster.steps < steps + gap:
        raise ValueError(
            (f"When using a ForecasterAutoregDirect, the combination of steps "
             f"+ gap ({steps + gap}) cannot be greater than the `steps` parameter "
             f"declared when the forecaster is initialized ({forecaster.steps}).")
        )

    metric_values, backtest_predictions = _backtesting_forecaster(
        forecaster            = forecaster,
        y                     = y,
        steps                 = steps,
        metric                = metric,
        initial_train_size    = initial_train_size,
        fixed_train_size      = fixed_train_size,
        gap                   = gap,
        skip_folds            = skip_folds,
        allow_incomplete_fold = allow_incomplete_fold,
        exog                  = exog,
        refit                 = refit,
        interval              = interval,
        n_boot                = n_boot,
        random_state          = random_state,
        in_sample_residuals   = in_sample_residuals,
        binned_residuals      = binned_residuals,
        n_jobs                = n_jobs,
        verbose               = verbose,
        show_progress         = show_progress
    )

    return metric_values, backtest_predictions

grid_search_forecaster(forecaster, y, param_grid, steps, metric, initial_train_size, fixed_train_size=True, gap=0, skip_folds=None, allow_incomplete_fold=True, exog=None, lags_grid=None, refit=False, return_best=True, n_jobs='auto', verbose=True, show_progress=True, output_file=None)

Exhaustive search over specified parameter values for a Forecaster object. Validation is done using time series backtesting.

Parameters:

Name	Type	Description	Default
forecaster	(ForecasterAutoreg, ForecasterAutoregCustom, ForecasterAutoregDirect)	Forecaster model.	required
y	pandas Series	Training time series.	required
param_grid	dict	Dictionary with parameters names (str) as keys and lists of parameter settings to try as values.	required
steps	int	Number of steps to predict.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'} If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. If list: List containing multiple strings and/or Callables.	required
initial_train_size	int	Number of samples in the initial train split.	required
fixed_train_size	bool	If True, train size doesn't increase but moves by steps in each iteration.	`True`
gap	int	Number of samples to be excluded after the end of each training set and before the test set.	`0`
skip_folds	(int, list)	If skip_folds is an integer, every 'skip_folds'-th is returned. If skip_folds is a list, the folds in the list are skipped. For example, if skip_folds = 3, and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If skip_folds is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].	`None`
allow_incomplete_fold	bool	Last fold is allowed to have a smaller number of samples than the test_size. If False, the last fold is excluded.	`True`
exog	pandas Series, pandas DataFrame	Exogenous variable/s included as predictor/s. Must have the same number of observations as y and should be aligned so that y[i] is regressed on exog[i].	`None`
lags_grid	(list, dict)	Lists of lags to try, containing int, lists, numpy ndarray, or range objects. If dict, the keys are used as labels in the results DataFrame, and the values are used as the lists of lags to try. Ignored if the forecaster is an instance of ForecasterAutoregCustom or ForecasterAutoregMultiSeriesCustom.	`None`
refit	(bool, int)	Whether to re-fit the forecaster in each iteration. If refit is an integer, the Forecaster will be trained every that number of iterations.	`False`
return_best	bool	Refit the forecaster using the best found parameters on the whole data.	`True`
n_jobs	(int, auto)	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting. New in version 0.9.0	`'auto'`
verbose	bool	Print number of folds used for cv or backtesting.	`True`
show_progress	bool	Whether to show a progress bar.	`True`
output_file	str	Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file. New in version 0.12.0	`None`

Returns:

Name	Type	Description
results	pandas DataFrame	Results for each combination of parameters. column lags: lags configuration for each iteration. column lags_label: descriptive label or alias for the lags. column params: parameters configuration for each iteration. column metric: metric value estimated for each iteration. additional n columns with param = value.

Source code in skforecast\model_selection\model_selection.py

def grid_search_forecaster(
    forecaster: object,
    y: pd.Series,
    param_grid: dict,
    steps: int,
    metric: Union[str, Callable, list],
    initial_train_size: int,
    fixed_train_size: bool = True,
    gap: int = 0,
    skip_folds: Optional[Union[int, list]] = None,
    allow_incomplete_fold: bool = True,
    exog: Optional[Union[pd.Series, pd.DataFrame]] = None,
    lags_grid: Optional[Union[list, dict]] = None,
    refit: Union[bool, int] = False,
    return_best: bool = True,
    n_jobs: Union[int, str] = 'auto',
    verbose: bool = True,
    show_progress: bool = True,
    output_file: Optional[str] = None
) -> pd.DataFrame:
    """
    Exhaustive search over specified parameter values for a Forecaster object.
    Validation is done using time series backtesting.

    Parameters
    ----------
    forecaster : ForecasterAutoreg, ForecasterAutoregCustom, ForecasterAutoregDirect
        Forecaster model.
    y : pandas Series
        Training time series. 
    param_grid : dict
        Dictionary with parameters names (`str`) as keys and lists of parameter
        settings to try as values.
    steps : int
        Number of steps to predict.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    initial_train_size : int 
        Number of samples in the initial train split.
    fixed_train_size : bool, default `True`
        If True, train size doesn't increase but moves by `steps` in each iteration.
    gap : int, default `0`
        Number of samples to be excluded after the end of each training set and 
        before the test set.
    skip_folds : int, list, default `None`
        If `skip_folds` is an integer, every 'skip_folds'-th is returned. If `skip_folds`
        is a list, the folds in the list are skipped. For example, if `skip_folds = 3`,
        and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If `skip_folds`
        is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].
    allow_incomplete_fold : bool, default `True`
        Last fold is allowed to have a smaller number of samples than the 
        `test_size`. If `False`, the last fold is excluded.
    exog : pandas Series, pandas DataFrame, default `None`
        Exogenous variable/s included as predictor/s. Must have the same
        number of observations as `y` and should be aligned so that y[i] is
        regressed on exog[i].
    lags_grid : list, dict, default `None`
        Lists of lags to try, containing int, lists, numpy ndarray, or range 
        objects. If `dict`, the keys are used as labels in the `results` 
        DataFrame, and the values are used as the lists of lags to try. Ignored 
        if the forecaster is an instance of `ForecasterAutoregCustom` or 
        `ForecasterAutoregMultiSeriesCustom`.
    refit : bool, int, default `False`
        Whether to re-fit the forecaster in each iteration. If `refit` is an integer, 
        the Forecaster will be trained every that number of iterations.
    return_best : bool, default `True`
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default `'auto'`
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting.
        **New in version 0.9.0**
    verbose : bool, default `True`
        Print number of folds used for cv or backtesting.
    show_progress : bool, default `True`
        Whether to show a progress bar.
    output_file : str, default `None`
        Specifies the filename or full path where the results should be saved. 
        The results will be saved in a tab-separated values (TSV) format. If 
        `None`, the results will not be saved to a file.
        **New in version 0.12.0**

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column lags: lags configuration for each iteration.
        - column lags_label: descriptive label or alias for the lags.
        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration.
        - additional n columns with param = value.

    """

    param_grid = list(ParameterGrid(param_grid))

    results = _evaluate_grid_hyperparameters(
        forecaster            = forecaster,
        y                     = y,
        param_grid            = param_grid,
        steps                 = steps,
        metric                = metric,
        initial_train_size    = initial_train_size,
        fixed_train_size      = fixed_train_size,
        gap                   = gap,
        skip_folds            = skip_folds,
        allow_incomplete_fold = allow_incomplete_fold,
        exog                  = exog,
        lags_grid             = lags_grid,
        refit                 = refit,
        return_best           = return_best,
        n_jobs                = n_jobs,
        verbose               = verbose,
        show_progress         = show_progress,
        output_file           = output_file
    )

    return results

random_search_forecaster(forecaster, y, param_distributions, steps, metric, initial_train_size, fixed_train_size=True, gap=0, skip_folds=None, allow_incomplete_fold=True, exog=None, lags_grid=None, refit=False, n_iter=10, random_state=123, return_best=True, n_jobs='auto', verbose=True, show_progress=True, output_file=None)

Random search over specified parameter values or distributions for a Forecaster object. Validation is done using time series backtesting.

Parameters:

Name	Type	Description	Default
forecaster	(ForecasterAutoreg, ForecasterAutoregCustom, ForecasterAutoregDirect)	Forecaster model.	required
y	pandas Series	Training time series.	required
param_distributions	dict	Dictionary with parameters names (str) as keys and distributions or lists of parameters to try.	required
steps	int	Number of steps to predict.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'} If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. If list: List containing multiple strings and/or Callables.	required
initial_train_size	int	Number of samples in the initial train split.	required
fixed_train_size	bool	If True, train size doesn't increase but moves by steps in each iteration.	`True`
gap	int	Number of samples to be excluded after the end of each training set and before the test set.	`0`
skip_folds	(int, list)	If skip_folds is an integer, every 'skip_folds'-th is returned. If skip_folds is a list, the folds in the list are skipped. For example, if skip_folds = 3, and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If skip_folds is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].	`None`
allow_incomplete_fold	bool	Last fold is allowed to have a smaller number of samples than the test_size. If False, the last fold is excluded.	`True`
exog	pandas Series, pandas DataFrame	Exogenous variable/s included as predictor/s. Must have the same number of observations as y and should be aligned so that y[i] is regressed on exog[i].	`None`
lags_grid	(list, dict)	Lists of lags to try, containing int, lists, numpy ndarray, or range objects. If dict, the keys are used as labels in the results DataFrame, and the values are used as the lists of lags to try. Ignored if the forecaster is an instance of ForecasterAutoregCustom or ForecasterAutoregMultiSeriesCustom.	`None`
refit	(bool, int)	Whether to re-fit the forecaster in each iteration. If refit is an integer, the Forecaster will be trained every that number of iterations.	`False`
n_iter	int	Number of parameter settings that are sampled per lags configuration. n_iter trades off runtime vs quality of the solution.	`10`
random_state	int	Sets a seed to the random sampling for reproducible output.	`123`
return_best	bool	Refit the forecaster using the best found parameters on the whole data.	`True`
n_jobs	(int, auto)	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting. New in version 0.9.0	`'auto'`
verbose	bool	Print number of folds used for cv or backtesting.	`True`
show_progress	bool	Whether to show a progress bar.	`True`
output_file	str	Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file. New in version 0.12.0	`None`

Returns:

Name	Type	Description
results	pandas DataFrame	Results for each combination of parameters. column lags: lags configuration for each iteration. column lags_label: descriptive label or alias for the lags. column params: parameters configuration for each iteration. column metric: metric value estimated for each iteration. additional n columns with param = value.

Source code in skforecast\model_selection\model_selection.py

def random_search_forecaster(
    forecaster: object,
    y: pd.Series,
    param_distributions: dict,
    steps: int,
    metric: Union[str, Callable, list],
    initial_train_size: int,
    fixed_train_size: bool = True,
    gap: int = 0,
    skip_folds: Optional[Union[int, list]] = None,
    allow_incomplete_fold: bool = True,
    exog: Optional[Union[pd.Series, pd.DataFrame]] = None,
    lags_grid: Optional[Union[list, dict]] = None,
    refit: Union[bool, int] = False,
    n_iter: int = 10,
    random_state: int = 123,
    return_best: bool = True,
    n_jobs: Union[int, str] = 'auto',
    verbose: bool = True,
    show_progress: bool = True,
    output_file: Optional[str] = None
) -> pd.DataFrame:
    """
    Random search over specified parameter values or distributions for a Forecaster 
    object. Validation is done using time series backtesting.

    Parameters
    ----------
    forecaster : ForecasterAutoreg, ForecasterAutoregCustom, ForecasterAutoregDirect
        Forecaster model.
    y : pandas Series
        Training time series. 
    param_distributions : dict
        Dictionary with parameters names (`str`) as keys and 
        distributions or lists of parameters to try.
    steps : int
        Number of steps to predict.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    initial_train_size : int 
        Number of samples in the initial train split.
    fixed_train_size : bool, default `True`
        If True, train size doesn't increase but moves by `steps` in each iteration.
    gap : int, default `0`
        Number of samples to be excluded after the end of each training set and 
        before the test set.
    skip_folds : int, list, default `None`
        If `skip_folds` is an integer, every 'skip_folds'-th is returned. If `skip_folds`
        is a list, the folds in the list are skipped. For example, if `skip_folds = 3`,
        and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If `skip_folds`
        is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].
    allow_incomplete_fold : bool, default `True`
        Last fold is allowed to have a smaller number of samples than the 
        `test_size`. If `False`, the last fold is excluded.
    exog : pandas Series, pandas DataFrame, default `None`
        Exogenous variable/s included as predictor/s. Must have the same
        number of observations as `y` and should be aligned so that y[i] is
        regressed on exog[i]. 
    lags_grid : list, dict, default `None`
        Lists of lags to try, containing int, lists, numpy ndarray, or range 
        objects. If `dict`, the keys are used as labels in the `results` 
        DataFrame, and the values are used as the lists of lags to try. Ignored 
        if the forecaster is an instance of `ForecasterAutoregCustom` or 
        `ForecasterAutoregMultiSeriesCustom`.
    refit : bool, int, default `False`
        Whether to re-fit the forecaster in each iteration. If `refit` is an integer, 
        the Forecaster will be trained every that number of iterations.
    n_iter : int, default `10`
        Number of parameter settings that are sampled per lags configuration. 
        n_iter trades off runtime vs quality of the solution.
    random_state : int, default `123`
        Sets a seed to the random sampling for reproducible output.
    return_best : bool, default `True`
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default `'auto'`
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting.
        **New in version 0.9.0**
    verbose : bool, default `True`
        Print number of folds used for cv or backtesting.
    show_progress : bool, default `True`
        Whether to show a progress bar.
    output_file : str, default `None`
        Specifies the filename or full path where the results should be saved. 
        The results will be saved in a tab-separated values (TSV) format. If 
        `None`, the results will not be saved to a file.
        **New in version 0.12.0**

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column lags: lags configuration for each iteration.
        - column lags_label: descriptive label or alias for the lags.
        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration.
        - additional n columns with param = value.

    """

    param_grid = list(ParameterSampler(param_distributions, n_iter=n_iter, random_state=random_state))

    results = _evaluate_grid_hyperparameters(
        forecaster            = forecaster,
        y                     = y,
        param_grid            = param_grid,
        steps                 = steps,
        metric                = metric,
        initial_train_size    = initial_train_size,
        fixed_train_size      = fixed_train_size,
        gap                   = gap,
        skip_folds            = skip_folds,
        allow_incomplete_fold = allow_incomplete_fold,
        exog                  = exog,
        lags_grid             = lags_grid,
        refit                 = refit,
        return_best           = return_best,
        n_jobs                = n_jobs,
        verbose               = verbose,
        show_progress         = show_progress,
        output_file           = output_file
    )

    return results

bayesian_search_forecaster(forecaster, y, search_space, steps, metric, initial_train_size, fixed_train_size=True, gap=0, skip_folds=None, allow_incomplete_fold=True, exog=None, refit=False, n_trials=10, random_state=123, return_best=True, n_jobs='auto', verbose=True, show_progress=True, output_file=None, engine='optuna', kwargs_create_study={}, kwargs_study_optimize={})

Bayesian optimization for a Forecaster object using time series backtesting and optuna library.

Parameters:

Name	Type	Description	Default
forecaster	(ForecasterAutoreg, ForecasterAutoregCustom, ForecasterAutoregDirect)	Forecaster model.	required
y	pandas Series	Training time series.	required
search_space	Callable(optuna)	Function with argument trial which returns a dictionary with parameters names (str) as keys and Trial object from optuna (trial.suggest_float, trial.suggest_int, trial.suggest_categorical) as values.	required
steps	int	Number of steps to predict.	required
metric	(str, Callable, list)	Metric used to quantify the goodness of fit of the model. If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'} If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float. If list: List containing multiple strings and/or Callables.	required
initial_train_size	int	Number of samples in the initial train split.	required
fixed_train_size	bool	If True, train size doesn't increase but moves by steps in each iteration.	`True`
gap	int	Number of samples to be excluded after the end of each training set and before the test set.	`0`
skip_folds	(int, list)	If skip_folds is an integer, every 'skip_folds'-th is returned. If skip_folds is a list, the folds in the list are skipped. For example, if skip_folds = 3, and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If skip_folds is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].	`None`
allow_incomplete_fold	bool	Last fold is allowed to have a smaller number of samples than the test_size. If False, the last fold is excluded.	`True`
exog	pandas Series, pandas DataFrame	Exogenous variable/s included as predictor/s. Must have the same number of observations as y and should be aligned so that y[i] is regressed on exog[i].	`None`
refit	(bool, int)	Whether to re-fit the forecaster in each iteration. If refit is an integer, the Forecaster will be trained every that number of iterations.	`False`
n_trials	int	Number of parameter settings that are sampled in each lag configuration.	`10`
random_state	int	Sets a seed to the sampling for reproducible output. When a new sampler is passed in kwargs_create_study, the seed must be set within the sampler. For example {'sampler': TPESampler(seed=145)}.	`123`
return_best	bool	Refit the forecaster using the best found parameters on the whole data.	`True`
n_jobs	(int, auto)	The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting. New in version 0.9.0	`'auto'`
verbose	bool	Print number of folds used for cv or backtesting.	`True`
show_progress	bool	Whether to show a progress bar.	`True`
output_file	str	Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file. New in version 0.12.0	`None`
engine	str	Bayesian optimization runs through the optuna library.	`'optuna'`
kwargs_create_study	dict	Keyword arguments (key, value mappings) to pass to optuna.create_study(). If default, the direction is set to 'minimize' and a TPESampler(seed=123) sampler is used during optimization.	`{}`
kwargs_study_optimize	dict	Only applies to engine='optuna'. Other keyword arguments (key, value mappings) to pass to study.optimize().	`{}`

Returns:

Name	Type	Description
results	pandas DataFrame	Results for each combination of parameters. column lags: lags configuration for each iteration. column params: parameters configuration for each iteration. column metric: metric value estimated for each iteration. additional n columns with param = value.
best_trial	optuna object	The best optimization result returned as a FrozenTrial optuna object.

Source code in skforecast\model_selection\model_selection.py

def bayesian_search_forecaster(
    forecaster: object,
    y: pd.Series,
    search_space: Callable,
    steps: int,
    metric: Union[str, Callable, list],
    initial_train_size: int,
    fixed_train_size: bool = True,
    gap: int = 0,
    skip_folds: Optional[Union[int, list]] = None,
    allow_incomplete_fold: bool = True,
    exog: Optional[Union[pd.Series, pd.DataFrame]] = None,
    refit: Union[bool, int] = False,
    n_trials: int = 10,
    random_state: int = 123,
    return_best: bool = True,
    n_jobs: Union[int, str] = 'auto',
    verbose: bool = True,
    show_progress: bool = True,
    output_file: Optional[str] = None,
    engine: str = 'optuna',
    kwargs_create_study: dict = {},
    kwargs_study_optimize: dict = {}
) -> Tuple[pd.DataFrame, object]:
    """
    Bayesian optimization for a Forecaster object using time series backtesting and 
    optuna library.

    Parameters
    ----------
    forecaster : ForecasterAutoreg, ForecasterAutoregCustom, ForecasterAutoregDirect
        Forecaster model.
    y : pandas Series
        Training time series. 
    search_space : Callable (optuna)
        Function with argument `trial` which returns a dictionary with parameters names 
        (`str`) as keys and Trial object from optuna (trial.suggest_float, 
        trial.suggest_int, trial.suggest_categorical) as values.
    steps : int
        Number of steps to predict.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    initial_train_size : int 
        Number of samples in the initial train split.
    fixed_train_size : bool, default `True`
        If True, train size doesn't increase but moves by `steps` in each iteration.
    gap : int, default `0`
        Number of samples to be excluded after the end of each training set and 
        before the test set.
    skip_folds : int, list, default `None`
        If `skip_folds` is an integer, every 'skip_folds'-th is returned. If `skip_folds`
        is a list, the folds in the list are skipped. For example, if `skip_folds = 3`,
        and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If `skip_folds`
        is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].
    allow_incomplete_fold : bool, default `True`
        Last fold is allowed to have a smaller number of samples than the 
        `test_size`. If `False`, the last fold is excluded.
    exog : pandas Series, pandas DataFrame, default `None`
        Exogenous variable/s included as predictor/s. Must have the same
        number of observations as `y` and should be aligned so that y[i] is
        regressed on exog[i].
    refit : bool, int, default `False`
        Whether to re-fit the forecaster in each iteration. If `refit` is an integer, 
        the Forecaster will be trained every that number of iterations.
    n_trials : int, default `10`
        Number of parameter settings that are sampled in each lag configuration.
    random_state : int, default `123`
        Sets a seed to the sampling for reproducible output. When a new sampler 
        is passed in `kwargs_create_study`, the seed must be set within the 
        sampler. For example `{'sampler': TPESampler(seed=145)}`.
    return_best : bool, default `True`
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default `'auto'`
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting.
        **New in version 0.9.0**
    verbose : bool, default `True`
        Print number of folds used for cv or backtesting.
    show_progress : bool, default `True`
        Whether to show a progress bar.
    output_file : str, default `None`
        Specifies the filename or full path where the results should be saved. 
        The results will be saved in a tab-separated values (TSV) format. If 
        `None`, the results will not be saved to a file.
        **New in version 0.12.0**
    engine : str, default `'optuna'`
        Bayesian optimization runs through the optuna library.
    kwargs_create_study : dict, default `{}`
        Keyword arguments (key, value mappings) to pass to optuna.create_study().
        If default, the direction is set to 'minimize' and a TPESampler(seed=123) 
        sampler is used during optimization.
    kwargs_study_optimize : dict, default `{}`
        Only applies to engine='optuna'. Other keyword arguments (key, value mappings) 
        to pass to study.optimize().

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column lags: lags configuration for each iteration.
        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration.
        - additional n columns with param = value.
    best_trial : optuna object
        The best optimization result returned as a FrozenTrial optuna object.

    """

    if return_best and exog is not None and (len(exog) != len(y)):
        raise ValueError(
            (f"`exog` must have same number of samples as `y`. "
             f"length `exog`: ({len(exog)}), length `y`: ({len(y)})")
        )

    if engine not in ['optuna']:
        raise ValueError(
            f"`engine` only allows 'optuna', got {engine}."
        )

    results, best_trial = _bayesian_search_optuna(
                              forecaster            = forecaster,
                              y                     = y,
                              exog                  = exog,
                              search_space          = search_space,
                              steps                 = steps,
                              metric                = metric,
                              refit                 = refit,
                              initial_train_size    = initial_train_size,
                              fixed_train_size      = fixed_train_size,
                              gap                   = gap,
                              skip_folds            = skip_folds,
                              allow_incomplete_fold = allow_incomplete_fold,
                              n_trials              = n_trials,
                              random_state          = random_state,
                              return_best           = return_best,
                              n_jobs                = n_jobs,
                              verbose               = verbose,
                              show_progress         = show_progress,
                              output_file           = output_file,
                              kwargs_create_study   = kwargs_create_study,
                              kwargs_study_optimize = kwargs_study_optimize
                          )

    return results, best_trial

select_features(forecaster, selector, y, exog=None, select_only=None, force_inclusion=None, subsample=0.5, random_state=123, verbose=True)

Feature selection using any of the sklearn.feature_selection module selectors (such as RFECV, SelectFromModel, etc.). Two groups of features are evaluated: autoregressive features and exogenous features. By default, the selection process is performed on both sets of features at the same time, so that the most relevant autoregressive and exogenous features are selected. However, using the select_only argument, the selection process can focus only on the autoregressive or exogenous features without taking into account the other features. Therefore, all other features will remain in the model. It is also possible to force the inclusion of certain features in the final list of selected features using the force_inclusion parameter.

Parameters:

Name	Type	Description	Default
forecaster	(ForecasterAutoreg, ForecasterAutoregCustom)	Forecaster model.	required
selector	object	A feature selector from sklearn.feature_selection.	required
y	pandas Series, pandas DataFrame	Target time series to which the feature selection will be applied.	required
exog	pandas Series, pandas DataFrame	Exogenous variable/s included as predictor/s. Must have the same number of observations as y and should be aligned so that y[i] is regressed on exog[i].	`None`
select_only	str	Decide what type of features to include in the selection process. If 'autoreg', only autoregressive features (lags or custom predictors) are evaluated by the selector. All exogenous features are included in the output (selected_exog). If 'exog', only exogenous features are evaluated without the presence of autoregressive features. All autoregressive features are included in the output (selected_autoreg). If None, all features are evaluated by the selector.	`None`
force_inclusion	(list, str)	Features to force include in the final list of selected features. If list, list of feature names to force include. If str, regular expression to identify features to force include. For example, if force_inclusion="^sun_", all features that begin with "sun_" will be included in the final list of selected features.	`None`
subsample	(int, float)	Proportion of records to use for feature selection.	`0.5`
random_state	int	Sets a seed for the random subsample so that the subsampling process is always deterministic.	`123`
verbose	bool	Print information about feature selection process.	`True`

Returns:

Name	Type	Description
selected_autoreg	list	List of selected autoregressive features.
selected_exog	list	List of selected exogenous features.

Source code in skforecast\model_selection\model_selection.py

def select_features(
    forecaster: object,
    selector: object,
    y: Union[pd.Series, pd.DataFrame],
    exog: Optional[Union[pd.Series, pd.DataFrame]] = None,
    select_only: Optional[str] = None,
    force_inclusion: Optional[Union[list, str]] = None,
    subsample: Union[int, float] = 0.5,
    random_state: int = 123,
    verbose: bool = True
) -> Union[list, list]:
    """
    Feature selection using any of the sklearn.feature_selection module selectors 
    (such as `RFECV`, `SelectFromModel`, etc.). Two groups of features are
    evaluated: autoregressive features and exogenous features. By default, the 
    selection process is performed on both sets of features at the same time, 
    so that the most relevant autoregressive and exogenous features are selected. 
    However, using the `select_only` argument, the selection process can focus 
    only on the autoregressive or exogenous features without taking into account 
    the other features. Therefore, all other features will remain in the model. 
    It is also possible to force the inclusion of certain features in the final 
    list of selected features using the `force_inclusion` parameter.

    Parameters
    ----------
    forecaster : ForecasterAutoreg, ForecasterAutoregCustom
        Forecaster model.
    selector : object
        A feature selector from sklearn.feature_selection.
    y : pandas Series, pandas DataFrame
        Target time series to which the feature selection will be applied.
    exog : pandas Series, pandas DataFrame, default `None`
        Exogenous variable/s included as predictor/s. Must have the same
        number of observations as `y` and should be aligned so that y[i] is
        regressed on exog[i].
    select_only : str, default `None`
        Decide what type of features to include in the selection process. 

        - If `'autoreg'`, only autoregressive features (lags or custom 
        predictors) are evaluated by the selector. All exogenous features are 
        included in the output (`selected_exog`).
        - If `'exog'`, only exogenous features are evaluated without the presence
        of autoregressive features. All autoregressive features are included 
        in the output (`selected_autoreg`).
        - If `None`, all features are evaluated by the selector.
    force_inclusion : list, str, default `None`
        Features to force include in the final list of selected features.

        - If `list`, list of feature names to force include.
        - If `str`, regular expression to identify features to force include. 
        For example, if `force_inclusion="^sun_"`, all features that begin 
        with "sun_" will be included in the final list of selected features.
    subsample : int, float, default `0.5`
        Proportion of records to use for feature selection.
    random_state : int, default `123`
        Sets a seed for the random subsample so that the subsampling process 
        is always deterministic.
    verbose : bool, default `True`
        Print information about feature selection process.

    Returns
    -------
    selected_autoreg : list
        List of selected autoregressive features.
    selected_exog : list
        List of selected exogenous features.

    """

    valid_forecasters = [
        'ForecasterAutoreg',
        'ForecasterAutoregCustom'
    ]

    if type(forecaster).__name__ not in valid_forecasters:
        raise TypeError(
            f"`forecaster` must be one of the following classes: {valid_forecasters}."
        )

    if select_only not in ['autoreg', 'exog', None]:
        raise ValueError(
            "`select_only` must be one of the following values: 'autoreg', 'exog', None."
        )

    if subsample <= 0 or subsample > 1:
        raise ValueError(
            "`subsample` must be a number greater than 0 and less than or equal to 1."
        )

    forecaster = deepcopy(forecaster)
    forecaster.fitted = False
    X_train, y_train = forecaster.create_train_X_y(y=y, exog=exog)

    if hasattr(forecaster, 'lags'):
        autoreg_cols = [f"lag_{lag}" for lag in forecaster.lags]
    else:
        if forecaster.name_predictors is not None:
            autoreg_cols = forecaster.name_predictors
        else:
            autoreg_cols = [
                col
                for col in X_train.columns
                if re.match(r'^custom_predictor_\d+', col)
            ]
    exog_cols = [col for col in X_train.columns if col not in autoreg_cols]

    forced_autoreg = []
    forced_exog = []
    if force_inclusion is not None:
        if isinstance(force_inclusion, list):
            forced_autoreg = [col for col in force_inclusion if col in autoreg_cols]
            forced_exog = [col for col in force_inclusion if col in exog_cols]
        elif isinstance(force_inclusion, str):
            forced_autoreg = [col for col in autoreg_cols if re.match(force_inclusion, col)]
            forced_exog = [col for col in exog_cols if re.match(force_inclusion, col)]

    if select_only == 'autoreg':
        X_train = X_train.drop(columns=exog_cols)
    elif select_only == 'exog':
        X_train = X_train.drop(columns=autoreg_cols)

    if isinstance(subsample, float):
        subsample = int(len(X_train) * subsample)

    rng = np.random.default_rng(seed=random_state)
    sample = rng.choice(X_train.index, size=subsample, replace=False)
    X_train_sample = X_train.loc[sample, :]
    y_train_sample = y_train.loc[sample]
    selector.fit(X_train_sample, y_train_sample)
    selected_features = selector.get_feature_names_out()

    if select_only == 'exog':
        selected_autoreg = autoreg_cols
    else:
        selected_autoreg = [
            feature
            for feature in selected_features
            if feature in autoreg_cols
        ]

    if select_only == 'autoreg':
        selected_exog = exog_cols
    else:
        selected_exog = [
            feature
            for feature in selected_features
            if feature in exog_cols
        ]

    if force_inclusion is not None: 
        if select_only != 'autoreg':
            forced_exog_not_selected = set(forced_exog) - set(selected_features)
            selected_exog.extend(forced_exog_not_selected)
            selected_exog.sort(key=exog_cols.index)
        if select_only != 'exog':
            forced_autoreg_not_selected = set(forced_autoreg) - set(selected_features)
            selected_autoreg.extend(forced_autoreg_not_selected)
            selected_autoreg.sort(key=autoreg_cols.index)

    if len(selected_autoreg) == 0:
        warnings.warn(
            ("No autoregressive features have been selected. Since a Forecaster "
             "cannot be created without them, be sure to include at least one "
             "using the `force_inclusion` parameter.")
        )
    else:
        if hasattr(forecaster, 'lags'):
            selected_autoreg = [int(feature.replace('lag_', '')) 
                                for feature in selected_autoreg]

    if verbose:
        print(f"Recursive feature elimination ({selector.__class__.__name__})")
        print("--------------------------------" + "-" * len(selector.__class__.__name__))
        print(f"Total number of records available: {X_train.shape[0]}")
        print(f"Total number of records used for feature selection: {X_train_sample.shape[0]}")
        print(f"Number of features available: {X_train.shape[1]}") 
        print(f"    Autoreg (n={len(autoreg_cols)})")
        print(f"    Exog    (n={len(exog_cols)})")
        print(f"Number of features selected: {len(selected_features)}")
        print(f"    Autoreg (n={len(selected_autoreg)}) : {selected_autoreg}")
        print(f"    Exog    (n={len(selected_exog)}) : {selected_exog}")

    return selected_autoreg, selected_exog