Skip to content

model_selection_multiseries

backtesting_forecaster_multiseries(forecaster, series, steps, metric, initial_train_size, fixed_train_size=True, gap=0, skip_folds=None, allow_incomplete_fold=True, levels=None, add_aggregated_metric=True, exog=None, refit=False, interval=None, n_boot=500, random_state=123, in_sample_residuals=True, n_jobs='auto', verbose=False, show_progress=True, suppress_warnings=False)

Backtesting for multi-series and multivariate forecasters.

  • If refit is False, the model will be trained only once using the initial_train_size first observations.
  • If refit is True, the model is trained on each iteration, increasing the training set.
  • If refit is an integer, the model will be trained every that number of iterations.
  • If forecaster is already trained and initial_train_size is None, no initial train will be done and all data will be used to evaluate the model. However, the first len(forecaster.last_window) observations are needed to create the initial predictors, so no predictions are calculated for them.

A copy of the original forecaster is created so that it is not modified during the process.

Parameters:

Name Type Description Default
forecaster (ForecasterAutoregMultiSeries, ForecasterAutoregMultiSeriesCustom, ForecasterAutoregMultiVariate, ForecasterRnn)

Forecaster model.

 required
series pandas DataFrame, dict

Training time series.

 required
steps int

Number of steps to predict.

 required
metric (str, Callable, list)

Metric used to quantify the goodness of fit of the model.

  • If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
  • If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float.
  • If list: List containing multiple strings and/or Callables.
 required
initial_train_size int

Number of samples in the initial train split. If None and forecaster is already trained, no initial train is done and all data is used to evaluate the model. However, the first len(forecaster.last_window) observations are needed to create the initial predictors, so no predictions are calculated for them. This useful to backtest the model on the same data used to train it. None is only allowed when refit is False and forecaster is already trained.

 `None`
fixed_train_size bool

If True, train size doesn't increase but moves by steps in each iteration.

 `True`
gap int

Number of samples to be excluded after the end of each training set and before the test set.

 `0`
skip_folds (int, list)

If skip_folds is an integer, every 'skip_folds'-th is returned. If skip_folds is a list, the folds in the list are skipped. For example, if skip_folds = 3, and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If skip_folds is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].

 `None`
allow_incomplete_fold bool

Last fold is allowed to have a smaller number of samples than the test_size. If False, the last fold is excluded.

 `True`
levels (str, list)

Time series to be predicted. If None all levels will be predicted.

 `None`
add_aggregated_metric bool

If True, and multiple series (levels) are predicted, the aggregated metrics (average, weighted average and pooled) are also returned.

  • 'average': the average (arithmetic mean) of all levels.
  • 'weighted_average': the average of the metrics weighted by the number of predicted values of each level.
  • 'pooling': the values of all levels are pooled and then the metric is calculated.
 `True`
exog pandas Series, pandas DataFrame, dict

Exogenous variables.

 `None`
refit (bool, int)

Whether to re-fit the forecaster in each iteration. If refit is an integer, the Forecaster will be trained every that number of iterations.

 `False`
interval list

Confidence of the prediction interval estimated. Sequence of percentiles to compute, which must be between 0 and 100 inclusive. If None, no intervals are estimated.

 `None`
n_boot int

Number of bootstrapping iterations used to estimate prediction intervals.

 `500`
random_state int

Sets a seed to the random generator, so that boot intervals are always deterministic.

 `123`
in_sample_residuals bool

If True, residuals from the training data are used as proxy of prediction error to create prediction intervals. If False, out_sample_residuals are used if they are already stored inside the forecaster.

 `True`
n_jobs (int, auto)

The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting. New in version 0.9.0

 `'auto'`
verbose bool

Print number of folds and index of training and validation sets used for backtesting.

 `False`
show_progress bool

Whether to show a progress bar.

 `True`
suppress_warnings bool

If True, skforecast warnings will be suppressed during the backtesting process. See skforecast.exceptions.warn_skforecast_categories for more information.

 False

Returns:

Name Type Description
metrics_levels pandas DataFrame

Value(s) of the metric(s). Index are the levels and columns the metrics.
backtest_predictions pandas DataFrame

Value of predictions and their estimated interval if interval is not None. If there is more than one level, this structure will be repeated for each of them.

  • column pred: predictions.
  • column lower_bound: lower bound of the interval.
  • column upper_bound: upper bound of the interval.
Source code in skforecast\model_selection_multiseries\model_selection_multiseries.py 
 829
 830
 831
 832
 833
 834
 835
 836
 837
 838
 839
 840
 841
 842
 843
 844
 845
 846
 847
 848
 849
 850
 851
 852
 853
 854
 855
 856
 857
 858
 859
 860
 861
 862
 863
 864
 865
 866
 867
 868
 869
 870
 871
 872
 873
 874
 875
 876
 877
 878
 879
 880
 881
 882
 883
 884
 885
 886
 887
 888
 889
 890
 891
 892
 893
 894
 895
 896
 897
 898
 899
 900
 901
 902
 903
 904
 905
 906
 907
 908
 909
 910
 911
 912
 913
 914
 915
 916
 917
 918
 919
 920
 921
 922
 923
 924
 925
 926
 927
 928
 929
 930
 931
 932
 933
 934
 935
 936
 937
 938
 939
 940
 941
 942
 943
 944
 945
 946
 947
 948
 949
 950
 951
 952
 953
 954
 955
 956
 957
 958
 959
 960
 961
 962
 963
 964
 965
 966
 967
 968
 969
 970
 971
 972
 973
 974
 975
 976
 977
 978
 979
 980
 981
 982
 983
 984
 985
 986
 987
 988
 989
 990
 991
 992
 993
 994
 995
 996
 997
 998
 999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
def backtesting_forecaster_multiseries(
    forecaster: object,
    series: Union[pd.DataFrame, dict],
    steps: int,
    metric: Union[str, Callable, list],
    initial_train_size: Optional[int],
    fixed_train_size: bool = True,
    gap: int = 0,
    skip_folds: Optional[Union[int, list]] = None,
    allow_incomplete_fold: bool = True,
    levels: Optional[Union[str, list]] = None,
    add_aggregated_metric: bool = True,
    exog: Optional[Union[pd.Series, pd.DataFrame, dict]] = None,
    refit: Union[bool, int] = False,
    interval: Optional[list] = None,
    n_boot: int = 500,
    random_state: int = 123,
    in_sample_residuals: bool = True,
    n_jobs: Union[int, str] = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False
) -> Tuple[pd.DataFrame, pd.DataFrame]:
    """
    Backtesting for multi-series and multivariate forecasters.

    - If `refit` is `False`, the model will be trained only once using the 
    `initial_train_size` first observations. 
    - If `refit` is `True`, the model is trained on each iteration, increasing
    the training set. 
    - If `refit` is an `integer`, the model will be trained every that number 
    of iterations.
    - If `forecaster` is already trained and `initial_train_size` is `None`,
    no initial train will be done and all data will be used to evaluate the model.
    However, the first `len(forecaster.last_window)` observations are needed
    to create the initial predictors, so no predictions are calculated for them.

    A copy of the original forecaster is created so that it is not modified during 
    the process.

    Parameters
    ----------
    forecaster : ForecasterAutoregMultiSeries, ForecasterAutoregMultiSeriesCustom, ForecasterAutoregMultiVariate, ForecasterRnn
        Forecaster model.
    series : pandas DataFrame, dict
        Training time series.
    steps : int
        Number of steps to predict.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    initial_train_size : int, default `None`
        Number of samples in the initial train split. If `None` and `forecaster` is 
        already trained, no initial train is done and all data is used to evaluate the 
        model. However, the first `len(forecaster.last_window)` observations are needed 
        to create the initial predictors, so no predictions are calculated for them. 
        This useful to backtest the model on the same data used to train it.
        `None` is only allowed when `refit` is `False` and `forecaster` is already
        trained.
    fixed_train_size : bool, default `True`
        If True, train size doesn't increase but moves by `steps` in each iteration.
    gap : int, default `0`
        Number of samples to be excluded after the end of each training set and 
        before the test set.
    skip_folds : int, list, default `None`
        If `skip_folds` is an integer, every 'skip_folds'-th is returned. If `skip_folds`
        is a list, the folds in the list are skipped. For example, if `skip_folds = 3`,
        and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If `skip_folds`
        is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].
    allow_incomplete_fold : bool, default `True`
        Last fold is allowed to have a smaller number of samples than the 
        `test_size`. If `False`, the last fold is excluded.
    levels : str, list, default `None`
        Time series to be predicted. If `None` all levels will be predicted.
    add_aggregated_metric : bool, default `True`
        If `True`, and multiple series (`levels`) are predicted, the aggregated
        metrics (average, weighted average and pooled) are also returned.

        - 'average': the average (arithmetic mean) of all levels.
        - 'weighted_average': the average of the metrics weighted by the number of
        predicted values of each level.
        - 'pooling': the values of all levels are pooled and then the metric is
        calculated.
    exog : pandas Series, pandas DataFrame, dict, default `None`
        Exogenous variables.
    refit : bool, int, default `False`
        Whether to re-fit the forecaster in each iteration. If `refit` is an 
        integer, the Forecaster will be trained every that number of iterations.
    interval : list, default `None`
        Confidence of the prediction interval estimated. Sequence of percentiles
        to compute, which must be between 0 and 100 inclusive. If `None`, no
        intervals are estimated.
    n_boot : int, default `500`
        Number of bootstrapping iterations used to estimate prediction
        intervals.
    random_state : int, default `123`
        Sets a seed to the random generator, so that boot intervals are always 
        deterministic.
    in_sample_residuals : bool, default `True`
        If `True`, residuals from the training data are used as proxy of prediction 
        error to create prediction intervals. If `False`, out_sample_residuals 
        are used if they are already stored inside the forecaster.
    n_jobs : int, 'auto', default `'auto'`
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting.
        **New in version 0.9.0**
    verbose : bool, default `False`
        Print number of folds and index of training and validation sets used 
        for backtesting.
    show_progress : bool, default `True`
        Whether to show a progress bar.
    suppress_warnings: bool, default `False`
        If `True`, skforecast warnings will be suppressed during the backtesting 
        process. See skforecast.exceptions.warn_skforecast_categories for more
        information.

    Returns
    -------
    metrics_levels : pandas DataFrame
        Value(s) of the metric(s). Index are the levels and columns the metrics.
    backtest_predictions : pandas DataFrame
        Value of predictions and their estimated interval if `interval` is not `None`.
        If there is more than one level, this structure will be repeated for each of them.

        - column pred: predictions.
        - column lower_bound: lower bound of the interval.
        - column upper_bound: upper bound of the interval.

    """

    multi_series_forecasters = [
        'ForecasterAutoregMultiSeries', 
        'ForecasterAutoregMultiSeriesCustom', 
        'ForecasterAutoregMultiVariate',
        'ForecasterRnn'
    ]

    forecaster_name = type(forecaster).__name__

    if forecaster_name not in multi_series_forecasters:
        raise TypeError(
            (f"`forecaster` must be of type {multi_series_forecasters}, "
             f"for all other types of forecasters use the functions available in "
             f"the `model_selection` module. Got {forecaster_name}")
        )

    check_backtesting_input(
        forecaster            = forecaster,
        steps                 = steps,
        metric                = metric,
        add_aggregated_metric = add_aggregated_metric,
        series                = series,
        exog                  = exog,
        initial_train_size    = initial_train_size,
        fixed_train_size      = fixed_train_size,
        gap                   = gap,
        skip_folds            = skip_folds,
        allow_incomplete_fold = allow_incomplete_fold,
        refit                 = refit,
        interval              = interval,
        n_boot                = n_boot,
        random_state          = random_state,
        in_sample_residuals   = in_sample_residuals,
        n_jobs                = n_jobs,
        verbose               = verbose,
        show_progress         = show_progress,
        suppress_warnings     = suppress_warnings
    )

    metrics_levels, backtest_predictions = _backtesting_forecaster_multiseries(
        forecaster            = forecaster,
        series                = series,
        steps                 = steps,
        levels                = levels,
        metric                = metric,
        add_aggregated_metric = add_aggregated_metric,
        initial_train_size    = initial_train_size,
        fixed_train_size      = fixed_train_size,
        gap                   = gap,
        skip_folds            = skip_folds,
        allow_incomplete_fold = allow_incomplete_fold,
        exog                  = exog,
        refit                 = refit,
        interval              = interval,
        n_boot                = n_boot,
        random_state          = random_state,
        in_sample_residuals   = in_sample_residuals,
        n_jobs                = n_jobs,
        verbose               = verbose,
        show_progress         = show_progress,
        suppress_warnings     = suppress_warnings
    )

    return metrics_levels, backtest_predictions

grid_search_forecaster_multiseries(forecaster, series, param_grid, steps, metric, initial_train_size, aggregate_metric=['weighted_average', 'average', 'pooling'], fixed_train_size=True, gap=0, skip_folds=None, allow_incomplete_fold=True, levels=None, exog=None, lags_grid=None, refit=False, return_best=True, n_jobs='auto', verbose=True, show_progress=True, suppress_warnings=False, output_file=None)

Exhaustive search over specified parameter values for a Forecaster object. Validation is done using multi-series backtesting.

Parameters:

Name Type Description Default
forecaster (ForecasterAutoregMultiSeries, ForecasterAutoregMultiSeriesCustom, ForecasterAutoregMultiVariate)

Forecaster model.

 required
series pandas DataFrame, dict

Training time series.

 required
param_grid dict

Dictionary with parameters names (str) as keys and lists of parameter settings to try as values.

 required
steps int

Number of steps to predict.

 required
metric (str, Callable, list)

Metric used to quantify the goodness of fit of the model.

  • If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
  • If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float.
  • If list: List containing multiple strings and/or Callables.
 required
initial_train_size int

Number of samples in the initial train split.

 required
aggregate_metric (str, list)

Aggregation method/s used to combine the metric/s of all levels (series) when multiple levels are predicted. If list, the first aggregation method is used to select the best parameters.

  • 'average': the average (arithmetic mean) of all levels.
  • 'weighted_average': the average of the metrics weighted by the number of predicted values of each level.
  • 'pooling': the values of all levels are pooled and then the metric is calculated.
 `['weighted_average', 'average', 'pooling']`
fixed_train_size bool

If True, train size doesn't increase but moves by steps in each iteration.

 `True`
gap int

Number of samples to be excluded after the end of each training set and before the test set.

 `0`
skip_folds (int, list)

If skip_folds is an integer, every 'skip_folds'-th is returned. If skip_folds is a list, the folds in the list are skipped. For example, if skip_folds = 3, and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If skip_folds is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].

 `None`
allow_incomplete_fold bool

Last fold is allowed to have a smaller number of samples than the test_size. If False, the last fold is excluded.

 `True`
levels (str, list)

level (str) or levels (list) at which the forecaster is optimized. If None, all levels are taken into account.

 `None`
exog pandas Series, pandas DataFrame, dict

Exogenous variables.

 `None`
lags_grid (list, dict)

Lists of lags to try, containing int, lists, numpy ndarray, or range objects. If dict, the keys are used as labels in the results DataFrame, and the values are used as the lists of lags to try. Ignored if the forecaster is an instance of ForecasterAutoregCustom or ForecasterAutoregMultiSeriesCustom.

 `None`
refit (bool, int)

Whether to re-fit the forecaster in each iteration. If refit is an integer, the Forecaster will be trained every that number of iterations.

 `False`
return_best bool

Refit the forecaster using the best found parameters on the whole data.

 `True`
n_jobs (int, auto)

The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting. New in version 0.9.0

 `'auto'`
verbose bool

Print number of folds used for cv or backtesting.

 `True`
show_progress bool

Whether to show a progress bar.

 `True`
suppress_warnings bool

If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information.

 False
output_file str

Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file. New in version 0.12.0

 `None`

Returns:

Name Type Description
results pandas DataFrame

Results for each combination of parameters.

  • column levels: levels configuration for each iteration.
  • column lags: lags configuration for each iteration.
  • column lags_label: descriptive label or alias for the lags.
  • column params: parameters configuration for each iteration.
  • column metric: metric value estimated for each iteration. The resulting metric will be the average of the optimization of all levels.
  • additional n columns with param = value.
Source code in skforecast\model_selection_multiseries\model_selection_multiseries.py 
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
def grid_search_forecaster_multiseries(
    forecaster: object,
    series: Union[pd.DataFrame, dict],
    param_grid: dict,
    steps: int,
    metric: Union[str, Callable, list],
    initial_train_size: int,
    aggregate_metric: Union[str, list] = ['weighted_average', 'average', 'pooling'],
    fixed_train_size: bool = True,
    gap: int = 0,
    skip_folds: Optional[Union[int, list]] = None,
    allow_incomplete_fold: bool = True,
    levels: Optional[Union[str, list]] = None,
    exog: Optional[Union[pd.Series, pd.DataFrame, dict]] = None,
    lags_grid: Optional[Union[list, dict]] = None,
    refit: Union[bool, int] = False,
    return_best: bool = True,
    n_jobs: Union[int, str] = 'auto',
    verbose: bool = True,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: Optional[str] = None
) -> pd.DataFrame:
    """
    Exhaustive search over specified parameter values for a Forecaster object.
    Validation is done using multi-series backtesting.

    Parameters
    ----------
    forecaster : ForecasterAutoregMultiSeries, ForecasterAutoregMultiSeriesCustom, ForecasterAutoregMultiVariate
        Forecaster model.
    series : pandas DataFrame, dict
        Training time series.
    param_grid : dict
        Dictionary with parameters names (`str`) as keys and lists of parameter
        settings to try as values.
    steps : int
        Number of steps to predict.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    initial_train_size : int 
        Number of samples in the initial train split.
    aggregate_metric : str, list, default `['weighted_average', 'average', 'pooling']`
        Aggregation method/s used to combine the metric/s of all levels (series)
        when multiple levels are predicted. If list, the first aggregation method
        is used to select the best parameters.

        - 'average': the average (arithmetic mean) of all levels.
        - 'weighted_average': the average of the metrics weighted by the number of
        predicted values of each level.
        - 'pooling': the values of all levels are pooled and then the metric is
        calculated.
    fixed_train_size : bool, default `True`
        If True, train size doesn't increase but moves by `steps` in each iteration.
    gap : int, default `0`
        Number of samples to be excluded after the end of each training set and 
        before the test set.
    skip_folds : int, list, default `None`
        If `skip_folds` is an integer, every 'skip_folds'-th is returned. If `skip_folds`
        is a list, the folds in the list are skipped. For example, if `skip_folds = 3`,
        and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If `skip_folds`
        is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].
    allow_incomplete_fold : bool, default `True`
        Last fold is allowed to have a smaller number of samples than the 
        `test_size`. If `False`, the last fold is excluded.
    levels : str, list, default `None`
        level (`str`) or levels (`list`) at which the forecaster is optimized. 
        If `None`, all levels are taken into account.
    exog : pandas Series, pandas DataFrame, dict, default `None`
        Exogenous variables.
    lags_grid : list, dict, default `None`
        Lists of lags to try, containing int, lists, numpy ndarray, or range 
        objects. If `dict`, the keys are used as labels in the `results` 
        DataFrame, and the values are used as the lists of lags to try. Ignored 
        if the forecaster is an instance of `ForecasterAutoregCustom` or 
        `ForecasterAutoregMultiSeriesCustom`.
    refit : bool, int, default `False`
        Whether to re-fit the forecaster in each iteration. If `refit` is an 
        integer, the Forecaster will be trained every that number of iterations.
    return_best : bool, default `True`
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default `'auto'`
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting.
        **New in version 0.9.0**
    verbose : bool, default `True`
        Print number of folds used for cv or backtesting.
    show_progress : bool, default `True`
        Whether to show a progress bar.
    suppress_warnings: bool, default `False`
        If `True`, skforecast warnings will be suppressed during the hyperparameter 
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.
    output_file : str, default `None`
        Specifies the filename or full path where the results should be saved. 
        The results will be saved in a tab-separated values (TSV) format. If 
        `None`, the results will not be saved to a file.
        **New in version 0.12.0**

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column levels: levels configuration for each iteration.
        - column lags: lags configuration for each iteration.
        - column lags_label: descriptive label or alias for the lags.
        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration. The resulting 
        metric will be the average of the optimization of all levels.
        - additional n columns with param = value.

    """

    param_grid = list(ParameterGrid(param_grid))

    results = _evaluate_grid_hyperparameters_multiseries(
                  forecaster            = forecaster,
                  series                = series,
                  param_grid            = param_grid,
                  steps                 = steps,
                  metric                = metric,
                  aggregate_metric      = aggregate_metric,
                  initial_train_size    = initial_train_size,
                  fixed_train_size      = fixed_train_size,
                  gap                   = gap,
                  skip_folds            = skip_folds,
                  allow_incomplete_fold = allow_incomplete_fold,
                  levels                = levels,
                  exog                  = exog,
                  lags_grid             = lags_grid,
                  refit                 = refit,
                  n_jobs                = n_jobs,
                  return_best           = return_best,
                  verbose               = verbose,
                  show_progress         = show_progress,
                  suppress_warnings     = suppress_warnings,
                  output_file           = output_file
              )

    return results

random_search_forecaster_multiseries(forecaster, series, param_distributions, steps, metric, initial_train_size, aggregate_metric=['weighted_average', 'average', 'pooling'], fixed_train_size=True, gap=0, skip_folds=None, allow_incomplete_fold=True, levels=None, exog=None, lags_grid=None, refit=False, n_iter=10, random_state=123, return_best=True, n_jobs='auto', verbose=True, show_progress=True, suppress_warnings=False, output_file=None)

Random search over specified parameter values or distributions for a Forecaster object. Validation is done using multi-series backtesting.

Parameters:

Name Type Description Default
forecaster (ForecasterAutoregMultiSeries, ForecasterAutoregMultiSeriesCustom, ForecasterAutoregMultiVariate)

Forecaster model.

 required
series pandas DataFrame, dict

Training time series.

 required
param_distributions dict

Dictionary with parameters names (str) as keys and distributions or lists of parameters to try.

 required
steps int

Number of steps to predict.

 required
metric (str, Callable, list)

Metric used to quantify the goodness of fit of the model.

  • If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
  • If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float.
  • If list: List containing multiple strings and/or Callables.
 required
initial_train_size int

Number of samples in the initial train split.

 required
aggregate_metric (str, list)

Aggregation method/s used to combine the metric/s of all levels (series) when multiple levels are predicted. If list, the first aggregation method is used to select the best parameters.

  • 'average': the average (arithmetic mean) of all levels.
  • 'weighted_average': the average of the metrics weighted by the number of predicted values of each level.
  • 'pooling': the values of all levels are pooled and then the metric is calculated.
 `['weighted_average', 'average', 'pooling']`
fixed_train_size bool

If True, train size doesn't increase but moves by steps in each iteration.

 `True`
gap int

Number of samples to be excluded after the end of each training set and before the test set.

 `0`
skip_folds (int, list)

If skip_folds is an integer, every 'skip_folds'-th is returned. If skip_folds is a list, the folds in the list are skipped. For example, if skip_folds = 3, and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If skip_folds is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].

 `None`
allow_incomplete_fold bool

Last fold is allowed to have a smaller number of samples than the test_size. If False, the last fold is excluded.

 `True`
levels (str, list)

level (str) or levels (list) at which the forecaster is optimized. If None, all levels are taken into account.

 `None`
exog pandas Series, pandas DataFrame, dict

Exogenous variables.

 `None`
lags_grid (list, dict)

Lists of lags to try, containing int, lists, numpy ndarray, or range objects. If dict, the keys are used as labels in the results DataFrame, and the values are used as the lists of lags to try. Ignored if the forecaster is an instance of ForecasterAutoregCustom or ForecasterAutoregMultiSeriesCustom.

 `None`
refit (bool, int)

Whether to re-fit the forecaster in each iteration. If refit is an integer, the Forecaster will be trained every that number of iterations.

 `False`
n_iter int

Number of parameter settings that are sampled per lags configuration. n_iter trades off runtime vs quality of the solution.

 `10`
random_state int

Sets a seed to the random sampling for reproducible output.

 `123`
return_best bool

Refit the forecaster using the best found parameters on the whole data.

 `True`
n_jobs (int, auto)

The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting. New in version 0.9.0

 `'auto'`
verbose bool

Print number of folds used for cv or backtesting.

 `True`
show_progress bool

Whether to show a progress bar.

 `True`
suppress_warnings bool

If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information.

 False
output_file str

Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file. New in version 0.12.0

 `None`

Returns:

Name Type Description
results pandas DataFrame

Results for each combination of parameters.

  • column levels: levels configuration for each iteration.
  • column lags: lags configuration for each iteration.
  • column lags_label: descriptive label or alias for the lags.
  • column params: parameters configuration for each iteration.
  • column metric: metric value estimated for each iteration. The resulting metric will be the average of the optimization of all levels.
  • additional n columns with param = value.
Source code in skforecast\model_selection_multiseries\model_selection_multiseries.py 
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
def random_search_forecaster_multiseries(
    forecaster: object,
    series: Union[pd.DataFrame, dict],
    param_distributions: dict,
    steps: int,
    metric: Union[str, Callable, list],
    initial_train_size: int,
    aggregate_metric: Union[str, list] = ['weighted_average', 'average', 'pooling'],
    fixed_train_size: bool = True,
    gap: int = 0,
    skip_folds: Optional[Union[int, list]] = None,
    allow_incomplete_fold: bool = True,
    levels: Optional[Union[str, list]] = None,
    exog: Optional[Union[pd.Series, pd.DataFrame, dict]] = None,
    lags_grid: Optional[Union[list, dict]] = None,
    refit: Union[bool, int] = False,
    n_iter: int = 10,
    random_state: int = 123,
    return_best: bool = True,
    n_jobs: Union[int, str] = 'auto',
    verbose: bool = True,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: Optional[str] = None
) -> pd.DataFrame:
    """
    Random search over specified parameter values or distributions for a Forecaster 
    object. Validation is done using multi-series backtesting.

    Parameters
    ----------
    forecaster : ForecasterAutoregMultiSeries, ForecasterAutoregMultiSeriesCustom, ForecasterAutoregMultiVariate
        Forecaster model.
    series : pandas DataFrame, dict
        Training time series.
    param_distributions : dict
        Dictionary with parameters names (`str`) as keys and distributions or 
        lists of parameters to try.
    steps : int
        Number of steps to predict.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    initial_train_size : int 
        Number of samples in the initial train split.
    aggregate_metric : str, list, default `['weighted_average', 'average', 'pooling']`
        Aggregation method/s used to combine the metric/s of all levels (series)
        when multiple levels are predicted. If list, the first aggregation method
        is used to select the best parameters.

        - 'average': the average (arithmetic mean) of all levels.
        - 'weighted_average': the average of the metrics weighted by the number of
        predicted values of each level.
        - 'pooling': the values of all levels are pooled and then the metric is
        calculated.
    fixed_train_size : bool, default `True`
        If True, train size doesn't increase but moves by `steps` in each iteration.
    gap : int, default `0`
        Number of samples to be excluded after the end of each training set and 
        before the test set.
    skip_folds : int, list, default `None`
        If `skip_folds` is an integer, every 'skip_folds'-th is returned. If `skip_folds`
        is a list, the folds in the list are skipped. For example, if `skip_folds = 3`,
        and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If `skip_folds`
        is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].
    allow_incomplete_fold : bool, default `True`
        Last fold is allowed to have a smaller number of samples than the 
        `test_size`. If `False`, the last fold is excluded.
    levels : str, list, default `None`
        level (`str`) or levels (`list`) at which the forecaster is optimized. 
        If `None`, all levels are taken into account.
    exog : pandas Series, pandas DataFrame, dict, default `None`
        Exogenous variables.
    lags_grid : list, dict, default `None`
        Lists of lags to try, containing int, lists, numpy ndarray, or range 
        objects. If `dict`, the keys are used as labels in the `results` 
        DataFrame, and the values are used as the lists of lags to try. Ignored 
        if the forecaster is an instance of `ForecasterAutoregCustom` or 
        `ForecasterAutoregMultiSeriesCustom`.
    refit : bool, int, default `False`
        Whether to re-fit the forecaster in each iteration. If `refit` is an 
        integer, the Forecaster will be trained every that number of iterations.
    n_iter : int, default `10`
        Number of parameter settings that are sampled per lags configuration. 
        n_iter trades off runtime vs quality of the solution.
    random_state : int, default `123`
        Sets a seed to the random sampling for reproducible output.
    return_best : bool, default `True`
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default `'auto'`
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting.
        **New in version 0.9.0**
    verbose : bool, default `True`
        Print number of folds used for cv or backtesting.
    show_progress : bool, default `True`
        Whether to show a progress bar.
    suppress_warnings: bool, default `False`
        If `True`, skforecast warnings will be suppressed during the hyperparameter 
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.
    output_file : str, default `None`
        Specifies the filename or full path where the results should be saved. 
        The results will be saved in a tab-separated values (TSV) format. If 
        `None`, the results will not be saved to a file.
        **New in version 0.12.0**

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column levels: levels configuration for each iteration.
        - column lags: lags configuration for each iteration.
        - column lags_label: descriptive label or alias for the lags.
        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration. The resulting 
        metric will be the average of the optimization of all levels.
        - additional n columns with param = value.

    """

    param_grid = list(ParameterSampler(param_distributions, n_iter=n_iter, 
                                       random_state=random_state))

    results = _evaluate_grid_hyperparameters_multiseries(
                  forecaster            = forecaster,
                  series                = series,
                  param_grid            = param_grid,
                  steps                 = steps,
                  metric                = metric,
                  aggregate_metric      = aggregate_metric,
                  initial_train_size    = initial_train_size,
                  fixed_train_size      = fixed_train_size,
                  gap                   = gap,
                  skip_folds            = skip_folds,
                  allow_incomplete_fold = allow_incomplete_fold,
                  levels                = levels,
                  exog                  = exog,
                  lags_grid             = lags_grid,
                  refit                 = refit,
                  return_best           = return_best,
                  n_jobs                = n_jobs,
                  verbose               = verbose,
                  show_progress         = show_progress,
                  suppress_warnings     = suppress_warnings,
                 output_file            = output_file
              )

    return results

bayesian_search_forecaster_multiseries(forecaster, series, search_space, steps, metric, initial_train_size, aggregate_metric=['weighted_average', 'average', 'pooling'], fixed_train_size=True, gap=0, skip_folds=None, allow_incomplete_fold=True, levels=None, exog=None, refit=False, n_trials=10, random_state=123, return_best=True, n_jobs='auto', verbose=True, show_progress=True, suppress_warnings=False, output_file=None, engine='optuna', kwargs_create_study={}, kwargs_study_optimize={})

Bayesian optimization for a Forecaster object using multi-series backtesting and optuna library. New in version 0.12.0

Parameters:

Name Type Description Default
forecaster (ForecasterAutoregMultiSeries, ForecasterAutoregMultiSeriesCustom, ForecasterAutoregMultiVariate)

Forecaster model.

 required
series pandas DataFrame, dict

Training time series.

 required
search_space Callable

Function with argument trial which returns a dictionary with parameters names (str) as keys and Trial object from optuna (trial.suggest_float, trial.suggest_int, trial.suggest_categorical) as values.

 required
steps int

Number of steps to predict.

 required
metric (str, Callable, list)

Metric used to quantify the goodness of fit of the model.

  • If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
  • If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float.
  • If list: List containing multiple strings and/or Callables.
 required
initial_train_size int

Number of samples in the initial train split.

 required
aggregate_metric (str, list)

Aggregation method/s used to combine the metric/s of all levels (series) when multiple levels are predicted. If list, the first aggregation method is used to select the best parameters.

  • 'average': the average (arithmetic mean) of all levels.
  • 'weighted_average': the average of the metrics weighted by the number of predicted values of each level.
  • 'pooling': the values of all levels are pooled and then the metric is calculated.
 `['weighted_average', 'average', 'pooling']`
fixed_train_size bool

If True, train size doesn't increase but moves by steps in each iteration.

 `True`
gap int

Number of samples to be excluded after the end of each training set and before the test set.

 `0`
skip_folds (int, list)

If skip_folds is an integer, every 'skip_folds'-th is returned. If skip_folds is a list, the folds in the list are skipped. For example, if skip_folds = 3, and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If skip_folds is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].

 `None`
allow_incomplete_fold bool

Last fold is allowed to have a smaller number of samples than the test_size. If False, the last fold is excluded.

 `True`
levels (str, list)

level (str) or levels (list) at which the forecaster is optimized. If None, all levels are taken into account.

 `None`
exog pandas Series, pandas DataFrame, dict

Exogenous variables.

 `None`
refit (bool, int)

Whether to re-fit the forecaster in each iteration. If refit is an integer, the Forecaster will be trained every that number of iterations.

 `False`
n_trials int

Number of parameter settings that are sampled in each lag configuration.

 `10`
random_state int

Sets a seed to the sampling for reproducible output.

 `123`
return_best bool

Refit the forecaster using the best found parameters on the whole data.

 `True`
n_jobs (int, auto)

The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting.

 `'auto'`
verbose bool

Print number of folds used for cv or backtesting.

 `True`
show_progress bool

Whether to show a progress bar.

 `True`
suppress_warnings bool

If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information.

 False
output_file str

Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file. New in version 0.12.0

 `None`
engine str

Bayesian optimization runs through the optuna library.

 `'optuna'`
kwargs_create_study dict

Keyword arguments (key, value mappings) to pass to optuna.create_study(). If default, the direction is set to 'minimize' and a TPESampler(seed=123) sampler is used during optimization.

 `{}`
kwargs_study_optimize dict

Other keyword arguments (key, value mappings) to pass to study.optimize().

 `{}`

Returns:

Name Type Description
results pandas DataFrame

Results for each combination of parameters.

  • column levels: levels configuration for each iteration.
  • column lags: lags configuration for each iteration.
  • column params: parameters configuration for each iteration.
  • column metric: metric value estimated for each iteration. The resulting metric will be the average of the optimization of all levels.
  • additional n columns with param = value.
best_trial optuna object

The best optimization result returned as a FrozenTrial optuna object.
Source code in skforecast\model_selection_multiseries\model_selection_multiseries.py 
1623
1624
1625
1626
1627
1628
1629
1630
1631
1632
1633
1634
1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
1654
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
1670
1671
1672
1673
1674
1675
1676
1677
1678
1679
1680
1681
1682
1683
1684
1685
1686
1687
1688
1689
1690
1691
1692
1693
1694
1695
1696
1697
1698
1699
1700
1701
1702
1703
1704
1705
1706
1707
1708
1709
1710
1711
1712
1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
1725
1726
1727
1728
1729
1730
1731
1732
1733
1734
1735
1736
1737
1738
1739
1740
1741
1742
1743
1744
1745
1746
1747
1748
1749
1750
1751
1752
1753
1754
1755
1756
1757
1758
1759
1760
1761
1762
1763
1764
1765
1766
1767
1768
1769
1770
1771
1772
1773
1774
1775
1776
1777
1778
1779
1780
1781
1782
1783
1784
1785
1786
1787
1788
1789
1790
1791
1792
1793
1794
1795
def bayesian_search_forecaster_multiseries(
    forecaster: object,
    series: Union[pd.DataFrame, dict],
    search_space: Callable,
    steps: int,
    metric: Union[str, Callable, list],
    initial_train_size: int,
    aggregate_metric: Union[str, list] = ['weighted_average', 'average', 'pooling'],
    fixed_train_size: bool = True,
    gap: int = 0,
    skip_folds: Optional[Union[int, list]] = None,
    allow_incomplete_fold: bool = True,
    levels: Optional[Union[str, list]] = None,
    exog: Optional[Union[pd.Series, pd.DataFrame, dict]] = None,
    refit: Union[bool, int] = False,
    n_trials: int = 10,
    random_state: int = 123,
    return_best: bool = True,
    n_jobs: Union[int, str] = 'auto',
    verbose: bool = True,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: Optional[str] = None,
    engine: str = 'optuna',
    kwargs_create_study: dict = {},
    kwargs_study_optimize: dict = {}
) -> Tuple[pd.DataFrame, object]:
    """
    Bayesian optimization for a Forecaster object using multi-series backtesting 
    and optuna library.
    **New in version 0.12.0**

    Parameters
    ----------
    forecaster : ForecasterAutoregMultiSeries, ForecasterAutoregMultiSeriesCustom, ForecasterAutoregMultiVariate
        Forecaster model.
    series : pandas DataFrame, dict
        Training time series.
    search_space : Callable
        Function with argument `trial` which returns a dictionary with parameters names 
        (`str`) as keys and Trial object from optuna (trial.suggest_float, 
        trial.suggest_int, trial.suggest_categorical) as values.
    steps : int
        Number of steps to predict.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    initial_train_size : int 
        Number of samples in the initial train split.
    aggregate_metric : str, list, default `['weighted_average', 'average', 'pooling']`
        Aggregation method/s used to combine the metric/s of all levels (series)
        when multiple levels are predicted. If list, the first aggregation method
        is used to select the best parameters.

        - 'average': the average (arithmetic mean) of all levels.
        - 'weighted_average': the average of the metrics weighted by the number of
        predicted values of each level.
        - 'pooling': the values of all levels are pooled and then the metric is
        calculated.
    fixed_train_size : bool, default `True`
        If True, train size doesn't increase but moves by `steps` in each iteration.
    gap : int, default `0`
        Number of samples to be excluded after the end of each training set and 
        before the test set.
    skip_folds : int, list, default `None`
        If `skip_folds` is an integer, every 'skip_folds'-th is returned. If `skip_folds`
        is a list, the folds in the list are skipped. For example, if `skip_folds = 3`,
        and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If `skip_folds`
        is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].
    allow_incomplete_fold : bool, default `True`
        Last fold is allowed to have a smaller number of samples than the 
        `test_size`. If `False`, the last fold is excluded.
    levels : str, list, default `None`
        level (`str`) or levels (`list`) at which the forecaster is optimized. 
        If `None`, all levels are taken into account.
    exog : pandas Series, pandas DataFrame, dict, default `None`
        Exogenous variables.
    refit : bool, int, default `False`
        Whether to re-fit the forecaster in each iteration. If `refit` is an 
        integer, the Forecaster will be trained every that number of iterations.
    n_trials : int, default `10`
        Number of parameter settings that are sampled in each lag configuration.
    random_state : int, default `123`
        Sets a seed to the sampling for reproducible output.
    return_best : bool, default `True`
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default `'auto'`
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting.
    verbose : bool, default `True`
        Print number of folds used for cv or backtesting.
    show_progress : bool, default `True`
        Whether to show a progress bar.
    suppress_warnings: bool, default `False`
        If `True`, skforecast warnings will be suppressed during the hyperparameter
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.
    output_file : str, default `None`
        Specifies the filename or full path where the results should be saved. 
        The results will be saved in a tab-separated values (TSV) format. If 
        `None`, the results will not be saved to a file.
        **New in version 0.12.0**
    engine : str, default `'optuna'`
        Bayesian optimization runs through the optuna library.
    kwargs_create_study : dict, default `{}`
        Keyword arguments (key, value mappings) to pass to optuna.create_study().
        If default, the direction is set to 'minimize' and a TPESampler(seed=123) 
        sampler is used during optimization.
    kwargs_study_optimize : dict, default `{}`
        Other keyword arguments (key, value mappings) to pass to study.optimize().

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column levels: levels configuration for each iteration.
        - column lags: lags configuration for each iteration.
        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration. The resulting 
        metric will be the average of the optimization of all levels.
        - additional n columns with param = value.
    best_trial : optuna object
        The best optimization result returned as a FrozenTrial optuna object.

    """

    if return_best and exog is not None and (len(exog) != len(series)):
        raise ValueError(
            (f"`exog` must have same number of samples as `series`. "
             f"length `exog`: ({len(exog)}), length `series`: ({len(series)})")
        )

    if engine not in ['optuna']:
        raise ValueError(
            f"`engine` only allows 'optuna', got {engine}."
        )

    results, best_trial = _bayesian_search_optuna_multiseries(
                              forecaster            = forecaster,
                              series                = series,
                              exog                  = exog,
                              levels                = levels, 
                              search_space          = search_space,
                              steps                 = steps,
                              metric                = metric,
                              aggregate_metric      = aggregate_metric,
                              refit                 = refit,
                              initial_train_size    = initial_train_size,
                              fixed_train_size      = fixed_train_size,
                              gap                   = gap,
                              skip_folds            = skip_folds,
                              allow_incomplete_fold = allow_incomplete_fold,
                              n_trials              = n_trials,
                              random_state          = random_state,
                              return_best           = return_best,
                              n_jobs                = n_jobs,
                              verbose               = verbose,
                              show_progress         = show_progress,
                              suppress_warnings     = suppress_warnings,
                              output_file           = output_file,
                              kwargs_create_study   = kwargs_create_study,
                              kwargs_study_optimize = kwargs_study_optimize
                          )

    return results, best_trial

select_features_multiseries(forecaster, selector, series, exog=None, select_only=None, force_inclusion=None, subsample=0.5, random_state=123, verbose=True)

Feature selection using any of the sklearn.feature_selection module selectors (such as RFECV, SelectFromModel, etc.). Two groups of features are evaluated: autoregressive features and exogenous features. By default, the selection process is performed on both sets of features at the same time, so that the most relevant autoregressive and exogenous features are selected. However, using the select_only argument, the selection process can focus only on the autoregressive or exogenous features without taking into account the other features. Therefore, all other features will remain in the model. It is also possible to force the inclusion of certain features in the final list of selected features using the force_inclusion parameter.

Parameters:

Name Type Description Default
forecaster (ForecasterAutoregMultiSeries, ForecasterAutoregMultiseriesCustom)

Forecaster model.

 required
selector object

A feature selector from sklearn.feature_selection.

 required
series pandas DataFrame

Target time series to which the feature selection will be applied.

 required
exog pandas Series, pandas DataFrame, dict

Exogenous variables.

 `None`
select_only str

Decide what type of features to include in the selection process.

  • If 'autoreg', only autoregressive features (lags or custom predictors) are evaluated by the selector. All exogenous features are included in the output (selected_exog).
  • If 'exog', only exogenous features are evaluated without the presence of autoregressive features. All autoregressive features are included in the output (selected_autoreg).
  • If None, all features are evaluated by the selector.
 `None`
force_inclusion (list, str)

Features to force include in the final list of selected features.

  • If list, list of feature names to force include.
  • If str, regular expression to identify features to force include. For example, if force_inclusion="^sun_", all features that begin with "sun_" will be included in the final list of selected features.
 `None`
subsample (int, float)

Proportion of records to use for feature selection.

 `0.5`
random_state int

Sets a seed for the random subsample so that the subsampling process is always deterministic.

 `123`
verbose bool

Print information about feature selection process.

 `True`

Returns:

Name Type Description
selected_autoreg list

List of selected autoregressive features.
selected_exog list

List of selected exogenous features.
Source code in skforecast\model_selection_multiseries\model_selection_multiseries.py 
2154
2155
2156
2157
2158
2159
2160
2161
2162
2163
2164
2165
2166
2167
2168
2169
2170
2171
2172
2173
2174
2175
2176
2177
2178
2179
2180
2181
2182
2183
2184
2185
2186
2187
2188
2189
2190
2191
2192
2193
2194
2195
2196
2197
2198
2199
2200
2201
2202
2203
2204
2205
2206
2207
2208
2209
2210
2211
2212
2213
2214
2215
2216
2217
2218
2219
2220
2221
2222
2223
2224
2225
2226
2227
2228
2229
2230
2231
2232
2233
2234
2235
2236
2237
2238
2239
2240
2241
2242
2243
2244
2245
2246
2247
2248
2249
2250
2251
2252
2253
2254
2255
2256
2257
2258
2259
2260
2261
2262
2263
2264
2265
2266
2267
2268
2269
2270
2271
2272
2273
2274
2275
2276
2277
2278
2279
2280
2281
2282
2283
2284
2285
2286
2287
2288
2289
2290
2291
2292
2293
2294
2295
2296
2297
2298
2299
2300
2301
2302
2303
2304
2305
2306
2307
2308
2309
2310
2311
2312
2313
2314
2315
2316
2317
2318
2319
2320
2321
2322
2323
2324
2325
2326
2327
2328
2329
2330
2331
2332
2333
2334
2335
2336
2337
2338
2339
2340
2341
2342
2343
2344
2345
2346
2347
2348
def select_features_multiseries(
    forecaster: object,
    selector: object,
    series: Union[pd.DataFrame, dict],
    exog: Optional[Union[pd.Series, pd.DataFrame, dict]] = None,
    select_only: Optional[str] = None,
    force_inclusion: Optional[Union[list, str]] = None,
    subsample: Union[int, float] = 0.5,
    random_state: int = 123,
    verbose: bool = True,
) -> Union[list, list]:
    """
    Feature selection using any of the sklearn.feature_selection module selectors 
    (such as `RFECV`, `SelectFromModel`, etc.). Two groups of features are
    evaluated: autoregressive features and exogenous features. By default, the 
    selection process is performed on both sets of features at the same time, 
    so that the most relevant autoregressive and exogenous features are selected. 
    However, using the `select_only` argument, the selection process can focus 
    only on the autoregressive or exogenous features without taking into account 
    the other features. Therefore, all other features will remain in the model. 
    It is also possible to force the inclusion of certain features in the final 
    list of selected features using the `force_inclusion` parameter.

    Parameters
    ----------
    forecaster : ForecasterAutoregMultiSeries, ForecasterAutoregMultiseriesCustom
        Forecaster model.
    selector : object
        A feature selector from sklearn.feature_selection.
    series : pandas DataFrame
        Target time series to which the feature selection will be applied.
    exog : pandas Series, pandas DataFrame, dict, default `None`
        Exogenous variables.
    select_only : str, default `None`
        Decide what type of features to include in the selection process. 

        - If `'autoreg'`, only autoregressive features (lags or custom 
        predictors) are evaluated by the selector. All exogenous features are 
        included in the output (`selected_exog`).
        - If `'exog'`, only exogenous features are evaluated without the presence
        of autoregressive features. All autoregressive features are included 
        in the output (`selected_autoreg`).
        - If `None`, all features are evaluated by the selector.
    force_inclusion : list, str, default `None`
        Features to force include in the final list of selected features.

        - If `list`, list of feature names to force include.
        - If `str`, regular expression to identify features to force include. 
        For example, if `force_inclusion="^sun_"`, all features that begin 
        with "sun_" will be included in the final list of selected features.
    subsample : int, float, default `0.5`
        Proportion of records to use for feature selection.
    random_state : int, default `123`
        Sets a seed for the random subsample so that the subsampling process 
        is always deterministic.
    verbose : bool, default `True`
        Print information about feature selection process.

    Returns
    -------
    selected_autoreg : list
        List of selected autoregressive features.
    selected_exog : list
        List of selected exogenous features.

    """

    valid_forecasters = [
        'ForecasterAutoregMultiSeries',
        'ForecasterAutoregMultiSeriesCustom',
    ]

    if type(forecaster).__name__ not in valid_forecasters:
        raise TypeError(
            f"`forecaster` must be one of the following classes: {valid_forecasters}."
        )

    if select_only not in ['autoreg', 'exog', None]:
        raise ValueError(
            "`select_only` must be one of the following values: 'autoreg', 'exog', None."
        )

    if subsample <= 0 or subsample > 1:
        raise ValueError(
            "`subsample` must be a number greater than 0 and less than or equal to 1."
        )

    forecaster = deepcopy(forecaster)
    forecaster.fitted = False
    output = forecaster._create_train_X_y(series=series, exog=exog)
    X_train = output[0]
    y_train = output[1]
    series_col_names = output[3]

    if forecaster.encoding == 'onehot':
        encoding_cols = series_col_names
    else:
        encoding_cols = ['_level_skforecast']

    if hasattr(forecaster, 'lags'):
        autoreg_cols = [f"lag_{lag}" for lag in forecaster.lags]
    else:
        if forecaster.name_predictors is not None:
            autoreg_cols = forecaster.name_predictors
        else:
            autoreg_cols = [
                col
                for col in X_train.columns
                if re.match(r'^custom_predictor_\d+', col)
            ]
    exog_cols = [
        col
        for col in X_train.columns
        if col not in autoreg_cols and col not in encoding_cols
    ]

    forced_autoreg = []
    forced_exog = []
    if force_inclusion is not None:
        if isinstance(force_inclusion, list):
            forced_autoreg = [col for col in force_inclusion if col in autoreg_cols]
            forced_exog = [col for col in force_inclusion if col in exog_cols]
        elif isinstance(force_inclusion, str):
            forced_autoreg = [col for col in autoreg_cols if re.match(force_inclusion, col)]
            forced_exog = [col for col in exog_cols if re.match(force_inclusion, col)]

    if select_only == 'autoreg':
        X_train = X_train.drop(columns=exog_cols + encoding_cols)
    elif select_only == 'exog':
        X_train = X_train.drop(columns=autoreg_cols + encoding_cols)
    else:
        X_train = X_train.drop(columns=encoding_cols)

    if isinstance(subsample, float):
        subsample = int(len(X_train) * subsample)

    rng = np.random.default_rng(seed=random_state)
    sample = rng.choice(X_train.index, size=subsample, replace=False)
    X_train_sample = X_train.loc[sample, :]
    y_train_sample = y_train.loc[sample]
    selector.fit(X_train_sample, y_train_sample)
    selected_features = selector.get_feature_names_out()

    if select_only == 'exog':
        selected_autoreg = autoreg_cols
    else:
        selected_autoreg = [
            feature
            for feature in selected_features
            if feature in autoreg_cols
        ]

    if select_only == 'autoreg':
        selected_exog = exog_cols
    else:
        selected_exog = [
            feature
            for feature in selected_features
            if feature in exog_cols
        ]

    if force_inclusion is not None: 
        if select_only != 'autoreg':
            forced_exog_not_selected = set(forced_exog) - set(selected_features)
            selected_exog.extend(forced_exog_not_selected)
            selected_exog.sort(key=exog_cols.index)
        if select_only != 'exog':
            forced_autoreg_not_selected = set(forced_autoreg) - set(selected_features)
            selected_autoreg.extend(forced_autoreg_not_selected)
            selected_autoreg.sort(key=autoreg_cols.index)

    if len(selected_autoreg) == 0:
        warnings.warn(
            ("No autoregressive features have been selected. Since a Forecaster "
             "cannot be created without them, be sure to include at least one "
             "using the `force_inclusion` parameter.")
        )
    else:
        if hasattr(forecaster, 'lags'):
            selected_autoreg = [int(feature.replace('lag_', '')) 
                                for feature in selected_autoreg] 

    if verbose:
        print(f"Recursive feature elimination ({selector.__class__.__name__})")
        print("--------------------------------" + "-" * len(selector.__class__.__name__))
        print(f"Total number of records available: {X_train.shape[0]}")
        print(f"Total number of records used for feature selection: {X_train_sample.shape[0]}")
        print(f"Number of features available: {len(autoreg_cols) + len(exog_cols)}") 
        print(f"    Autoreg (n={len(autoreg_cols)})")
        print(f"    Exog    (n={len(exog_cols)})")
        print(f"Number of features selected: {len(selected_features)}")
        print(f"    Autoreg (n={len(selected_autoreg)}) : {selected_autoreg}")
        print(f"    Exog    (n={len(selected_exog)}) : {selected_exog}")

    return selected_autoreg, selected_exog

backtesting_forecaster_multivariate(forecaster, series, steps, metric, initial_train_size, add_aggregated_metric=True, fixed_train_size=True, gap=0, skip_folds=None, allow_incomplete_fold=True, levels=None, exog=None, refit=False, interval=None, n_boot=500, random_state=123, in_sample_residuals=True, n_jobs='auto', verbose=False, show_progress=True, suppress_warnings=False)

This function is an alias of backtesting_forecaster_multiseries.

Backtesting for multi-series and multivariate forecasters.

If refit is False, the model is trained only once using the initial_train_size first observations. If refit is True, the model is trained in each iteration increasing the training set. A copy of the original forecaster is created so it is not modified during the process.

Parameters:

Name Type Description Default
forecaster (ForecasterAutoregMultiSeries, ForecasterAutoregMultiSeriesCustom, ForecasterAutoregMultiVariate)

Forecaster model.

 required
series pandas DataFrame, dict

Training time series.

 required
steps int

Number of steps to predict.

 required
metric (str, Callable, list)

Metric used to quantify the goodness of fit of the model.

  • If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
  • If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float.
  • If list: List containing multiple strings and/or Callables.
 required
initial_train_size int

Number of samples in the initial train split. If None and forecaster is already trained, no initial train is done and all data is used to evaluate the model. However, the first len(forecaster.last_window) observations are needed to create the initial predictors, so no predictions are calculated for them. This useful to backtest the model on the same data used to train it. None is only allowed when refit is False and forecaster is already trained.

 `None`
add_aggregated_metric bool

If True, the metric is calculated for each level and an aggregated metric is calculated using the aggregate_metric method.

  • 'average': the average (arithmetic mean) of all levels.
  • 'weighted_average': the average of the metrics weighted by the number of predicted values of each level.
  • 'pooling': the values of all levels are pooled and then the metric is calculated.
 `True`
fixed_train_size bool

If True, train size doesn't increase but moves by steps in each iteration.

 `True`
gap int

Number of samples to be excluded after the end of each training set and before the test set.

 `0`
skip_folds (int, list)

If skip_folds is an integer, every 'skip_folds'-th is returned. If skip_folds is a list, the folds in the list are skipped. For example, if skip_folds = 3, and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If skip_folds is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].

 `None`
allow_incomplete_fold bool

Last fold is allowed to have a smaller number of samples than the test_size. If False, the last fold is excluded.

 `True`
levels (str, list)

Time series to be predicted. If None all levels will be predicted.

 `None`
exog pandas Series, pandas DataFrame, dict

Exogenous variables.

 `None`
refit (bool, int)

Whether to re-fit the forecaster in each iteration. If refit is an integer, the Forecaster will be trained every that number of iterations.

 `False`
interval list

Confidence of the prediction interval estimated. Sequence of percentiles to compute, which must be between 0 and 100 inclusive. If None, no intervals are estimated.

 `None`
n_boot int

Number of bootstrapping iterations used to estimate prediction intervals.

 `500`
random_state int

Sets a seed to the random generator, so that boot intervals are always deterministic.

 `123`
in_sample_residuals bool

If True, residuals from the training data are used as proxy of prediction error to create prediction intervals. If False, out_sample_residuals are used if they are already stored inside the forecaster.

 `True`
n_jobs (int, auto)

The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting. New in version 0.9.0

 `'auto'`
verbose bool

Print number of folds and index of training and validation sets used for backtesting.

 `False`
show_progress bool

Whether to show a progress bar.

 `True`
suppress_warnings bool

If True, skforecast warnings will be suppressed during the backtesting process. See skforecast.exceptions.warn_skforecast_categories for more information.

 False

Returns:

Name Type Description
metrics_levels pandas DataFrame

Value(s) of the metric(s). Index are the levels and columns the metrics.
backtest_predictions pandas DataFrame

Value of predictions and their estimated interval if interval is not None. If there is more than one level, this structure will be repeated for each of them.

  • column pred: predictions.
  • column lower_bound: lower bound of the interval.
  • column upper_bound: upper bound of the interval.
Source code in skforecast\model_selection_multiseries\model_selection_multiseries.py 
2353
2354
2355
2356
2357
2358
2359
2360
2361
2362
2363
2364
2365
2366
2367
2368
2369
2370
2371
2372
2373
2374
2375
2376
2377
2378
2379
2380
2381
2382
2383
2384
2385
2386
2387
2388
2389
2390
2391
2392
2393
2394
2395
2396
2397
2398
2399
2400
2401
2402
2403
2404
2405
2406
2407
2408
2409
2410
2411
2412
2413
2414
2415
2416
2417
2418
2419
2420
2421
2422
2423
2424
2425
2426
2427
2428
2429
2430
2431
2432
2433
2434
2435
2436
2437
2438
2439
2440
2441
2442
2443
2444
2445
2446
2447
2448
2449
2450
2451
2452
2453
2454
2455
2456
2457
2458
2459
2460
2461
2462
2463
2464
2465
2466
2467
2468
2469
2470
2471
2472
2473
2474
2475
2476
2477
2478
2479
2480
2481
2482
2483
2484
2485
2486
2487
2488
2489
2490
2491
2492
2493
2494
2495
2496
2497
2498
2499
2500
2501
2502
2503
2504
2505
2506
2507
def backtesting_forecaster_multivariate(
    forecaster: object,
    series: Union[pd.DataFrame, dict],
    steps: int,
    metric: Union[str, Callable, list],
    initial_train_size: Optional[int],
    add_aggregated_metric: bool = True,
    fixed_train_size: bool = True,
    gap: int = 0,
    skip_folds: Optional[Union[int, list]] = None,
    allow_incomplete_fold: bool = True,
    levels: Optional[Union[str, list]] = None,
    exog: Optional[Union[pd.Series, pd.DataFrame, dict]] = None,
    refit: Union[bool, int] = False,
    interval: Optional[list] = None,
    n_boot: int = 500,
    random_state: int = 123,
    in_sample_residuals: bool = True,
    n_jobs: Union[int, str] = 'auto',
    verbose: bool = False,
    show_progress: bool = True,
    suppress_warnings: bool = False
) -> Tuple[pd.DataFrame, pd.DataFrame]:
    """
    This function is an alias of backtesting_forecaster_multiseries.

    Backtesting for multi-series and multivariate forecasters.

    If `refit` is False, the model is trained only once using the `initial_train_size`
    first observations. If `refit` is True, the model is trained in each iteration
    increasing the training set. A copy of the original forecaster is created so 
    it is not modified during the process.

    Parameters
    ----------
    forecaster : ForecasterAutoregMultiSeries, ForecasterAutoregMultiSeriesCustom, ForecasterAutoregMultiVariate
        Forecaster model.
    series : pandas DataFrame, dict
        Training time series.
    steps : int
        Number of steps to predict.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    initial_train_size : int, default `None`
        Number of samples in the initial train split. If `None` and `forecaster` is 
        already trained, no initial train is done and all data is used to evaluate the 
        model. However, the first `len(forecaster.last_window)` observations are needed 
        to create the initial predictors, so no predictions are calculated for them. 
        This useful to backtest the model on the same data used to train it.
        `None` is only allowed when `refit` is `False` and `forecaster` is already
        trained.
    add_aggregated_metric : bool, default `True`
        If `True`, the metric is calculated for each level and an aggregated metric
        is calculated using the `aggregate_metric` method.

        - 'average': the average (arithmetic mean) of all levels.
        - 'weighted_average': the average of the metrics weighted by the number of
        predicted values of each level.
        - 'pooling': the values of all levels are pooled and then the metric is
        calculated.
    fixed_train_size : bool, default `True`
        If True, train size doesn't increase but moves by `steps` in each iteration.
    gap : int, default `0`
        Number of samples to be excluded after the end of each training set and 
        before the test set.
    skip_folds : int, list, default `None`
        If `skip_folds` is an integer, every 'skip_folds'-th is returned. If `skip_folds`
        is a list, the folds in the list are skipped. For example, if `skip_folds = 3`,
        and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If `skip_folds`
        is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].
    allow_incomplete_fold : bool, default `True`
        Last fold is allowed to have a smaller number of samples than the 
        `test_size`. If `False`, the last fold is excluded.
    levels : str, list, default `None`
        Time series to be predicted. If `None` all levels will be predicted.
    exog : pandas Series, pandas DataFrame, dict, default `None`
        Exogenous variables.
    refit : bool, int, default `False`
        Whether to re-fit the forecaster in each iteration. If `refit` is an 
        integer, the Forecaster will be trained every that number of iterations.
    interval : list, default `None`
        Confidence of the prediction interval estimated. Sequence of percentiles
        to compute, which must be between 0 and 100 inclusive. If `None`, no
        intervals are estimated.
    n_boot : int, default `500`
        Number of bootstrapping iterations used to estimate prediction
        intervals.
    random_state : int, default `123`
        Sets a seed to the random generator, so that boot intervals are always 
        deterministic.
    in_sample_residuals : bool, default `True`
        If `True`, residuals from the training data are used as proxy of prediction 
        error to create prediction intervals.  If `False`, out_sample_residuals 
        are used if they are already stored inside the forecaster.
    n_jobs : int, 'auto', default `'auto'`
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting.
        **New in version 0.9.0** 
    verbose : bool, default `False`
        Print number of folds and index of training and validation sets used 
        for backtesting.
    show_progress : bool, default `True`
        Whether to show a progress bar.
    suppress_warnings: bool, default `False`
        If `True`, skforecast warnings will be suppressed during the backtesting 
        process. See skforecast.exceptions.warn_skforecast_categories for more
        information.

    Returns
    -------
    metrics_levels : pandas DataFrame
        Value(s) of the metric(s). Index are the levels and columns the metrics.
    backtest_predictions : pandas DataFrame
        Value of predictions and their estimated interval if `interval` is not `None`.
        If there is more than one level, this structure will be repeated for each of them.

        - column pred: predictions.
        - column lower_bound: lower bound of the interval.
        - column upper_bound: upper bound of the interval.

    """

    metrics_levels, backtest_predictions = backtesting_forecaster_multiseries(
        forecaster            = forecaster,
        series                = series,
        steps                 = steps,
        metric                = metric,
        add_aggregated_metric = add_aggregated_metric,
        initial_train_size    = initial_train_size,
        fixed_train_size      = fixed_train_size,
        gap                   = gap,
        skip_folds            = skip_folds,
        allow_incomplete_fold = allow_incomplete_fold,
        levels                = levels,
        exog                  = exog,
        refit                 = refit,
        interval              = interval,
        n_boot                = n_boot,
        random_state          = random_state,
        in_sample_residuals   = in_sample_residuals,
        n_jobs                = n_jobs,
        verbose               = verbose,
        show_progress         = show_progress,
        suppress_warnings     = suppress_warnings
    )

    return metrics_levels, backtest_predictions

grid_search_forecaster_multivariate(forecaster, series, param_grid, steps, metric, initial_train_size, aggregate_metric=['weighted_average', 'average', 'pooling'], fixed_train_size=True, gap=0, skip_folds=None, allow_incomplete_fold=True, levels=None, exog=None, lags_grid=None, refit=False, return_best=True, n_jobs='auto', verbose=True, show_progress=True, suppress_warnings=False, output_file=None)

This function is an alias of grid_search_forecaster_multiseries.

Exhaustive search over specified parameter values for a Forecaster object. Validation is done using multi-series backtesting.

Parameters:

Name Type Description Default
forecaster (ForecasterAutoregMultiSeries, ForecasterAutoregMultiSeriesCustom, ForecasterAutoregMultiVariate)

Forecaster model.

 required
series pandas DataFrame, dict

Training time series.

 required
param_grid dict

Dictionary with parameters names (str) as keys and lists of parameter settings to try as values.

 required
steps int

Number of steps to predict.

 required
metric (str, Callable, list)

Metric used to quantify the goodness of fit of the model.

  • If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
  • If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float.
  • If list: List containing multiple strings and/or Callables.
 required
initial_train_size int

Number of samples in the initial train split.

 required
aggregate_metric (str, list)

Aggregation method/s used to combine the metric/s of all levels (series) when multiple levels are predicted. If list, the first aggregation method is used to select the best parameters.

  • 'average': the average (arithmetic mean) of all levels.
  • 'weighted_average': the average of the metrics weighted by the number of predicted values of each level.
  • 'pooling': the values of all levels are pooled and then the metric is calculated.
 `['weighted_average', 'average', 'pooling']`
fixed_train_size bool

If True, train size doesn't increase but moves by steps in each iteration.

 `True`
gap int

Number of samples to be excluded after the end of each training set and before the test set.

 `0`
skip_folds (int, list)

If skip_folds is an integer, every 'skip_folds'-th is returned. If skip_folds is a list, the folds in the list are skipped. For example, if skip_folds = 3, and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If skip_folds is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].

 `None`
allow_incomplete_fold bool

Last fold is allowed to have a smaller number of samples than the test_size. If False, the last fold is excluded.

 `True`
levels (str, list)

level (str) or levels (list) at which the forecaster is optimized. If None, all levels are taken into account. The resulting metric will be the average of the optimization of all levels.

 `None`
exog pandas Series, pandas DataFrame, dict

Exogenous variables.

 `None`
lags_grid (list, dict)

Lists of lags to try, containing int, lists, numpy ndarray, or range objects. If dict, the keys are used as labels in the results DataFrame, and the values are used as the lists of lags to try. Ignored if the forecaster is an instance of ForecasterAutoregCustom or ForecasterAutoregMultiSeriesCustom.

 `None`
refit (bool, int)

Whether to re-fit the forecaster in each iteration. If refit is an integer, the Forecaster will be trained every that number of iterations.

 `False`
return_best bool

Refit the forecaster using the best found parameters on the whole data.

 `True`
n_jobs (int, auto)

The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting. New in version 0.9.0

 `'auto'`
verbose bool

Print number of folds used for cv or backtesting.

 `True`
show_progress bool

Whether to show a progress bar.

 `True`
suppress_warnings bool

If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information.

 False
output_file str

Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file. New in version 0.12.0

 `None`

Returns:

Name Type Description
results pandas DataFrame

Results for each combination of parameters.

  • column levels: levels configuration for each iteration.
  • column lags: lags configuration for each iteration.
  • column lags_label: descriptive label or alias for the lags.
  • column params: parameters configuration for each iteration.
  • column metric: metric value estimated for each iteration. The resulting metric will be the average of the optimization of all levels.
  • additional n columns with param = value.
Source code in skforecast\model_selection_multiseries\model_selection_multiseries.py 
2510
2511
2512
2513
2514
2515
2516
2517
2518
2519
2520
2521
2522
2523
2524
2525
2526
2527
2528
2529
2530
2531
2532
2533
2534
2535
2536
2537
2538
2539
2540
2541
2542
2543
2544
2545
2546
2547
2548
2549
2550
2551
2552
2553
2554
2555
2556
2557
2558
2559
2560
2561
2562
2563
2564
2565
2566
2567
2568
2569
2570
2571
2572
2573
2574
2575
2576
2577
2578
2579
2580
2581
2582
2583
2584
2585
2586
2587
2588
2589
2590
2591
2592
2593
2594
2595
2596
2597
2598
2599
2600
2601
2602
2603
2604
2605
2606
2607
2608
2609
2610
2611
2612
2613
2614
2615
2616
2617
2618
2619
2620
2621
2622
2623
2624
2625
2626
2627
2628
2629
2630
2631
2632
2633
2634
2635
2636
2637
2638
2639
2640
2641
2642
2643
2644
2645
2646
2647
2648
2649
2650
2651
2652
2653
2654
2655
2656
2657
2658
2659
def grid_search_forecaster_multivariate(
    forecaster: object,
    series: Union[pd.DataFrame, dict],
    param_grid: dict,
    steps: int,
    metric: Union[str, Callable, list],
    initial_train_size: int,
    aggregate_metric: Union[str, list] = ['weighted_average', 'average', 'pooling'],
    fixed_train_size: bool = True,
    gap: int = 0,
    skip_folds: Optional[Union[int, list]] = None,
    allow_incomplete_fold: bool = True,
    levels: Optional[Union[str, list]] = None,
    exog: Optional[Union[pd.Series, pd.DataFrame, dict]] = None,
    lags_grid: Optional[Union[list, dict]] = None,
    refit: Union[bool, int] = False,
    return_best: bool = True,
    n_jobs: Union[int, str] = 'auto',
    verbose: bool = True,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: Optional[str] = None
) -> pd.DataFrame:
    """
    This function is an alias of grid_search_forecaster_multiseries.

    Exhaustive search over specified parameter values for a Forecaster object.
    Validation is done using multi-series backtesting.

    Parameters
    ----------
    forecaster : ForecasterAutoregMultiSeries, ForecasterAutoregMultiSeriesCustom, ForecasterAutoregMultiVariate
        Forecaster model.
    series : pandas DataFrame, dict
        Training time series.
    param_grid : dict
        Dictionary with parameters names (`str`) as keys and lists of parameter
        settings to try as values.
    steps : int
        Number of steps to predict.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    initial_train_size : int 
        Number of samples in the initial train split.
    aggregate_metric : str, list, default `['weighted_average', 'average', 'pooling']`
        Aggregation method/s used to combine the metric/s of all levels (series)
        when multiple levels are predicted. If list, the first aggregation method
        is used to select the best parameters.

        - 'average': the average (arithmetic mean) of all levels.
        - 'weighted_average': the average of the metrics weighted by the number of
        predicted values of each level.
        - 'pooling': the values of all levels are pooled and then the metric is
        calculated.
    fixed_train_size : bool, default `True`
        If True, train size doesn't increase but moves by `steps` in each iteration.
    gap : int, default `0`
        Number of samples to be excluded after the end of each training set and 
        before the test set.
    skip_folds : int, list, default `None`
        If `skip_folds` is an integer, every 'skip_folds'-th is returned. If `skip_folds`
        is a list, the folds in the list are skipped. For example, if `skip_folds = 3`,
        and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If `skip_folds`
        is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].
    allow_incomplete_fold : bool, default `True`
        Last fold is allowed to have a smaller number of samples than the 
        `test_size`. If `False`, the last fold is excluded.
    levels : str, list, default `None`
        level (`str`) or levels (`list`) at which the forecaster is optimized. 
        If `None`, all levels are taken into account. The resulting metric will be
        the average of the optimization of all levels.
    exog : pandas Series, pandas DataFrame, dict, default `None`
        Exogenous variables.
    lags_grid : list, dict, default `None`
        Lists of lags to try, containing int, lists, numpy ndarray, or range 
        objects. If `dict`, the keys are used as labels in the `results` 
        DataFrame, and the values are used as the lists of lags to try. Ignored 
        if the forecaster is an instance of `ForecasterAutoregCustom` or 
        `ForecasterAutoregMultiSeriesCustom`.
    refit : bool, int, default `False`
        Whether to re-fit the forecaster in each iteration. If `refit` is an 
        integer, the Forecaster will be trained every that number of iterations.
    return_best : bool, default `True`
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default `'auto'`
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting.
        **New in version 0.9.0**
    verbose : bool, default `True`
        Print number of folds used for cv or backtesting.
    show_progress : bool, default `True`
        Whether to show a progress bar.
    suppress_warnings: bool, default `False`
        If `True`, skforecast warnings will be suppressed during the hyperparameter 
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.
    output_file : str, default `None`
        Specifies the filename or full path where the results should be saved. 
        The results will be saved in a tab-separated values (TSV) format. If 
        `None`, the results will not be saved to a file.
        **New in version 0.12.0**

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column levels: levels configuration for each iteration.
        - column lags: lags configuration for each iteration.
        - column lags_label: descriptive label or alias for the lags.
        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration. The resulting 
        metric will be the average of the optimization of all levels.
        - additional n columns with param = value.

    """

    results = grid_search_forecaster_multiseries(
        forecaster            = forecaster,
        series                = series,
        param_grid            = param_grid,
        steps                 = steps,
        metric                = metric,
        aggregate_metric      = aggregate_metric,
        initial_train_size    = initial_train_size,
        fixed_train_size      = fixed_train_size,
        gap                   = gap,
        skip_folds            = skip_folds,
        allow_incomplete_fold = allow_incomplete_fold,
        levels                = levels,
        exog                  = exog,
        lags_grid             = lags_grid,
        refit                 = refit,
        return_best           = return_best,
        n_jobs                = n_jobs,
        verbose               = verbose,
        show_progress         = show_progress,
        suppress_warnings     = suppress_warnings,
        output_file           = output_file
    )

    return results

random_search_forecaster_multivariate(forecaster, series, param_distributions, steps, metric, initial_train_size, aggregate_metric=['weighted_average', 'average', 'pooling'], fixed_train_size=True, gap=0, skip_folds=None, allow_incomplete_fold=True, levels=None, exog=None, lags_grid=None, refit=False, n_iter=10, random_state=123, return_best=True, n_jobs='auto', verbose=True, show_progress=True, suppress_warnings=False, output_file=None)

This function is an alias of random_search_forecaster_multiseries.

Random search over specified parameter values or distributions for a Forecaster object. Validation is done using multi-series backtesting.

Parameters:

Name Type Description Default
forecaster (ForecasterAutoregMultiSeries, ForecasterAutoregMultiSeriesCustom, ForecasterAutoregMultiVariate)

Forecaster model.

 required
series pandas DataFrame, dict

Training time series.

 required
param_distributions dict

Dictionary with parameters names (str) as keys and distributions or lists of parameters to try.

 required
steps int

Number of steps to predict.

 required
metric (str, Callable, list)

Metric used to quantify the goodness of fit of the model.

  • If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
  • If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float.
  • If list: List containing multiple strings and/or Callables.
 required
initial_train_size int

Number of samples in the initial train split.

 required
aggregate_metric (str, list)

Aggregation method/s used to combine the metric/s of all levels (series) when multiple levels are predicted. If list, the first aggregation method is used to select the best parameters.

  • 'average': the average (arithmetic mean) of all levels.
  • 'weighted_average': the average of the metrics weighted by the number of predicted values of each level.
  • 'pooling': the values of all levels are pooled and then the metric is calculated.
 `['weighted_average', 'average', 'pooling']`
fixed_train_size bool

If True, train size doesn't increase but moves by steps in each iteration.

 `True`
gap int

Number of samples to be excluded after the end of each training set and before the test set.

 `0`
skip_folds (int, list)

If skip_folds is an integer, every 'skip_folds'-th is returned. If skip_folds is a list, the folds in the list are skipped. For example, if skip_folds = 3, and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If skip_folds is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].

 `None`
allow_incomplete_fold bool

Last fold is allowed to have a smaller number of samples than the test_size. If False, the last fold is excluded.

 `True`
levels (str, list)

level (str) or levels (list) at which the forecaster is optimized. If None, all levels are taken into account. The resulting metric will be the average of the optimization of all levels.

 `None`
exog pandas Series, pandas DataFrame, dict

Exogenous variables.

 `None`
lags_grid (list, dict)

Lists of lags to try, containing int, lists, numpy ndarray, or range objects. If dict, the keys are used as labels in the results DataFrame, and the values are used as the lists of lags to try. Ignored if the forecaster is an instance of ForecasterAutoregCustom or ForecasterAutoregMultiSeriesCustom.

 `None`
refit (bool, int)

Whether to re-fit the forecaster in each iteration. If refit is an integer, the Forecaster will be trained every that number of iterations.

 `False`
n_iter int

Number of parameter settings that are sampled per lags configuration. n_iter trades off runtime vs quality of the solution.

 `10`
random_state int

Sets a seed to the random sampling for reproducible output.

 `123`
return_best bool

Refit the forecaster using the best found parameters on the whole data.

 `True`
n_jobs (int, auto)

The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting. New in version 0.9.0

 `'auto'`
verbose bool

Print number of folds used for cv or backtesting.

 `True`
show_progress bool

Whether to show a progress bar.

 `True`
suppress_warnings bool

If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information.

 False
output_file str

Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file. New in version 0.12.0

 `None`

Returns:

Name Type Description
results pandas DataFrame

Results for each combination of parameters.

  • column levels: levels configuration for each iteration.
  • column lags: lags configuration for each iteration.
  • column lags_label: descriptive label or alias for the lags.
  • column params: parameters configuration for each iteration.
  • column metric: metric value estimated for each iteration. The resulting metric will be the average of the optimization of all levels.
  • additional n columns with param = value.
Source code in skforecast\model_selection_multiseries\model_selection_multiseries.py 
2662
2663
2664
2665
2666
2667
2668
2669
2670
2671
2672
2673
2674
2675
2676
2677
2678
2679
2680
2681
2682
2683
2684
2685
2686
2687
2688
2689
2690
2691
2692
2693
2694
2695
2696
2697
2698
2699
2700
2701
2702
2703
2704
2705
2706
2707
2708
2709
2710
2711
2712
2713
2714
2715
2716
2717
2718
2719
2720
2721
2722
2723
2724
2725
2726
2727
2728
2729
2730
2731
2732
2733
2734
2735
2736
2737
2738
2739
2740
2741
2742
2743
2744
2745
2746
2747
2748
2749
2750
2751
2752
2753
2754
2755
2756
2757
2758
2759
2760
2761
2762
2763
2764
2765
2766
2767
2768
2769
2770
2771
2772
2773
2774
2775
2776
2777
2778
2779
2780
2781
2782
2783
2784
2785
2786
2787
2788
2789
2790
2791
2792
2793
2794
2795
2796
2797
2798
2799
2800
2801
2802
2803
2804
2805
2806
2807
2808
2809
2810
2811
2812
2813
2814
2815
2816
2817
2818
2819
2820
def random_search_forecaster_multivariate(
    forecaster: object,
    series: Union[pd.DataFrame, dict],
    param_distributions: dict,
    steps: int,
    metric: Union[str, Callable, list],
    initial_train_size: int,
    aggregate_metric: Union[str, list] = ['weighted_average', 'average', 'pooling'],
    fixed_train_size: bool = True,
    gap: int = 0,
    skip_folds: Optional[Union[int, list]] = None,
    allow_incomplete_fold: bool = True,
    levels: Optional[Union[str, list]] = None,
    exog: Optional[Union[pd.Series, pd.DataFrame, dict]] = None,
    lags_grid: Optional[Union[list, dict]] = None,
    refit: Union[bool, int] = False,
    n_iter: int = 10,
    random_state: int = 123,
    return_best: bool = True,
    n_jobs: Union[int, str] = 'auto',
    verbose: bool = True,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: Optional[str] = None
) -> pd.DataFrame:
    """
    This function is an alias of random_search_forecaster_multiseries.

    Random search over specified parameter values or distributions for a Forecaster 
    object. Validation is done using multi-series backtesting.

    Parameters
    ----------
    forecaster : ForecasterAutoregMultiSeries, ForecasterAutoregMultiSeriesCustom, ForecasterAutoregMultiVariate
        Forecaster model.
    series : pandas DataFrame, dict
        Training time series.
    param_distributions : dict
        Dictionary with parameters names (`str`) as keys and distributions or 
        lists of parameters to try.
    steps : int
        Number of steps to predict.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    initial_train_size : int 
        Number of samples in the initial train split.
    aggregate_metric : str, list, default `['weighted_average', 'average', 'pooling']`
        Aggregation method/s used to combine the metric/s of all levels (series)
        when multiple levels are predicted. If list, the first aggregation method
        is used to select the best parameters.

        - 'average': the average (arithmetic mean) of all levels.
        - 'weighted_average': the average of the metrics weighted by the number of
        predicted values of each level.
        - 'pooling': the values of all levels are pooled and then the metric is
        calculated.
    fixed_train_size : bool, default `True`
        If True, train size doesn't increase but moves by `steps` in each iteration.
    gap : int, default `0`
        Number of samples to be excluded after the end of each training set and 
        before the test set.
    skip_folds : int, list, default `None`
        If `skip_folds` is an integer, every 'skip_folds'-th is returned. If `skip_folds`
        is a list, the folds in the list are skipped. For example, if `skip_folds = 3`,
        and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If `skip_folds`
        is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].
    allow_incomplete_fold : bool, default `True`
        Last fold is allowed to have a smaller number of samples than the 
        `test_size`. If `False`, the last fold is excluded.
    levels : str, list, default `None`
        level (`str`) or levels (`list`) at which the forecaster is optimized. 
        If `None`, all levels are taken into account. The resulting metric will be
        the average of the optimization of all levels.
    exog : pandas Series, pandas DataFrame, dict, default `None`
        Exogenous variables.
    lags_grid : list, dict, default `None`
        Lists of lags to try, containing int, lists, numpy ndarray, or range 
        objects. If `dict`, the keys are used as labels in the `results` 
        DataFrame, and the values are used as the lists of lags to try. Ignored 
        if the forecaster is an instance of `ForecasterAutoregCustom` or 
        `ForecasterAutoregMultiSeriesCustom`.
    refit : bool, int, default `False`
        Whether to re-fit the forecaster in each iteration. If `refit` is an 
        integer, the Forecaster will be trained every that number of iterations.
    n_iter : int, default `10`
        Number of parameter settings that are sampled per lags configuration. 
        n_iter trades off runtime vs quality of the solution.
    random_state : int, default `123`
        Sets a seed to the random sampling for reproducible output.
    return_best : bool, default `True`
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default `'auto'`
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting.
        **New in version 0.9.0**
    verbose : bool, default `True`
        Print number of folds used for cv or backtesting.
    show_progress : bool, default `True`
        Whether to show a progress bar.
    suppress_warnings: bool, default `False`
        If `True`, skforecast warnings will be suppressed during the hyperparameter 
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.
    output_file : str, default `None`
        Specifies the filename or full path where the results should be saved. 
        The results will be saved in a tab-separated values (TSV) format. If 
        `None`, the results will not be saved to a file.
        **New in version 0.12.0**

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column levels: levels configuration for each iteration.
        - column lags: lags configuration for each iteration.
        - column lags_label: descriptive label or alias for the lags.
        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration. The resulting 
        metric will be the average of the optimization of all levels.
        - additional n columns with param = value.

    """

    results = random_search_forecaster_multiseries(
        forecaster            = forecaster,
        series                = series,
        param_distributions   = param_distributions,
        steps                 = steps,
        metric                = metric,
        aggregate_metric      = aggregate_metric,
        initial_train_size    = initial_train_size,
        fixed_train_size      = fixed_train_size,
        gap                   = gap,
        skip_folds            = skip_folds,
        allow_incomplete_fold = allow_incomplete_fold,
        levels                = levels,
        exog                  = exog,
        lags_grid             = lags_grid,
        refit                 = refit,
        n_iter                = n_iter,
        random_state          = random_state,
        return_best           = return_best,
        n_jobs                = n_jobs,
        verbose               = verbose,
        show_progress         = show_progress,
        suppress_warnings     = suppress_warnings,
        output_file           = output_file
    ) 

    return results

bayesian_search_forecaster_multivariate(forecaster, series, search_space, steps, metric, initial_train_size, aggregate_metric=['weighted_average', 'average', 'pooling'], fixed_train_size=True, gap=0, skip_folds=None, allow_incomplete_fold=True, levels=None, exog=None, refit=False, n_trials=10, random_state=123, return_best=True, n_jobs='auto', verbose=True, show_progress=True, suppress_warnings=False, output_file=None, engine='optuna', kwargs_create_study={}, kwargs_study_optimize={})

This function is an alias of bayesian_search_forecaster_multiseries.

Bayesian optimization for a Forecaster object using multi-series backtesting and optuna library. New in version 0.12.0

Parameters:

Name Type Description Default
forecaster (ForecasterAutoregMultiSeries, ForecasterAutoregMultiSeriesCustom, ForecasterAutoregMultiVariate)

Forecaster model.

 required
series pandas DataFrame, dict

Training time series.

 required
search_space Callable

Function with argument trial which returns a dictionary with parameters names (str) as keys and Trial object from optuna (trial.suggest_float, trial.suggest_int, trial.suggest_categorical) as values.

 required
steps int

Number of steps to predict.

 required
metric (str, Callable, list)

Metric used to quantify the goodness of fit of the model.

  • If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
  • If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float.
  • If list: List containing multiple strings and/or Callables.
 required
initial_train_size int

Number of samples in the initial train split.

 required
aggregate_metric (str, list)

Aggregation method/s used to combine the metric/s of all levels (series) when multiple levels are predicted. If list, the first aggregation method is used to select the best parameters.

  • 'average': the average (arithmetic mean) of all levels.
  • 'weighted_average': the average of the metrics weighted by the number of predicted values of each level.
  • 'pooling': the values of all levels are pooled and then the metric is calculated.
 `['weighted_average', 'average', 'pooling']`
fixed_train_size bool

If True, train size doesn't increase but moves by steps in each iteration.

 `True`
gap int

Number of samples to be excluded after the end of each training set and before the test set.

 `0`
skip_folds (int, list)

If skip_folds is an integer, every 'skip_folds'-th is returned. If skip_folds is a list, the folds in the list are skipped. For example, if skip_folds = 3, and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If skip_folds is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].

 `None`
allow_incomplete_fold bool

Last fold is allowed to have a smaller number of samples than the test_size. If False, the last fold is excluded.

 `True`
levels (str, list)

level (str) or levels (list) at which the forecaster is optimized. If None, all levels are taken into account. The resulting metric will be the average of the optimization of all levels.

 `None`
exog pandas Series, pandas DataFrame, dict

Exogenous variables.

 `None`
refit (bool, int)

Whether to re-fit the forecaster in each iteration. If refit is an integer, the Forecaster will be trained every that number of iterations.

 `False`
n_trials int

Number of parameter settings that are sampled in each lag configuration.

 `10`
random_state int

Sets a seed to the sampling for reproducible output.

 `123`
return_best bool

Refit the forecaster using the best found parameters on the whole data.

 `True`
n_jobs (int, auto)

The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting. New in version 0.9.0

 `'auto'`
verbose bool

Print number of folds used for cv or backtesting.

 `True`
show_progress bool

Whether to show a progress bar.

 `True`
suppress_warnings bool

If True, skforecast warnings will be suppressed during the hyperparameter search. See skforecast.exceptions.warn_skforecast_categories for more information.

 False
output_file str

Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file. New in version 0.12.0

 `None`
engine str

Bayesian optimization runs through the optuna library.

 `'optuna'`
kwargs_create_study dict

Keyword arguments (key, value mappings) to pass to optuna.create_study(). If default, the direction is set to 'minimize' and a TPESampler(seed=123) sampler is used during optimization.

 `{}`
kwargs_study_optimize dict

Other keyword arguments (key, value mappings) to pass to study.optimize().

 `{}`

Returns:

Name Type Description
results pandas DataFrame

Results for each combination of parameters.

  • column levels: levels configuration for each iteration.
  • column lags: lags configuration for each iteration.
  • column params: parameters configuration for each iteration.
  • column metric: metric value estimated for each iteration. The resulting metric will be the average of the optimization of all levels.
  • additional n columns with param = value.
best_trial optuna object

The best optimization result returned as a FrozenTrial optuna object.
Source code in skforecast\model_selection_multiseries\model_selection_multiseries.py 
2823
2824
2825
2826
2827
2828
2829
2830
2831
2832
2833
2834
2835
2836
2837
2838
2839
2840
2841
2842
2843
2844
2845
2846
2847
2848
2849
2850
2851
2852
2853
2854
2855
2856
2857
2858
2859
2860
2861
2862
2863
2864
2865
2866
2867
2868
2869
2870
2871
2872
2873
2874
2875
2876
2877
2878
2879
2880
2881
2882
2883
2884
2885
2886
2887
2888
2889
2890
2891
2892
2893
2894
2895
2896
2897
2898
2899
2900
2901
2902
2903
2904
2905
2906
2907
2908
2909
2910
2911
2912
2913
2914
2915
2916
2917
2918
2919
2920
2921
2922
2923
2924
2925
2926
2927
2928
2929
2930
2931
2932
2933
2934
2935
2936
2937
2938
2939
2940
2941
2942
2943
2944
2945
2946
2947
2948
2949
2950
2951
2952
2953
2954
2955
2956
2957
2958
2959
2960
2961
2962
2963
2964
2965
2966
2967
2968
2969
2970
2971
2972
2973
2974
2975
2976
2977
2978
2979
2980
2981
2982
2983
2984
2985
2986
2987
2988
2989
def bayesian_search_forecaster_multivariate(
    forecaster: object,
    series: Union[pd.DataFrame, dict],
    search_space: Callable,
    steps: int,
    metric: Union[str, Callable, list],
    initial_train_size: int,
    aggregate_metric: Union[str, list] = ['weighted_average', 'average', 'pooling'],
    fixed_train_size: bool = True,
    gap: int = 0,
    skip_folds: Optional[Union[int, list]] = None,
    allow_incomplete_fold: bool = True,
    levels: Optional[Union[str, list]] = None,
    exog: Optional[Union[pd.Series, pd.DataFrame, dict]] = None,
    refit: Union[bool, int] = False,
    n_trials: int = 10,
    random_state: int = 123,
    return_best: bool = True,
    n_jobs: Union[int, str] = 'auto',
    verbose: bool = True,
    show_progress: bool = True,
    suppress_warnings: bool = False,
    output_file: Optional[str] = None,
    engine: str = 'optuna',
    kwargs_create_study: dict = {},
    kwargs_study_optimize: dict = {}
) -> Tuple[pd.DataFrame, object]:
    """
    This function is an alias of bayesian_search_forecaster_multiseries.

    Bayesian optimization for a Forecaster object using multi-series backtesting 
    and optuna library.
    **New in version 0.12.0**

    Parameters
    ----------
    forecaster : ForecasterAutoregMultiSeries, ForecasterAutoregMultiSeriesCustom, ForecasterAutoregMultiVariate
        Forecaster model.
    series : pandas DataFrame, dict
        Training time series.
    search_space : Callable
        Function with argument `trial` which returns a dictionary with parameters names 
        (`str`) as keys and Trial object from optuna (trial.suggest_float, 
        trial.suggest_int, trial.suggest_categorical) as values.
    steps : int
        Number of steps to predict.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    initial_train_size : int 
        Number of samples in the initial train split.
    aggregate_metric : str, list, default `['weighted_average', 'average', 'pooling']`
        Aggregation method/s used to combine the metric/s of all levels (series)
        when multiple levels are predicted. If list, the first aggregation method
        is used to select the best parameters.

        - 'average': the average (arithmetic mean) of all levels.
        - 'weighted_average': the average of the metrics weighted by the number of
        predicted values of each level.
        - 'pooling': the values of all levels are pooled and then the metric is
        calculated.
    fixed_train_size : bool, default `True`
        If True, train size doesn't increase but moves by `steps` in each iteration.
    gap : int, default `0`
        Number of samples to be excluded after the end of each training set and 
        before the test set.
    skip_folds : int, list, default `None`
        If `skip_folds` is an integer, every 'skip_folds'-th is returned. If `skip_folds`
        is a list, the folds in the list are skipped. For example, if `skip_folds = 3`,
        and there are 10 folds, the folds returned will be [0, 3, 6, 9]. If `skip_folds`
        is a list [1, 2, 3], the folds returned will be [0, 4, 5, 6, 7, 8, 9].
    allow_incomplete_fold : bool, default `True`
        Last fold is allowed to have a smaller number of samples than the 
        `test_size`. If `False`, the last fold is excluded.
    levels : str, list, default `None`
        level (`str`) or levels (`list`) at which the forecaster is optimized. 
        If `None`, all levels are taken into account. The resulting metric will be
        the average of the optimization of all levels.
    exog : pandas Series, pandas DataFrame, dict, default `None`
        Exogenous variables.
    refit : bool, int, default `False`
        Whether to re-fit the forecaster in each iteration. If `refit` is an 
        integer, the Forecaster will be trained every that number of iterations.
    n_trials : int, default `10`
        Number of parameter settings that are sampled in each lag configuration.
    random_state : int, default `123`
        Sets a seed to the sampling for reproducible output.
    return_best : bool, default `True`
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default `'auto'`
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting.
        **New in version 0.9.0**
    verbose : bool, default `True`
        Print number of folds used for cv or backtesting.
    show_progress : bool, default `True`
        Whether to show a progress bar.
    suppress_warnings: bool, default `False`
        If `True`, skforecast warnings will be suppressed during the hyperparameter
        search. See skforecast.exceptions.warn_skforecast_categories for more
        information.
    output_file : str, default `None`
        Specifies the filename or full path where the results should be saved. 
        The results will be saved in a tab-separated values (TSV) format. If 
        `None`, the results will not be saved to a file.
        **New in version 0.12.0**
    engine : str, default `'optuna'`
        Bayesian optimization runs through the optuna library.
    kwargs_create_study : dict, default `{}`
        Keyword arguments (key, value mappings) to pass to optuna.create_study().
        If default, the direction is set to 'minimize' and a TPESampler(seed=123) 
        sampler is used during optimization.
    kwargs_study_optimize : dict, default `{}`
        Other keyword arguments (key, value mappings) to pass to study.optimize().

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column levels: levels configuration for each iteration.
        - column lags: lags configuration for each iteration.
        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration. The resulting 
        metric will be the average of the optimization of all levels.
        - additional n columns with param = value.
    best_trial : optuna object
        The best optimization result returned as a FrozenTrial optuna object.

    """

    results, best_trial = bayesian_search_forecaster_multiseries(
                              forecaster            = forecaster,
                              series                = series,
                              exog                  = exog,
                              levels                = levels,
                              search_space          = search_space,
                              steps                 = steps,
                              metric                = metric,
                              aggregate_metric      = aggregate_metric,
                              refit                 = refit,
                              initial_train_size    = initial_train_size,
                              fixed_train_size      = fixed_train_size,
                              gap                   = gap,
                              skip_folds            = skip_folds,
                              allow_incomplete_fold = allow_incomplete_fold,
                              n_trials              = n_trials,
                              random_state          = random_state,
                              return_best           = return_best,
                              n_jobs                = n_jobs,
                              verbose               = verbose,
                              show_progress         = show_progress,
                              suppress_warnings     = suppress_warnings,
                              output_file           = output_file,
                              engine                = engine,
                              kwargs_create_study   = kwargs_create_study,
                              kwargs_study_optimize = kwargs_study_optimize
                          )

    return results, best_trial