Skip to content

model_selection_sarimax

backtesting_sarimax(forecaster, y, steps, metric, initial_train_size, fixed_train_size=True, gap=0, allow_incomplete_fold=True, exog=None, refit=False, alpha=None, interval=None, n_jobs='auto', verbose=False, suppress_warnings_fit=False, show_progress=True)

Backtesting of ForecasterSarimax.

  • If refit is False, the model will be trained only once using the initial_train_size first observations.
  • If refit is True, the model is trained on each iteration, increasing the training set.
  • If refit is an integer, the model will be trained every that number of iterations.

A copy of the original forecaster is created so that it is not modified during the process.

Parameters:

Name Type Description Default
forecaster ForecasterSarimax

Forecaster model.

required
y pandas Series

Training time series.

required
steps int

Number of steps to predict.

required
metric (str, Callable, list)

Metric used to quantify the goodness of fit of the model.

  • If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
  • If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float.
  • If list: List containing multiple strings and/or Callables.
required
initial_train_size int

Number of samples in the initial train split. The backtest forecaster is trained using the first initial_train_size observations.

required
fixed_train_size bool

If True, train size doesn't increase but moves by steps in each iteration.

`True`
gap int

Number of samples to be excluded after the end of each training set and before the test set.

`0`
allow_incomplete_fold bool

Last fold is allowed to have a smaller number of samples than the test_size. If False, the last fold is excluded.

`True`
exog pandas Series, pandas DataFrame

Exogenous variable/s included as predictor/s. Must have the same number of observations as y and should be aligned so that y[i] is regressed on exog[i].

`None`
refit (bool, int)

Whether to re-fit the forecaster in each iteration. If refit is an integer, the Forecaster will be trained every that number of iterations.

`False`
alpha float

The confidence intervals for the forecasts are (1 - alpha) %. If both, alpha and interval are provided, alpha will be used.

`0.05`
interval list

Confidence of the prediction interval estimated. The values must be symmetric. Sequence of percentiles to compute, which must be between 0 and 100 inclusive. For example, interval of 95% should be as interval = [2.5, 97.5]. If both, alpha and interval are provided, alpha will be used.

`None`
n_jobs (int, auto)

The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting. New in version 0.9.0

`'auto'`
verbose bool

Print number of folds and index of training and validation sets used for backtesting.

`False`
suppress_warnings_fit bool

If True, warnings generated during fitting will be ignored. New in version 0.10.0

`False`
show_progress bool

Whether to show a progress bar.

`True`

Returns:

Name Type Description
metric_values pandas DataFrame

Value(s) of the metric(s).

backtest_predictions pandas DataFrame

Value of predictions and their estimated interval if interval is not None.

  • column pred: predictions.
  • column lower_bound: lower bound of the interval.
  • column upper_bound: upper bound of the interval.
Source code in skforecast\model_selection_sarimax\model_selection_sarimax.py
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
def backtesting_sarimax(
    forecaster: object,
    y: pd.Series,
    steps: int,
    metric: Union[str, Callable, list],
    initial_train_size: int,
    fixed_train_size: bool = True,
    gap: int = 0,
    allow_incomplete_fold: bool = True,
    exog: Optional[Union[pd.Series, pd.DataFrame]] = None,
    refit: Union[bool, int] = False,
    alpha: Optional[float] = None,
    interval: Optional[list] = None,
    n_jobs: Union[int, str] = 'auto',
    verbose: bool = False,
    suppress_warnings_fit: bool = False,
    show_progress: bool = True
) -> Tuple[pd.DataFrame, pd.DataFrame]:
    """
    Backtesting of ForecasterSarimax.

    - If `refit` is `False`, the model will be trained only once using the 
    `initial_train_size` first observations. 
    - If `refit` is `True`, the model is trained on each iteration, increasing
    the training set. 
    - If `refit` is an `integer`, the model will be trained every that number 
    of iterations.

    A copy of the original forecaster is created so that it is not modified during 
    the process.

    Parameters
    ----------
    forecaster : ForecasterSarimax
        Forecaster model.
    y : pandas Series
        Training time series.
    steps : int
        Number of steps to predict.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    initial_train_size : int
        Number of samples in the initial train split. The backtest forecaster is
        trained using the first `initial_train_size` observations.
    fixed_train_size : bool, default `True`
        If True, train size doesn't increase but moves by `steps` in each iteration.
    gap : int, default `0`
        Number of samples to be excluded after the end of each training set and 
        before the test set.
    allow_incomplete_fold : bool, default `True`
        Last fold is allowed to have a smaller number of samples than the 
        `test_size`. If `False`, the last fold is excluded.
    exog : pandas Series, pandas DataFrame, default `None`
        Exogenous variable/s included as predictor/s. Must have the same
        number of observations as `y` and should be aligned so that y[i] is
        regressed on exog[i].
    refit : bool, int, default `False`
        Whether to re-fit the forecaster in each iteration. If `refit` is an 
        integer, the Forecaster will be trained every that number of iterations.
    alpha : float, default `0.05`
        The confidence intervals for the forecasts are (1 - alpha) %.
        If both, `alpha` and `interval` are provided, `alpha` will be used.
    interval : list, default `None`
        Confidence of the prediction interval estimated. The values must be
        symmetric. Sequence of percentiles to compute, which must be between 
        0 and 100 inclusive. For example, interval of 95% should be as 
        `interval = [2.5, 97.5]`. If both, `alpha` and `interval` are 
        provided, `alpha` will be used.
    n_jobs : int, 'auto', default `'auto'`
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting.
        **New in version 0.9.0**     
    verbose : bool, default `False`
        Print number of folds and index of training and validation sets used 
        for backtesting.
    suppress_warnings_fit : bool, default `False`
        If `True`, warnings generated during fitting will be ignored.
        **New in version 0.10.0**
    show_progress : bool, default `True`
        Whether to show a progress bar.

    Returns
    -------
    metric_values : pandas DataFrame
        Value(s) of the metric(s).
    backtest_predictions : pandas DataFrame
        Value of predictions and their estimated interval if `interval` is not `None`.

        - column pred: predictions.
        - column lower_bound: lower bound of the interval.
        - column upper_bound: upper bound of the interval.

    """

    if type(forecaster).__name__ not in ['ForecasterSarimax']:
        raise TypeError(
            ("`forecaster` must be of type `ForecasterSarimax`, for all other "
             "types of forecasters use the functions available in the other "
             "`model_selection` modules.")
        )

    check_backtesting_input(
        forecaster            = forecaster,
        steps                 = steps,
        metric                = metric,
        y                     = y,
        initial_train_size    = initial_train_size,
        fixed_train_size      = fixed_train_size,
        gap                   = gap,
        allow_incomplete_fold = allow_incomplete_fold,
        refit                 = refit,
        interval              = interval,
        alpha                 = alpha,
        n_jobs                = n_jobs,
        verbose               = verbose,
        show_progress         = show_progress
    )

    metric_values, backtest_predictions = _backtesting_sarimax(
        forecaster            = forecaster,
        y                     = y,
        steps                 = steps,
        metric                = metric,
        initial_train_size    = initial_train_size,
        fixed_train_size      = fixed_train_size,
        gap                   = gap,
        allow_incomplete_fold = allow_incomplete_fold,
        exog                  = exog,
        refit                 = refit,
        alpha                 = alpha,
        interval              = interval,
        n_jobs                = n_jobs,
        verbose               = verbose,
        suppress_warnings_fit = suppress_warnings_fit,
        show_progress         = show_progress
    )

    return metric_values, backtest_predictions

grid_search_sarimax(forecaster, y, param_grid, steps, metric, initial_train_size, fixed_train_size=True, gap=0, allow_incomplete_fold=True, exog=None, refit=False, return_best=True, n_jobs='auto', verbose=True, suppress_warnings_fit=False, show_progress=True, output_file=None)

Exhaustive search over specified parameter values for a ForecasterSarimax object. Validation is done using time series backtesting.

Parameters:

Name Type Description Default
forecaster ForecasterSarimax

Forecaster model.

required
y pandas Series

Training time series.

required
param_grid dict

Dictionary with parameters names (str) as keys and lists of parameter settings to try as values.

required
steps int

Number of steps to predict.

required
metric (str, Callable, list)

Metric used to quantify the goodness of fit of the model.

  • If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
  • If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float.
  • If list: List containing multiple strings and/or Callables.
required
initial_train_size int

Number of samples in the initial train split. The backtest forecaster is trained using the first initial_train_size observations.

required
fixed_train_size bool

If True, train size doesn't increase but moves by steps in each iteration.

`True`
gap int

Number of samples to be excluded after the end of each training set and before the test set.

`0`
allow_incomplete_fold bool

Last fold is allowed to have a smaller number of samples than the test_size. If False, the last fold is excluded.

`True`
exog pandas Series, pandas DataFrame

Exogenous variable/s included as predictor/s. Must have the same number of observations as y and should be aligned so that y[i] is regressed on exog[i].

`None`
refit (bool, int)

Whether to re-fit the forecaster in each iteration. If refit is an integer, the Forecaster will be trained every that number of iterations.

`False`
return_best bool

Refit the forecaster using the best found parameters on the whole data.

`True`
n_jobs (int, auto)

The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting. New in version 0.9.0

`'auto'`
verbose bool

Print number of folds used for cv or backtesting.

`True`
suppress_warnings_fit bool

If True, warnings generated during fitting will be ignored. New in version 0.10.0

`False`
show_progress bool

Whether to show a progress bar.

`True`
output_file str

Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file. New in version 0.12.0

`None`

Returns:

Name Type Description
results pandas DataFrame

Results for each combination of parameters.

  • column params: parameters configuration for each iteration.
  • column metric: metric value estimated for each iteration.
  • additional n columns with param = value.
Source code in skforecast\model_selection_sarimax\model_selection_sarimax.py
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
def grid_search_sarimax(
    forecaster: object,
    y: pd.Series,
    param_grid: dict,
    steps: int,
    metric: Union[str, Callable, list],
    initial_train_size: int,
    fixed_train_size: bool = True,
    gap: int = 0,
    allow_incomplete_fold: bool = True,
    exog: Optional[Union[pd.Series, pd.DataFrame]] = None,
    refit: Union[bool, int] = False,
    return_best: bool = True,
    n_jobs: Union[int, str] = 'auto',
    verbose: bool = True,
    suppress_warnings_fit: bool = False,
    show_progress: bool = True,
    output_file: Optional[str] = None
) -> pd.DataFrame:
    """
    Exhaustive search over specified parameter values for a ForecasterSarimax object.
    Validation is done using time series backtesting.

    Parameters
    ----------
    forecaster : ForecasterSarimax
        Forecaster model.
    y : pandas Series
        Training time series. 
    param_grid : dict
        Dictionary with parameters names (`str`) as keys and lists of parameter
        settings to try as values.
    steps : int
        Number of steps to predict.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    initial_train_size : int 
        Number of samples in the initial train split. The backtest forecaster is
        trained using the first `initial_train_size` observations.
    fixed_train_size : bool, default `True`
        If True, train size doesn't increase but moves by `steps` in each iteration.
    gap : int, default `0`
        Number of samples to be excluded after the end of each training set and 
        before the test set.
    allow_incomplete_fold : bool, default `True`
        Last fold is allowed to have a smaller number of samples than the 
        `test_size`. If `False`, the last fold is excluded.
    exog : pandas Series, pandas DataFrame, default `None`
        Exogenous variable/s included as predictor/s. Must have the same
        number of observations as `y` and should be aligned so that y[i] is
        regressed on exog[i].
    refit : bool, int, default `False`
        Whether to re-fit the forecaster in each iteration. If `refit` is an 
        integer, the Forecaster will be trained every that number of iterations.
    return_best : bool, default `True`
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default `'auto'`
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting.
        **New in version 0.9.0**
    verbose : bool, default `True`
        Print number of folds used for cv or backtesting.
    suppress_warnings_fit : bool, default `False`
        If `True`, warnings generated during fitting will be ignored.
        **New in version 0.10.0**
    show_progress : bool, default `True`
        Whether to show a progress bar.
    output_file : str, default `None`
        Specifies the filename or full path where the results should be saved. 
        The results will be saved in a tab-separated values (TSV) format. If 
        `None`, the results will not be saved to a file.
        **New in version 0.12.0**

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration.
        - additional n columns with param = value.

    """

    param_grid = list(ParameterGrid(param_grid))

    results = _evaluate_grid_hyperparameters_sarimax(
        forecaster            = forecaster,
        y                     = y,
        param_grid            = param_grid,
        steps                 = steps,
        metric                = metric,
        initial_train_size    = initial_train_size,
        fixed_train_size      = fixed_train_size,
        gap                   = gap,
        allow_incomplete_fold = allow_incomplete_fold,
        exog                  = exog,
        refit                 = refit,
        return_best           = return_best,
        n_jobs                = n_jobs,
        verbose               = verbose,
        suppress_warnings_fit = suppress_warnings_fit,
        show_progress         = show_progress,
        output_file           = output_file
    )

    return results

random_search_sarimax(forecaster, y, param_distributions, steps, metric, initial_train_size, fixed_train_size=True, gap=0, allow_incomplete_fold=True, exog=None, refit=False, n_iter=10, random_state=123, return_best=True, n_jobs='auto', verbose=True, suppress_warnings_fit=False, show_progress=True, output_file=None)

Random search over specified parameter values or distributions for a Forecaster object. Validation is done using time series backtesting.

Parameters:

Name Type Description Default
forecaster ForecasterSarimax

Forecaster model.

required
y pandas Series

Training time series.

required
param_distributions dict

Dictionary with parameters names (str) as keys and distributions or lists of parameters to try.

required
steps int

Number of steps to predict.

required
metric (str, Callable, list)

Metric used to quantify the goodness of fit of the model.

  • If string: {'mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error', 'mean_squared_log_error', 'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
  • If Callable: Function with arguments y_true, y_pred and y_train (Optional) that returns a float.
  • If list: List containing multiple strings and/or Callables.
required
initial_train_size int

Number of samples in the initial train split. The backtest forecaster is trained using the first initial_train_size observations.

required
fixed_train_size bool

If True, train size doesn't increase but moves by steps in each iteration.

`True`
gap int

Number of samples to be excluded after the end of each training set and before the test set.

`0`
allow_incomplete_fold bool

Last fold is allowed to have a smaller number of samples than the test_size. If False, the last fold is excluded.

`True`
exog pandas Series, pandas DataFrame

Exogenous variable/s included as predictor/s. Must have the same number of observations as y and should be aligned so that y[i] is regressed on exog[i].

`None`
refit (bool, int)

Whether to re-fit the forecaster in each iteration. If refit is an integer, the Forecaster will be trained every that number of iterations.

`False`
n_iter int

Number of parameter settings that are sampled. n_iter trades off runtime vs quality of the solution.

`10`
random_state int

Sets a seed to the random sampling for reproducible output.

`123`
return_best bool

Refit the forecaster using the best found parameters on the whole data.

`True`
n_jobs (int, auto)

The number of jobs to run in parallel. If -1, then the number of jobs is set to the number of cores. If 'auto', n_jobs is set using the function skforecast.utils.select_n_jobs_backtesting. New in version 0.9.0

`'auto'`
verbose bool

Print number of folds used for cv or backtesting.

`True`
suppress_warnings_fit bool

If True, warnings generated during fitting will be ignored. New in version 0.10.0

`False`
show_progress bool

Whether to show a progress bar.

`True`
output_file str

Specifies the filename or full path where the results should be saved. The results will be saved in a tab-separated values (TSV) format. If None, the results will not be saved to a file. New in version 0.12.0

`None`

Returns:

Name Type Description
results pandas DataFrame

Results for each combination of parameters.

  • column params: parameters configuration for each iteration.
  • column metric: metric value estimated for each iteration.
  • additional n columns with param = value.
Source code in skforecast\model_selection_sarimax\model_selection_sarimax.py
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
def random_search_sarimax(
    forecaster: object,
    y: pd.Series,
    param_distributions: dict,
    steps: int,
    metric: Union[str, Callable, list],
    initial_train_size: int,
    fixed_train_size: bool = True,
    gap: int = 0,
    allow_incomplete_fold: bool = True,
    exog: Optional[Union[pd.Series, pd.DataFrame]] = None,
    refit: Union[bool, int] = False,
    n_iter: int = 10,
    random_state: int = 123,
    return_best: bool = True,
    n_jobs: Union[int, str] = 'auto',
    verbose: bool = True,
    suppress_warnings_fit: bool = False,
    show_progress: bool = True,
    output_file: Optional[str] = None
) -> pd.DataFrame:
    """
    Random search over specified parameter values or distributions for a Forecaster 
    object. Validation is done using time series backtesting.

    Parameters
    ----------
    forecaster : ForecasterSarimax
        Forecaster model.
    y : pandas Series
        Training time series. 
    param_distributions : dict
        Dictionary with parameters names (`str`) as keys and 
        distributions or lists of parameters to try.
    steps : int
        Number of steps to predict.
    metric : str, Callable, list
        Metric used to quantify the goodness of fit of the model.

        - If `string`: {'mean_squared_error', 'mean_absolute_error',
        'mean_absolute_percentage_error', 'mean_squared_log_error',
        'mean_absolute_scaled_error', 'root_mean_squared_scaled_error'}
        - If `Callable`: Function with arguments `y_true`, `y_pred` and `y_train`
        (Optional) that returns a float.
        - If `list`: List containing multiple strings and/or Callables.
    initial_train_size : int 
        Number of samples in the initial train split. The backtest forecaster is
        trained using the first `initial_train_size` observations.
    fixed_train_size : bool, default `True`
        If True, train size doesn't increase but moves by `steps` in each iteration.
    gap : int, default `0`
        Number of samples to be excluded after the end of each training set and 
        before the test set.
    allow_incomplete_fold : bool, default `True`
        Last fold is allowed to have a smaller number of samples than the 
        `test_size`. If `False`, the last fold is excluded.
    exog : pandas Series, pandas DataFrame, default `None`
        Exogenous variable/s included as predictor/s. Must have the same
        number of observations as `y` and should be aligned so that y[i] is
        regressed on exog[i].
    refit : bool, int, default `False`
        Whether to re-fit the forecaster in each iteration. If `refit` is an 
        integer, the Forecaster will be trained every that number of iterations.
    n_iter : int, default `10`
        Number of parameter settings that are sampled. 
        n_iter trades off runtime vs quality of the solution.
    random_state : int, default `123`
        Sets a seed to the random sampling for reproducible output.
    return_best : bool, default `True`
        Refit the `forecaster` using the best found parameters on the whole data.
    n_jobs : int, 'auto', default `'auto'`
        The number of jobs to run in parallel. If `-1`, then the number of jobs is 
        set to the number of cores. If 'auto', `n_jobs` is set using the function
        skforecast.utils.select_n_jobs_backtesting.
        **New in version 0.9.0**
    verbose : bool, default `True`
        Print number of folds used for cv or backtesting.
    suppress_warnings_fit : bool, default `False`
        If `True`, warnings generated during fitting will be ignored.
        **New in version 0.10.0**
    show_progress : bool, default `True`
        Whether to show a progress bar.
    output_file : str, default `None`
        Specifies the filename or full path where the results should be saved. 
        The results will be saved in a tab-separated values (TSV) format. If 
        `None`, the results will not be saved to a file.
        **New in version 0.12.0**

    Returns
    -------
    results : pandas DataFrame
        Results for each combination of parameters.

        - column params: parameters configuration for each iteration.
        - column metric: metric value estimated for each iteration.
        - additional n columns with param = value.

    """

    param_grid = list(ParameterSampler(param_distributions, n_iter=n_iter, random_state=random_state))

    results = _evaluate_grid_hyperparameters_sarimax(
        forecaster            = forecaster,
        y                     = y,
        param_grid            = param_grid,
        steps                 = steps,
        metric                = metric,
        initial_train_size    = initial_train_size,
        fixed_train_size      = fixed_train_size,
        gap                   = gap,
        allow_incomplete_fold = allow_incomplete_fold,
        exog                  = exog,
        refit                 = refit,
        return_best           = return_best,
        n_jobs                = n_jobs,
        verbose               = verbose,
        suppress_warnings_fit = suppress_warnings_fit,
        show_progress         = show_progress,
        output_file           = output_file
    )

    return results