层次模型教程#
本教程说明了如何使用GluonTS的基于深度学习的层次模型 DeepVarHierarchical。我们首先解释层次/分组时间序列的数据准备，然后展示模型训练、预测和评估的常见用例。
介绍#
使用层次模型的一个重要方面是对层次时间序列数据的正确准备。构建层次或分组时间序列的最小要求是层次底层的时间序列和层次/分组聚合矩阵。 GluonTS 提供了一种简单的方法，通过从 csv 文件中读取底层时间序列和聚合矩阵来构建层次时间序列。在这里，我们首先描述层次时间序列的准备，然后讨论层次模型的训练。
层次时间序列的准备#
底层时间序列假定以列的形式在csv文件中可用。csv文件还应包括索引列，列出每行的时间戳。请注意，聚合时间序列是自动构建的，不应提供。这里是一个底层时间序列的示例csv。类似地，聚合矩阵也可以从csv文件中读取；这里是一个示例。我们使用求和矩阵的标准格式；例如，请参见教材《预测：原则与实践》的公式10.3。请注意，分组时间序列也可以按照类似于层次时间序列的方式通过聚合矩阵表示，因此这里呈现的材料同样适用于分组时间序列。
[1]:
import pandas as pd
from gluonts.dataset.hierarchical import HierarchicalTimeSeries
# Load (only!) the time series at the bottom level of the hierarchy.
ts_at_bottom_level_csv = (
"https://gist.githubusercontent.com/rshyamsundar/39e57075743537c4100a716a7b7ec047/"
"raw/f02f5aeadad73e3f3e9cf54478606d3507826492/example_bottom_ts.csv"
)
# Make sure the dataframe has `PeriodIndex` by explicitly casting it to `PeriodIndex`.
ts_at_bottom_level = pd.read_csv(
ts_at_bottom_level_csv,
index_col=0,
parse_dates=True,
).to_period()
# Load the aggregation matrix `S`.
S_csv = (
"https://gist.githubusercontent.com/rshyamsundar/17084fd1f28021867bcf6f2d69d9b73a/raw/"
"32780ca43f57a78f2d521a75e73b136b17f34a02/example_agg_mat.csv"
)
S = pd.read_csv(S_csv).values
hts = HierarchicalTimeSeries(
ts_at_bottom_level=ts_at_bottom_level,
S=S,
)
可以通过 ts_at_all_levels 属性访问层次结构的所有时间序列。
[2]:
hts.ts_at_all_levels.head()
[2]:
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | |
|---|---|---|---|---|---|---|---|
| 2020-03-22 00:00 | 0.686671 | 0.156873 | 0.529798 | 0.056962 | 0.099911 | 0.039827 | 0.489971 |
| 2020-03-22 01:00 | 2.189128 | 0.669261 | 1.519866 | 0.246535 | 0.422727 | 0.763164 | 0.756702 |
| 2020-03-22 02:00 | 1.152853 | 0.582213 | 0.570640 | 0.314393 | 0.267820 | 0.169645 | 0.400996 |
| 2020-03-22 03:00 | 1.198889 | 0.653139 | 0.545750 | 0.609158 | 0.043981 | 0.235009 | 0.310741 |
| 2020-03-22 04:00 | 2.069197 | 0.678490 | 1.390707 | 0.380788 | 0.297702 | 0.898429 | 0.492278 |
模型训练与预测#
我们现在展示最简单的用例，其中我们希望在可用的整个数据集上训练模型，并为未来的时间步骤生成预测。请注意，这就是在实际操作中模型的使用方式，一旦根据某些用户特定的评估标准选定了最佳模型；请参见下一节的模型评估。
我们假设已经按照上面的描述构建了层次时间序列 hts，其类型为 HierarchicalTimeSeries。第一步是将这个层次时间序列转换为可以进行小批量训练的 Dataset。在这里，我们将其转换为 gluonts.dataset.pandas.PandasDataset。
[3]:
dataset = hts.to_dataset()
一旦数据集被创建，使用层次模型就很简单了。在这里，为了快速说明，我们固定预测长度并选择较小的训练轮数。我们在整个数据集上进行训练，并将相同的数据作为输入给训练好的模型（称为预测器）以生成未来/未见时间步的预测（或预测结果）。最终输出 forecasts 是 gluonts.model.forecast.SampleForecast 的一个实例，包含层次中所有时间序列的基于样本的预测。
[4]:
from gluonts.mx.model.deepvar_hierarchical import DeepVARHierarchicalEstimator
from gluonts.mx.trainer import Trainer
prediction_length = 24
estimator = DeepVARHierarchicalEstimator(
freq=hts.freq,
prediction_length=prediction_length,
trainer=Trainer(epochs=2),
S=S,
)
predictor = estimator.train(dataset)
forecast_it = predictor.predict(dataset)
# There is only one element in `forecast_it` containing forecasts for all the time series in the hierarchy.
forecasts = next(forecast_it)
100%|██████████| 50/50 [00:32<00:00, 1.53it/s, epoch=1/2, avg_epoch_loss=178]
100%|██████████| 50/50 [00:32<00:00, 1.53it/s, epoch=2/2, avg_epoch_loss=146]
使用外部动态特性#
默认情况下，层次模型 DeepVarHierarchical 会内部创建多个基于时间/动态特征用于模型训练。这些特征是根据目标时间序列的频率自动推导出的季节性特征。如果可用，用户也可以向模型提供外部动态特征。这里我们展示了这是如何完成的；我们从层次时间序列 hts 已经创建的位置重新开始。
我们首先从csv文件中加载可用的外部特征。
[5]:
dynamic_features_csv = (
"https://gist.githubusercontent.com/rshyamsundar/d8e63bad43397c95a4f5daaa17e122f8/"
"raw/a50657cf89f86d48cee41122f02cf5b1fcafdd2f/example_dynamic_features.csv"
)
dynamic_features_df = pd.read_csv(
dynamic_features_csv,
index_col=0,
parse_dates=True,
).to_period()
动态特征假定在“训练范围”（目标可用的时间点）以及“预测范围”（需要预测的未来时间点）都可用。因此，动态特征的时间步长比目标时间序列长 prediction_length 个时间步。
为了训练模型，我们只需要训练范围内的动态特征。
[6]:
dynamic_features_df_train = dynamic_features_df.iloc[:-prediction_length, :]
我们通过传递外部特征 dynamic_features_df_train 创建训练数据集并在其上训练模型。
[7]:
dataset_train = hts.to_dataset(feat_dynamic_real=dynamic_features_df_train)
estimator = DeepVARHierarchicalEstimator(
freq=hts.freq,
prediction_length=prediction_length,
trainer=Trainer(epochs=2),
S=S,
)
predictor = estimator.train(dataset_train)
100%|██████████| 50/50 [00:33<00:00, 1.48it/s, epoch=1/2, avg_epoch_loss=192]
100%|██████████| 50/50 [00:33<00:00, 1.49it/s, epoch=2/2, avg_epoch_loss=147]
为了生成未来/未见时间步的预测，我们需要将过去的目标（即训练范围内的目标）以及完整的特征 dynamic_features_df（包括那些在预测范围内的特征）传递给训练好的模型。因此，我们需要创建一个与 dataset_train 分开的新数据集，而不是像之前的情况那样。
[8]:
predictor_input = hts.to_dataset(feat_dynamic_real=dynamic_features_df)
forecast_it = predictor.predict(predictor_input)
# There is only one element in `forecast_it` containing forecasts for all the time series in the hierarchy.
forecasts = next(forecast_it)
通过回测进行模型评估#
我们现在解释如何通过回测评估层次模型。为了方便说明，我们描述了在外部动态特征可用的情况下的模型评估。然而，如果外部特征不可用，修改代码非常简单；只需在下面调用无任何参数的函数 to_dataset()。
我们假设底层的时间序列 ts_at_bottom_level 和聚合矩阵 S 如上所述已经被创建。我们通过保留最后 prediction_length 个时间点进行评估，将其余部分用于训练，从而在时间轴上创建训练-测试分割。
[9]:
prediction_length = 24
hts_train = HierarchicalTimeSeries(
ts_at_bottom_level=ts_at_bottom_level.iloc[:-prediction_length, :],
S=S,
)
hts_test_label = HierarchicalTimeSeries(
ts_at_bottom_level=ts_at_bottom_level.iloc[-prediction_length:, :],
S=S,
)
我们还加载外部特征，并切片与训练范围对应的特征。
[10]:
dynamic_features_csv = (
"https://gist.githubusercontent.com/rshyamsundar/d8e63bad43397c95a4f5daaa17e122f8/"
"raw/a50657cf89f86d48cee41122f02cf5b1fcafdd2f/example_dynamic_features.csv"
)
dynamic_features_df = pd.read_csv(
dynamic_features_csv,
index_col=0,
parse_dates=True,
).to_period()
dynamic_features_df_train = dynamic_features_df.iloc[:-prediction_length, :]
我们通过传递外部特征 dynamic_features_df_train 将 hts_train 转换为 PandasDataset 并在其上训练层次模型。
[11]:
dataset_train = hts_train.to_dataset(feat_dynamic_real=dynamic_features_df_train)
estimator = DeepVARHierarchicalEstimator(
freq=hts.freq,
prediction_length=prediction_length,
trainer=Trainer(epochs=2),
S=S,
)
predictor = estimator.train(dataset_train)
100%|██████████| 50/50 [00:33<00:00, 1.48it/s, epoch=1/2, avg_epoch_loss=176]
100%|██████████| 50/50 [00:33<00:00, 1.49it/s, epoch=2/2, avg_epoch_loss=144]
为了生成与被隐藏观察值对应的时间点的预测，我们需要将完整的特征以及训练范围内的目标传递给训练好的模型。因此，我们相应地创建预测器的输入数据集并生成预测。
[12]:
predictor_input = hts_train.to_dataset(feat_dynamic_real=dynamic_features_df)
forecast_it = predictor.predict(predictor_input)
一旦获得预测，我们可以将其与保留的观测值进行评估。 GluonTS 评估器将真实（保留的）观测值及相应的预测结果的迭代器作为输入进行评估。我们的预测已经是正确的格式，而我们的保留观测值在 hts_test_label 中。
[13]:
from gluonts.evaluation import MultivariateEvaluator
evaluator = MultivariateEvaluator()
agg_metrics, item_metrics = evaluator(
ts_iterator=[hts_test_label.ts_at_all_levels],
fcst_iterator=forecast_it,
)
print(
f"Mean (weighted) quantile loss over all time series: "
f"{agg_metrics['mean_wQuantileLoss']}"
)
Running evaluation: 1it [00:00, 145.95it/s]
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
return arr.astype(dtype, copy=True)
Running evaluation: 1it [00:00, 154.48it/s]
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
return arr.astype(dtype, copy=True)
Running evaluation: 1it [00:00, 156.65it/s]
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
return arr.astype(dtype, copy=True)
Running evaluation: 1it [00:00, 157.82it/s]
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
return arr.astype(dtype, copy=True)
Running evaluation: 1it [00:00, 163.84it/s]
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
return arr.astype(dtype, copy=True)
Running evaluation: 1it [00:00, 140.81it/s]
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
return arr.astype(dtype, copy=True)
Running evaluation: 1it [00:00, 164.84it/s]
Mean (weighted) quantile loss over all time series: 0.27951188855488635
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
return arr.astype(dtype, copy=True)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandas/core/dtypes/astype.py:138: UserWarning: Warning: converting a masked element to nan.
return arr.astype(dtype, copy=True)