Making a Future Time Series Index using timetk

Matt Dancho

2017-08-17

A collection of tools for working with time series in R

A common task in forecasting is generating a future date sequence that mimics the existing index. This task can be incredibly important to the validity of the model. In this vignette the user will learn several methods to generate a future time series index from an existing index using the tk_make_future_timeseries() function along with pros and cons of each and testing for accuracy. We’ll focus on making future dates with a daily frequency, which tends to be the most difficult to generate due to holidays, seasonality, weekends, etc. We’ll use two cases to illustrate the pros and cons:

A simple example case with weekends and the last two weeks of the year missing

A example case using daily trade data for FB from 2013 through 2016

Prerequisites

Before we get started, load the following packages.

library(tidyquant)
library(timetk)

Example 1: Remove Weekends and Last Two Weeks of Year

This is a simulated example that shows how to use the tk_make_future_timeseries() function. We’ll create a sequence of dates beginning in 2010, and extending through the first two weeks of 2013. The sequence is daily periodicity. We’ll split into training and test sets. The train set will be used to predict the future dates. The test set will be used for comparison.

First we visualize the frequency of the simulated test index, which is what we will generate as a future time series. The tk_get_timeseries_signature() function retrieves the differences (frequency or “diff” column) of the test dataset. We can then plot using ggplot(). The frequency (diff) is in seconds. Since it’s daily periodicity, the base 86,400 seconds. Each small spike to 253,800 seconds is a weekend, and the large spike to 1,555,200 seconds is the missing two week period at the end of the year.

Inspect weekdays

inspect_weekdays is useful for finding and removing missing dates that occur on a weekly, bi-weekly, tri-weekly or quad-weekly frequency (such as weekends or every other Friday off). When applied to the example, the algorithm finds and removes omitted weekends of the future data set.

Inspect months

inspect_months is useful for finding and removing missing dates that occur on a monthly, quarterly, yearly frequency (such as last couple weeks of the year off). Applying to the example, the algorithm finds and removes omitted dates in the last two weeks of the year of the future data set.

Inspect both weekdays and months

We can combine both inspect_weekdays and inspect_months to detect both sequences of missing dates. Applying both to the example removes the weekends and last two weeks of each year. However, there is one error made due to the interactions of the two algorithms.

Reviewing errors

We can inspect further by analyzing both the Type I errors (errors that the algorithm removed incorrectly) and Type II errors (errors that the prediction that algorithm failed to remove).

Type I Errors

Errors that algorithm removed incorrectly. These errors are the most dangerous because users may not know which days were removed incorrectly. If known, they can be accounted for with insert_values. The easiest way to tell is by reviewing the frequency chart for larger than normal spikes.

idx_test[!(idx_test %in%idx_future_wdays_and_months)]

## [1] "Date of length 0"

Type II Errors

Errorst that algorithm failed to remove. These errors are the eaisest to manage because typically the analyst generally knows which days should be removed. These errors can be addressed with skip_values provided prediction length is manageable.

Using skip values and insert values

insert_values adds values to the future time series, and skip_values removes values from the future time series. The Type I errors (incorrectly removed observations) can be addressed with insert_values. The Type II errors (incorrectly kept observations) can be addressed with skip_values. We had one Type II error in the example, and we’ll correct with skip_values. Make sure the class of the value passed to skip_values matches the class of the time series index.

Example 2: FB days traded

The FB data from the FANG data set has missing dates that follow primarily a weekly pattern with irregular holidays mixed in. We’ll predict which days will be traded using combinations of inspect_weekdays and inspect_months.

Inspect weekdays and months

If we apply both inspect weekdays and inspect months the errors actually increase. This is due to the difficulty in predicting holidays (days off throughout the year), which tend to occur with unique rules (e.g. Memorial Day is last Monday of May).

Skip values

We can correct for this using the skip_values and inspect_weekdays arguments. The former removes specific observations while the latter uses a logistic regression algorithm to identify the probability of specific weekdays being present in the future index. The algorithm for inspecting weekdays will check for any weekdays that are missing on a weekly, bi-weekly, tri-weekly, or quad-weekly frequency and automatically remove these days. As shown below, holidays and weekends are accounted for.

Parting Thoughts

Building the future date sequence is an essential part of forecasting using timetk. An innaccurate date sequence can hurt the performance of the forecast. The tk_make_future_index() function has methods to remove frequently missing dates. However, the user should always review the output to ensure an accurate future date sequence is achieved.

Further, holidays are difficult to predict using inspect_months due to the irregular nature. It is likely better for the user to select inspect_weekdays = TRUE and use skip_values to remove irregular holidays. Use inspect_months when there are missing dates that occur on a monthly, quarterly, or yearly frequency as part of a regular pattern. Use skip_values and insert_values to remove and add dates as necessary to account for irregular missing days.