Quick walk through of tsfresh with financial data (Part I)

tsfresh is a powerful feature engineering package for time series data. But there are limited examples out there on how to use it on financial time series data.

So here is a simple and easy to follow walk through on how to use tsfresh package on stock price data.

functions in tsfresh package

I use roll_time_series function to prepare data and extract_features function to generate features from Close price data. I also use functions under feature_calculators module to validate generated features.

• roll_time_series prepares the raw time series data in the format that can be processed by extract_features function.

• extract_features function requires the input dataset to have at least three columns: a ‘id' column, a ‘time' column, and a numeric column that new features will be generated from. Note that ‘id' column and ‘time' column do not need to be named as ‘id' and ‘time'.

• feature_calculators module includes all the functions encapsuled in extract_features function. For example, extract_features returns a feature with suffix ‘abs_energy', the values in this feature are the same as the results of using feature_calculators.abs_energy function with same input data.

1. load data

load csv data into a pandas dataframe.

import pandas as pd
import numpy as np
import os

pd.options.display.max_rows = 999
pd.options.display.max_columns = 999

df = pd.read_csv('data/^GSPC.csv')

df.head()

df.columns

Index(['Date', 'Open', 'High', 'Low', 'Close', 'Adj Close', 'Volume'], dtype='object')

2. pre-process data

create dummy id column

#keep only Date and Close
df = df[['Date', 'Close']].copy(deep=True)

#create a dummy id column
df['id']=1

df.head()

df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3301 entries, 0 to 3300
Data columns (total 3 columns):
Date     3301 non-null object
Close    3301 non-null float64
id       3301 non-null int64
dtypes: float64(1), int64(1), object(1)
memory usage: 77.5+ KB

3. prepare data using roll_time_series

1. id column will be converted into a tuple;
2. Date columns is used in column_sort parameter.
3. max_timeshift and min_timeshift: must be a integer >= than min_timeshift
   • setup 1: max_timeshift=6, min_timeshift=6
   • setup 2: max_timeshift=6, min_timeshift=4

from tsfresh.utilities.dataframe_functions import roll_time_series
from tsfresh import extract_features

df_rolled = roll_time_series(df, column_id="id", column_sort="Date", max_timeshift=6, min_timeshift=6, disable_progressbar=True)

df_rolled.head(7)

print(df.shape, df_rolled.shape)

(3301, 3) (23065, 3)

df.head()

df_rolled2 = roll_time_series(df, column_id="id", column_sort="Date", max_timeshift=6, min_timeshift=4, disable_progressbar=True)

df_rolled2.head(7)

compare the two setups of max_timeshift and min_timeshift

1. setup 1: max_timeshift=6, min_timeshift=6
2. setup 2: max_timeshift=6, min_timeshift=4

df_rolled[df_rolled['Date']=='2007-01-03']

df_rolled2[df_rolled2['Date']=='2007-01-03']

df_rolled.tail(3)

df_rolled2.tail(3)

print(df.shape, df_rolled.shape, df_rolled2.shape)

(3301, 3) (23065, 3) (23076, 3)

df_rolled[df_rolled['id']==df_rolled['id'].values[0]]

df_rolled2[df_rolled2['id']==df_rolled['id'].values[0]]

df_rolled[df_rolled['Date']=='2007-01-10']

df_rolled2[df_rolled2['Date']=='2007-01-10']

4. create new features using extract_features

df_features = extract_features(df_rolled, column_id="id", column_sort="Date")

Feature Extraction: 100%|██████████████████████████████████████████████████████████████| 20/20 [00:40<00:00,  2.05s/it]

df_features2 = extract_features(df_rolled2, column_id="id", column_sort="Date")

Feature Extraction: 100%|██████████████████████████████████████████████████████████████| 20/20 [00:39<00:00,  1.96s/it]

print(df_features.shape, df_features2.shape)

(3295, 779) (3297, 779)

df_features.head()

df_features2.head()