Multi-Dataset Time Series Modeling - Stationarity, Seasonality, and Forecasting Performance

Abstract

This project presents a comprehensive comparative analysis of time series forecasting methodologies applied to six diverse datasets: financial markets (Gold, S&P 500, FTSE), cryptocurrency (USDC/USDT), and non-financial time series (StackOverflow questions, cinema ticket sales). The study systematically evaluates forecasting performance across multiple algorithmic approaches, from simple baseline methods to advanced stochastic models.

Methodology

Data preprocessing included stationarity testing using KPSS and Augmented Dickey-Fuller (ADF) tests, with first-order differencing applied to achieve stationarity across all datasets. Seasonality was investigated through manual decomposition, Fast Fourier Transform (FFT) analysis, and visual inspection of ACF/PACF plots. Ten forecasting techniques were implemented and compared: Naive Forecasting, Seasonal Forecasting, Average Forecasting, Average Difference Forecasting, Autoregression (AR), ARIMA with varying orders, SARIMA with multiple seasonal configurations, and a manual EMA-based ARIMA approach. Advanced applications included Monte Carlo simulation for uncertainty quantification and a simulated trading strategy using AR predictions.

Key Findings

Performance varied significantly by dataset characteristics. For financial data (Gold, S&P 500, FTSE, USDC/USDT close prices), Naive Forecasting consistently achieved the lowest error metrics, suggesting these series exhibit random walk behavior where the best prediction is simply the last observed value. For datasets with clear seasonal patterns, more sophisticated models excelled, such as SARIMA models achieved superior performance on cinema ticket sales and Seasonal Forecasting performed best for USDC/USDT tradecount. The analysis revealed that first-order differencing (I=1) was necessary and sufficient for achieving stationarity across all datasets, while second-order differencing generally degraded performance.

The study demonstrates that model selection should be guided by dataset characteristics rather than model complexity. For random-walk financial series, simple methods often outperform sophisticated models, while seasonal data benefits from ARIMA/SARIMA frameworks. The Monte Carlo simulations successfully generated probabilistic forecasts, though the trading strategy implementation revealed the challenges of translating statistical predictions into profitable trades. FFT analysis confirmed strong weekly seasonality in ticket sales but minimal seasonal components in financial markets, validating the differential performance of seasonal models across datasets.

Library Initialization

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import statsmodels.api as sm
import statsmodels.formula.api as smf
import os
import math

Datasets

The dataset that I have chosen for this assignment is using 4 of the available dataset provided in the UOL website. Gold data, S&P 500 data, FTSE data and USDCUSDT data. The other 2 dataset were found on www.kaggle.com. One of them is about the types of questions on stackoverflow over time, and one is about cinema tickets.

Gold, S&P 500, FTSE Dataset

We will explore the Gold, S&P 500 and FTSE dataset columns and plotting them

sp500_data = pd.read_csv('Datasets/SP 500 04072014 2011.csv')
sp500_data.head()

sp500_data['Date']

0         04/04/2014
1         04/04/2014
2         04/04/2014
3         04/04/2014
4         04/04/2014
             ...
501719    11/03/2009
501720    11/03/2009
501721    11/03/2009
501722    11/03/2009
501723    11/03/2009
Name: Date, Length: 501724, dtype: object

sp500_data.describe()

gold_data = pd.read_csv('Datasets/Gold 07042014 2011.csv')
gold_data.head()

gold_data['Date']

0         07/04/2014
1         07/04/2014
2         07/04/2014
3         07/04/2014
4         07/04/2014
             ...
886796    26/10/2011
886797    26/10/2011
886798    26/10/2011
886799    26/10/2011
886800    26/10/2011
Name: Date, Length: 886801, dtype: object

gold_data.describe()

ftse_data = pd.read_csv('Datasets/FTSE 04072014 2011.csv')

ftse_data.head()

ftse_data['Date']

0         07/04/2014
1         07/04/2014
2         07/04/2014
3         07/04/2014
4         07/04/2014
             ...
408801    01/02/2011
408802    01/02/2011
408803    01/02/2011
408804    01/02/2011
408805    01/02/2011
Name: Date, Length: 408806, dtype: object

ftse_data.describe()

A quick load and using the .head() function in pandas, gives us a quick view on all 3 of the dataset. They have similar columns where there are date and time separtely. While it is unsure about the 'Bar#' and 'Bar Index', the tick range could be the difference between the high and low columns.

gold_data['High_Low_Difference'] = gold_data['High'] - gold_data['Low']
gold_data[['Tick Range','High_Low_Difference']].head()

Based on the above result, we know that tick range is the difference between the high and the low columns, multiplied by 10. Not sure if we will need to use the result and information later.

We will now plot the columns open, close, high and low.

gold_data_test = gold_data
gold_data_test['Datetime'] = pd.to_datetime(
    gold_data_test['Date'] + ' ' + gold_data_test['Time'],
    format='%d/%m/%Y %H:%M:%S'
)

plt.plot(gold_data_test['Datetime'][:100], gold_data_test['Open'][:100])
plt.plot(gold_data_test['Datetime'][:100], gold_data_test['Close'][:100])
plt.plot(gold_data_test['Datetime'][:100], gold_data_test['High'][:100])
plt.plot(gold_data_test['Datetime'][:100], gold_data_test['Low'][:100])
plt.legend(['Open', 'Close', 'High', 'Low'])

As shown above, column 'high' shows the highest price that it has reached at the given time and the column 'low' shows the the opposite. While the high and low prices should be useful when using other models, but as of now, we will just stick to using only the close price.

def plot_gold_sp_ftse(data, title):
    data['Datetime'] = pd.to_datetime(
        data['Date'] + ' ' + data['Time'],
        format='%d/%m/%Y %H:%M:%S'
    )

    plt.plot(data['Datetime'], data['Close'])
    plt.title(title)
    plt.show()

plt.rcParams['figure.figsize'] = [7, 5]
plot_gold_sp_ftse(gold_data, 'Gold')
plot_gold_sp_ftse(sp500_data, 'S&P 500')
plot_gold_sp_ftse(ftse_data, 'FTSE')

The above are the normal graphs for

• Gold price, from Oct 2011 - Apr 2014

• S&P500 price, from Nov 2009 - Apr 2014

• FTSE price, from Feb 2011 - Apr 2014.

We will try to see if we can explore the relationship between them. Because just in a quick glance, we can see that Gold and S&P500 are inversely related, while FTSE is somewhat correlate to S&P500.

USDCUSDT Dataset

Next we will be looking into the USDCUSDT Dataset.

After using both methods, it seems to be quite quite undetermined.

usdcusdt_data = pd.read_csv('Datasets/USDC-USDT.csv')

usdcusdt_data.head()

usdcusdt_data['date']

0       2021-10-30 00:00:00
1       2021-10-29 00:00:00
2       2021-10-28 00:00:00
3       2021-10-27 00:00:00
4       2021-10-26 00:00:00
               ...
1046    2018-12-19 00:00:00
1047    2018-12-18 00:00:00
1048    2018-12-17 00:00:00
1049    2018-12-16 00:00:00
1050    2018-12-15 00:00:00
Name: date, Length: 1051, dtype: object

usdcusdt_data.describe()

Based on the data descriptions above, we can see that unix and date can be used for the datetime column, as both of the columns provide datetime information. Symbol column would not be useful in this context, among the price columns, we will be using the close price. As for the volume USDC, volume USDT, and tradecount, they will definitely benefit with other models, but we can explore how they look when we were to plot them

usdcusdt_explore_columns = ['close', 'Volume USDC', 'Volume USDT', 'tradecount']
usdcusdt_data['date'] = pd.to_datetime(usdcusdt_data['date'])

for column in usdcusdt_explore_columns:
    plt.plot(usdcusdt_data['date'], usdcusdt_data[column])
    plt.title(column)
    plt.show()

With them visualized, we can easily pick them to perform any form of analysis when required.

We can observe that the 'close' price column that at the start, the data has very high variance but exponentially decrease in its volatility, but the mean of the data might be the same overall.

For the volume, both had very low amounts and then increased. It can be seen that the entire portion of the graph shifted upwards with the mean.

Tradecount column data seem to have some seasonality to it, with some form of repeating waves.

Cinema Ticket Dataset

While this dataset consists of time component and it is time series, it is not directly apparent. This is because the dataset not only includes a datetime information, but also status of the ticket sale of the specific cinema. Therefore, we will need to first do some pre-processing.

cinemaTicket_data = pd.read_csv('Datasets/cinemaTicket_Ref.csv')

cinemaTicket_data.head()

Let's say for this project, we would like t be able to predict when is the best time to increase teh discount, so that it can encourage ticket sales during off peak hours. We will first need to explore which columns to be using that can best represent our analysis.

cinemaTicket_data.columns

Index(['film_code', 'cinema_code', 'total_sales', 'tickets_sold',
       'tickets_out', 'show_time', 'occu_perc', 'ticket_price', 'ticket_use',
       'capacity', 'date', 'month', 'quarter', 'day'],
      dtype='object')

cinemaTicket_data.describe()

When inspecting the columns, the .head() and .describe() output, we noted that there are film code identified, as well as cinema_code. While these details might be useful for other analysis, we might just explore ticket_sold, to which should be good enough to indicate the performance of a given cinema. Because there also got a date column, we can use it as the datetime column. There are also other time related columns, like month, quarter, and day. However, similar justification that it might be useful for other analysis, but for this project will use only the date column and ticket sold.

Then, it is also further noted that because there is cinema codes, identifying the cinema, let's explore this aspect.

cinemaTicket_data['Datetime'] = pd.to_datetime(cinemaTicket_data['date'])
cinemaTicket_data

cinemaTicket_data['show_time'].plot()

cinemaTicket_data['show_time'].hist()

After checking the column showtime, we cannot conclude what does the column represent. At first glance, we might be able to use it to indicate the hours in datetime, and we might even be able to dig into finding out the hours that have lower sales. However, that would not be the case and we would look only into which day has the highest and lowest sale, and see if we are able to identify any trends overall.

# list of unique cinema
unique_cinemas = cinemaTicket_data['cinema_code'].unique()
unique_cinemas

array([304, 352, 489, 429, 524,  71, 163, 450,  51, 522,  43, 529,  82,
       344,  73, 485, 518, 448, 431,  72, 144, 456, 238, 312, 168, 254,
       214, 474, 445, 300, 362, 324, 452, 291, 479, 210, 428, 277,  56,
       253, 243,  39, 310, 201, 457, 191, 532, 167, 266, 204, 380, 169,
       162, 506, 537, 513, 467, 509, 165, 262, 486, 198, 508, 222,  98,
       230, 156, 181, 141, 528,  94, 350, 442, 556, 475, 142,  35,  89,
        34, 225, 182, 396,  50,  61, 487,  88, 338, 417, 194,  57, 285,
       187, 159, 184,  81, 207, 339, 326, 531, 505, 492, 299, 507, 316,
       333, 172, 526, 414, 115, 468, 490, 441, 430, 472, 511, 480, 470,
       496, 466, 381, 368, 498, 195, 546, 516, 425, 488, 535, 196, 453,
       321, 152, 390, 166, 247, 454, 464, 499, 460, 251, 481, 315, 307,
       120, 250, 533, 221, 248, 313, 164,  70, 180, 160, 495, 314, 415,
       174, 259, 471, 245,  83,  91, 365, 359, 286,  64, 426, 237, 536,
       397, 476, 503, 491, 517,  55, 170, 175,  62, 539, 541, 540, 484,
       514, 548, 432, 501, 447, 186, 477, 331, 515,  48,  33, 185, 155,
       455, 461, 534, 273, 504, 249,  52, 482, 323, 451, 341, 306, 413,
       292, 463, 223, 497, 525, 427, 231, 519, 183,  32, 553, 271, 276,
       154, 521, 502, 402, 158, 473, 465, 561, 328, 558, 562, 424, 512,
       560, 458, 637, 520, 557, 555, 543, 542, 565, 215, 376, 566],
      dtype=int64)

plt.rcParams['figure.figsize'] = [20, 12]
for cinema in unique_cinemas:

    temp = cinemaTicket_data[cinemaTicket_data['cinema_code'] == cinema]
    temp = temp[['Datetime','tickets_sold']]
    temp = temp.groupby(['Datetime']).sum()

    plt.plot(temp.index,temp['tickets_sold'])

# the legend list will be too long
# plt.legend(unique_cinemas)

By getting the total sale at the given day for each cinema, and then plotting it, does not seem like a good idea, especially since because they have varying number of ticket sale.

However, we are able to note that there some form of repeatable patterns, trends or seasonality in the data. Let's use only 10 cinemas and plot it for visualization. Furthermore, later we shall look into take the average sales on a given day, and use that as the dataset moving forward for the cinema tickets dataset.

plt.rcParams['figure.figsize'] = [20, 12]

for cinema in unique_cinemas[:10]:

    temp = cinemaTicket_data[cinemaTicket_data['cinema_code'] == cinema]
    temp = temp[['Datetime','tickets_sold']]
    temp = temp.groupby(['Datetime']).sum()

    plt.plot(temp.index,temp['tickets_sold'],label=cinema)
    plt.legend()

Using only 10 cinemas, the patterns are now more obvious.

ticketSales_data = cinemaTicket_data[['Datetime', 'tickets_sold']]
ticketSales_data = ticketSales_data.groupby('Datetime').mean()
ticketSales_data

ticketSales_data.plot()

It seems that the first two entries are not properly maintained, or it could be outliers. Therefore, we would be dropping the first two entries.

ticketSales_data = ticketSales_data[2:]
ticketSales_data.plot()

Stackoverflow Dataset

This dataset is a little more direct, where each columns signifies the topic that is being asked. For this project, we can look into the questions asked on the site overtime, from 2009 to 2019

mlStackoverflow_data = pd.read_csv('Datasets/MLTollsStackOverflow.csv')

mlStackoverflow_data.head()

mlStackoverflow_data.columns

Index(['month', 'nltk', 'spacy', 'stanford-nlp', 'python', 'r', 'numpy',
       'scipy', 'matlab', 'machine-learning', 'pandas', 'pytorch', 'keras',
       'nlp', 'apache-spark', 'hadoop', 'pyspark', 'python-3.x', 'tensorflow',
       'deep-learning', 'neural-network', 'lstm', 'time-series', 'pillow',
       'rasa', 'opencv', 'pipenv', 'seaborn', 'Dask', 'jupyter', 'AllenNLP',
       'Theano', 'plotly', 'scikit-learn', 'BeautifulSoup', 'scrapy', 'Gensim',
       'FastText', 'Pydot', 'Pybrain', 'Pytil', 'Pygame', 'Colab', 'Shogun',
       'KNIME', 'Apache', 'Gunicorn', 'Pygtk', 'Weka', 'Conda', 'Ray',
       'matlab.1', 'accord.net', 'regression', 'classification', 'correlation',
       'cluster-analysis', 'H2o', 'Mallet', 'Numba', 'Tableau', 'Trifacta',
       'PyArrow', 'Rasterio', 'Orange3', 'PyMC3', 'Opennn', 'Oryx', 'Istio',
       'Venes', 'Plotnine', 'Gluon', 'Plato', 'Sympy', 'Flair',
       'stanford-nlp.1', 'pyqt', 'Nolearn', 'Lasagne', 'OCR',
       'Apache-spark-mlib', 'azure-virtual-machine'],
      dtype='object')

mlStackoverflow_data['month']

0      09-Jan
1      09-Feb
2      09-Mar
3      09-Apr
4      09-May
        ...
127    19-Aug
128    19-Sep
129    19-Oct
130    19-Nov
131    19-Dec
Name: month, Length: 132, dtype: object

Given that the topics of machine learning should have increase over the 10 years, especially with the recent explosion of interest for generative AI. We will see some relevant topics over the years.

mlStackoverflow_data['Datetime'] = pd.to_datetime(mlStackoverflow_data['month'], format='%y-%b')
mlStackoverflow_data.head()

relevant_topics = ['python', 'machine-learning', 'deep-learning', 'time-series', 'regression', 'pytorch', 'tensorflow', 'classification']

plt.rcParams['figure.figsize'] = [7,5]

for topic in relevant_topics:
    plt.plot(mlStackoverflow_data['Datetime'], mlStackoverflow_data[topic])
    plt.scatter(mlStackoverflow_data['Datetime'], mlStackoverflow_data[topic], s=10)
    plt.title(topic)
    plt.show()

All of them, shows different properties in the graph. For example, while pytorch and tensorflow had beginnings around 2016, but they seem to increase a lot when observing the slope.

Multiplicative or Additive - Visual Identification

We will look into some dataset to see if there are multiplicative or additive. We will use the gold dataset, python and deep learning topic in the stackoverflow dataset and the cinema dataset to see if they are additive or multiplicative.

Based on the textbook that this course is based on, the difference between additive and multiplicative datsets are that one is the sum of seasonal component, trend component and residual component, and one is the product of all three. As shown in the equations below:

• Additive:
  $y_{t} = S_{t} + T_{t} + R_{t}$

• Multiplicative:
  $y_{t} = S_{t} \times T_{t} \times R_{t}$

Hence, to truly find out, we can decompose the time series into the components mentioned, seasonality, trend and residual.

However, based on the courses' lectures, it is also mentioned that we can identify based on the dispersion and shape of the differenced dataset, as shown below.

Gold Dataset - Close Price

gold_data['Close'].plot()

gold_data['Diff'] = gold_data['Close'] - gold_data['Close'].shift()
gold_data.plot.scatter(x = 'Close', y = 'Diff')

As observed, regardless or what value x, the y stays very consistent. Therefore, the changes between the price are not steep or drastic. Which represent an additive chart.

Stakeoverflow Dataset - Python

While the chart shows that it is linearly increase over time, when we check for additivity or multiplicity, turns out this is multiplicative, as shown in the scatter plot below. Where there are instance of x with very little change, while on the other end, the x values have higher change.

mlStackoverflow_data['python'].plot()

mlStackoverflow_data['python-diff'] = mlStackoverflow_data['python'] - mlStackoverflow_data['python'].shift()
mlStackoverflow_data.plot.scatter(x='python', y = 'python-diff')

This is a multiplicative dataset because there is an obvious dispersion of y values, as x increases.

mlStackoverflow_data['logged_python'] = np.log(mlStackoverflow_data['python'])
mlStackoverflow_data['logged_python'].plot()

Even after logging, the chart still looks like it might be multiplicative.

mlStackoverflow_data['logged_python_diff'] = mlStackoverflow_data['logged_python'] - mlStackoverflow_data['logged_python'].shift()
mlStackoverflow_data.plot.scatter(x='logged_python', y='logged_python_diff')

However, based on the result, the chart now looks more additive than it is multiplicative as the y values are now more consistent with the increase in x.

Stackoverflow Dataset - Machine Learning

Machine learning dataset was chosen because of its higher increase at the end of the chart. This shows that it is highly likely to be multiplicative.

mlStackoverflow_data['machine-learning'].plot()

mlStackoverflow_data['machine-learning-diff'] = mlStackoverflow_data['machine-learning'] - mlStackoverflow_data['machine-learning'].shift()
mlStackoverflow_data.plot.scatter(x='machine-learning', y='machine-learning-diff')

As guessed, it is a multiplicative chart.

mlStackoverflow_data['logged_ml'] = np.log(mlStackoverflow_data['machine-learning'])
mlStackoverflow_data['logged_ml'].plot()

Now we shall explore the logged chart. Now the charts looks more linear, which leads more to additive.

mlStackoverflow_data['logged_ml_diff'] = mlStackoverflow_data['logged_ml'] - mlStackoverflow_data['logged_ml'].shift()
mlStackoverflow_data.plot.scatter(x='logged_ml', y='logged_ml_diff')

As shown, while there are still areas of higher dispersed y values, it is better than it was without logging.

Cinema Ticket Dataset - Tickets Sold

This dataset was chosen because out of curiously, a strong seasonal dataset, would be additive or multiplicative.

ticketSales_data['tickets_sold'].plot()

ticketSales_data['tickets_sold_diff'] = ticketSales_data['tickets_sold'] - ticketSales_data['tickets_sold'].shift()
ticketSales_data.plot.scatter(x='tickets_sold', y ='tickets_sold_diff')

Based on the scatter plot, it seems like it is multiplicative, let's try logging it.

ticketSales_data['logged_sales'] = np.log(ticketSales_data['tickets_sold'])
ticketSales_data['logged_sales'].plot()

ticketSales_data['logged_sales_diff'] = ticketSales_data['logged_sales'] - ticketSales_data['logged_sales'].shift()
ticketSales_data.plot.scatter(x='logged_sales', y ='logged_sales_diff')

Based on the result, it looks like the multiplicity of has decreased and looks more additive with the values dispersed more consistently has compared to the one before.

Stationarity

See if any of data are stationary?

Visually, upon first glance, we can see that all of the current dataset, is not stationary, as most of them have some form trend, seasonality, varying means and variance. Therefore, let's implement ways to make them stationary and ensure that they are stationary. The dataset that we will be using is gold dataset and machine learning.

gold_data['Close'].plot()

mlStackoverflow_data['machine-learning'].plot()

Random walk model or Simple Differencing

For many cases, just by finding the difference between the current period and previous period, should be enough to make the time-series data to be stationary.

gold_data['Close_Diffrenced'] = gold_data['Close'] - gold_data['Close'].shift()
gold_data['Close_Diffrenced'].plot()

mlStackoverflow_data['machine-learning_differenced'] = mlStackoverflow_data['machine-learning'] - mlStackoverflow_data['machine-learning'].shift()
mlStackoverflow_data['machine-learning_differenced'].plot()

While we can visually see that both the graph has no more trend and has become more stationary than before, there are still some things to note. For the gold data, it is basically the ideal stationary data outcome, with some areas of outliers, but mostly, it is have a very steady mean and variance. However, even though we have use this method, the stackoverflow questions dataset still as an increase of variance overtime. Let's use some test to see if it is stationary or not. If not, we shall apply more differencing.

KPSS and ADF Test

We will be using the KPSS and ADF test to see if both of the above are stationary data.

from statsmodels.tsa.stattools import kpss

# Referred the below for the function and the use of kpss
# https://www.machinelearningplus.com/time-series/kpss-test-for-stationarity/

def kpss_test(series):
    statistic, p_value, n_lags, critical_values = kpss(series)
    print('KPSS Stat: ', statistic)
    print('p-value: ', p_value)
    print('Number of lags:', n_lags)
    print('Critial Values:')

    for key, value in critical_values.items():
        print(f'   {key} : {value}')
    print(f'Result: The series is {"not " if p_value < 0.05 else ""}stationary')

print('Gold Data-------------------')
print('=====Before Differencing:')
kpss_test(gold_data['Close'])
print()
print('=====After Differencing:')
kpss_test(gold_data['Close_Diffrenced'][1:])

Gold Data-------------------
=====Before Differencing:
KPSS Stat:  133.7235404045421
p-value:  0.01
Number of lags: 518
Critial Values:
   10% : 0.347
   5% : 0.463
   2.5% : 0.574
   1% : 0.739
Result: The series is not stationary

=====After Differencing:
KPSS Stat:  0.06604071713292024
p-value:  0.1
Number of lags: 44
Critial Values:
   10% : 0.347
   5% : 0.463
   2.5% : 0.574
   1% : 0.739
Result: The series is stationary

print('StackOverFlow ML Data-------------------')
print('=====Before Differencing:')
kpss_test(mlStackoverflow_data['machine-learning'])
print()
print('=====After Differencing:')
kpss_test(mlStackoverflow_data['machine-learning_differenced'][1:])

StackOverFlow ML Data-------------------
=====Before Differencing:
KPSS Stat:  1.8470519859412013
p-value:  0.01
Number of lags: 6
Critial Values:
   10% : 0.347
   5% : 0.463
   2.5% : 0.574
   1% : 0.739
Result: The series is not stationary

=====After Differencing:
KPSS Stat:  0.29869005236660856
p-value:  0.1
Number of lags: 10
Critial Values:
   10% : 0.347
   5% : 0.463
   2.5% : 0.574
   1% : 0.739
Result: The series is stationary

According to the KPSS test, even after the simple differencing for the the questions in stackoverflow, is enough to be stationary. Let's see for the adfuller test.

For the adfuller test, the closer the p-value it is to 0, the higher the likelihood for us to assume that there is no unit root in the time-series and that it is stationary. Assuming we were to use p-values of 0.05.

from statsmodels.tsa.stattools import adfuller

def adftest(series):
    res = adfuller(series)
    print('AdfTest Stat: ', res[0])
    print('p-value: ', res[1])
    print('Number of lags:', res[2])
    print('Number of observation:', res[3])
    print('Critial Values:')

    for key, value in res[4].items():
        print(f'   {key} : {value}')

    if res[1] < 0.05:
        print('The series is stationary')
    else:
        print('The series is not stationary')

adftest(mlStackoverflow_data['machine-learning'])

AdfTest Stat:  0.6666371271225812
p-value:  0.9891479891628309
Number of lags: 12
Number of observation: 119
Critial Values:
   1% : -3.4865346059036564
   5% : -2.8861509858476264
   10% : -2.579896092790057
The series is not stationary

adftest(mlStackoverflow_data['machine-learning_differenced'][1:])

AdfTest Stat:  -2.054928456415778
p-value:  0.2630197887870384
Number of lags: 11
Number of observation: 119
Critial Values:
   1% : -3.4865346059036564
   5% : -2.8861509858476264
   10% : -2.579896092790057
The series is not stationary

Using the adfuller test, the simple differencing was not enough. While the p-values have dropped from 0.98 to 0.26, we can still try to make the dataset more stationary.

Second Order Differencing

firstDifference = mlStackoverflow_data['machine-learning'] - mlStackoverflow_data['machine-learning'].shift()

firstDifference

0        NaN
1       -4.0
2       -1.0
3        7.0
4       -3.0
       ...
127    -37.0
128    -45.0
129    212.0
130    168.0
131     65.0
Name: machine-learning, Length: 132, dtype: float64

secondDifference = mlStackoverflow_data['machine-learning'].shift() - mlStackoverflow_data['machine-learning'].shift(2)
secondDifference

0        NaN
1        NaN
2       -4.0
3       -1.0
4        7.0
       ...
127     80.0
128    -37.0
129    -45.0
130    212.0
131    168.0
Name: machine-learning, Length: 132, dtype: float64

secondOrderDifference = firstDifference - secondDifference
secondOrderDifference

0        NaN
1        NaN
2        3.0
3        8.0
4      -10.0
       ...
127   -117.0
128     -8.0
129    257.0
130    -44.0
131   -103.0
Name: machine-learning, Length: 132, dtype: float64

plt.plot(firstDifference)
plt.plot(secondOrderDifference)
plt.show()

adftest(secondOrderDifference[2:])

AdfTest Stat:  -6.611299198025336
p-value:  6.367085815585583e-09
Number of lags: 10
Number of observation: 119
Critial Values:
   1% : -3.4865346059036564
   5% : -2.8861509858476264
   10% : -2.579896092790057
The series is stationary

We can see that visually, the difference between the simple difference and second order differencing, the second order differencing looked like the variance increased as we do see higher highs and lower low in the graph as compared to the simple differencing. The orange graph is the second order differencing. However, based on the adfuller test, it is stationary.

Seasonal Differencing

Seasonal differencing is based on the formula below, which is subtract by the current value with a value in the past $m$ observations, called lag. The higher the lag, the further in the past you subtract. This is one way to remove seasonality.

$y = y_{t} - y_{m}$

However, we will need to loop through some values to find this. Another method of seasonal differencing is to understand a season timeframe, then get the mean of each season, and apply subtraction according to respective season's mean.

lags = [3,5,10,20,50]

plt.rcParams['figure.figsize'] = [15,7]

for lag in lags:
    temp = mlStackoverflow_data['machine-learning'] - mlStackoverflow_data['machine-learning'].shift(lag)
    plt.plot(temp, label=str(lag) + " lag")

    print('Test for lag ' + str(lag) + "  =====================================")
    kpss_test(temp[lag:])
    print()
    adftest(temp[lag:])
    print()

plt.legend()
plt.show()

Test for lag 3  =====================================
KPSS Stat:  0.2898885946621235
p-value:  0.1
Number of lags: 4
Critial Values:
   10% : 0.347
   5% : 0.463
   2.5% : 0.574
   1% : 0.739
Result: The series is stationary

AdfTest Stat:  -2.25147303187454
p-value:  0.18809655506315798
Number of lags: 13
Number of observation: 115
Critial Values:
   1% : -3.4885349695076844
   5% : -2.887019521656941
   10% : -2.5803597920604915
The series is not stationary

Test for lag 5  =====================================
KPSS Stat:  0.27908581626858614
p-value:  0.1
Number of lags: 4
Critial Values:
   10% : 0.347
   5% : 0.463
   2.5% : 0.574
   1% : 0.739
Result: The series is stationary

AdfTest Stat:  -2.1942366663365926
p-value:  0.20834386047176706
Number of lags: 12
Number of observation: 114
Critial Values:
   1% : -3.489057523907491
   5% : -2.887246327182993
   10% : -2.5804808802708528
The series is not stationary

Test for lag 10  =====================================
KPSS Stat:  0.35328244736991793
p-value:  0.09729204854744916
Number of lags: 5
Critial Values:
   10% : 0.347
   5% : 0.463
   2.5% : 0.574
   1% : 0.739
Result: The series is stationary

AdfTest Stat:  -2.4158493770328726
p-value:  0.13731291632025688
Number of lags: 12
Number of observation: 109
Critial Values:
   1% : -3.49181775886872
   5% : -2.8884437992971588
   10% : -2.5811201893779985
The series is not stationary

Test for lag 20  =====================================
KPSS Stat:  0.5230701441606662
p-value:  0.0364706882521022
Number of lags: 5
Critial Values:
   10% : 0.347
   5% : 0.463
   2.5% : 0.574
   1% : 0.739
Result: The series is not stationary

AdfTest Stat:  -2.157009917873902
p-value:  0.2222165170167978
Number of lags: 12
Number of observation: 99
Critial Values:
   1% : -3.498198082189098
   5% : -2.891208211860468
   10% : -2.5825959973472097
The series is not stationary

Test for lag 50  =====================================
KPSS Stat:  1.3102598299838486
p-value:  0.01
Number of lags: 5
Critial Values:
   10% : 0.347
   5% : 0.463
   2.5% : 0.574
   1% : 0.739
Result: The series is not stationary

AdfTest Stat:  -0.5612811021960334
p-value:  0.8794931265993677
Number of lags: 12
Number of observation: 69
Critial Values:
   1% : -3.528889992207215
   5% : -2.9044395987933362
   10% : -2.589655654274312
The series is not stationary

Surprisingly, the further you lag, does not mean that the data will become more stationary. As shown with the kpss and adfuller test, we can see that even using difference of lag 3, adfuller test already did not consider the time series as stationary. The KPSS result shows that from lag 20 onwards, the data is no longer stationary. We can also observed this because visually as the red and purple graph slowly becoming a upward trending graph.

Log and log differences

Let's see how does logging a time-series or getting the log differences, can effect the stationarity of a time-series. We will continue using the stackoverflow dataset on machine learning.

loggedMLtopic = np.log(mlStackoverflow_data['machine-learning'])
loggedMLtopic.plot()

It has made it into a trending time series, so now we can try apply simple differencing

differencedLoggedMLtopic = loggedMLtopic - loggedMLtopic.shift()
differencedLoggedMLtopic.plot()

kpss_test(differencedLoggedMLtopic[1:])

KPSS Stat:  0.28055757025438893
p-value:  0.1
Number of lags: 55
Critial Values:
   10% : 0.347
   5% : 0.463
   2.5% : 0.574
   1% : 0.739
Result: The series is stationary

adftest(differencedLoggedMLtopic[1:])

AdfTest Stat:  -13.691197822513615
p-value:  1.342938733545792e-25
Number of lags: 1
Number of observation: 129
Critial Values:
   1% : -3.482087964046026
   5% : -2.8842185101614626
   10% : -2.578864381347275
The series is stationary

Based on the both the test and visually, we have able to make the machine learning topic time series to be stationary.

Seasonality

Seasonality are data that has some form of repeating pattern and precipitability at a certain time frame.

Based on the current set of data we have, we can clearly see that the cinema dataset has repeatable pattern, very noticeable peak and trough. Therefore, we will look into this dataset for the seasonality adjustments.

Cinema Tickets

ticketSales_data.index.weekday

Int64Index([2, 3, 4, 5, 6, 0, 1, 2, 3, 4,
            ...
            4, 5, 6, 0, 1, 2, 3, 4, 5, 6],
           dtype='int64', name='Datetime', length=232)

ticketSales_data.index.weekday.unique()

Int64Index([2, 3, 4, 5, 6, 0, 1], dtype='int64', name='Datetime')

ticketSales_data = ticketSales_data.assign(day=ticketSales_data.index.weekday)
ticketSales_data

# Because the timeframe in the dataset is not a full calendar year.
# Get the list of months that are involved
days = ticketSales_data.index.weekday.unique()

# Initializing season count and sum
seasonSum = [0] * len(days)
seasonCount = [0] * len(days)

for rowCount in range(len(ticketSales_data)):

    currentValue, currentDay = ticketSales_data.iloc[rowCount,[0,1]]

    seasonSum[int(currentDay)] += currentValue
    seasonCount[int(currentDay)] += 1

print(seasonCount)
print(seasonSum)

[32, 33, 32, 33, 34, 34, 34]
[2431.9228952621716, 8661.859799267784, 3712.1669864735404, 4930.762704784601, 5691.3748326994255, 3488.7014251378805, 2745.6502726189774]

# Season Mean
seasonMean = [0] * len(days)

for count in range(len(seasonCount)):
    seasonMean[count] = seasonSum[count] / seasonCount[count]

seasonMean

[75.99759047694286,
 262.48059997781166,
 116.00521832729814,
 149.4170516601394,
 167.39337743233605,
 102.60886544523179,
 80.7544197829111]

dataMean = np.mean(ticketSales_data['tickets_sold'].values)
dataMean

136.47602981139818

seasonOffsets = [x - dataMean for x in seasonMean]
seasonOffsets

[-60.47843933445532,
 126.00457016641349,
 -20.47081148410004,
 12.94102184874123,
 30.917347620937875,
 -33.86716436616639,
 -55.72161002848708]

adjustedTicketsSold = []
for count in range(len(ticketSales_data)):

    currentValue, currentDay = ticketSales_data.iloc[count,[0,1]]
    adjustedTicketsSold.append(currentValue - seasonOffsets[int(currentDay)])

adjustedTicketsSold

[146.12081148410005,
 165.73430282658344,
 144.54366932821466,
 148.0593565583586,
 150.55356595135208,
 ...
 91.05451518780374,
 100.71249927801934,
 95.90721378257089,
 109.29834181226921,
 105.61634687059234]

ticketSales_data = ticketSales_data.assign(adjustedTicketSale = adjustedTicketsSold)
plt.rcParams['figure.figsize'] = [12, 5]
ticketSales_data.iloc[:, [0,2]].plot()

As we can observe, the orange plot is the adjusted ticket sales. The adjustment works because we can see that the peaks are now lower, while the trough are now higher generally. However, we are still able to note that there are still peaks and troughs even after the adjustment.

secondAdjustmentCount = [0] * len(days)
secondAdjustmentSum = [0] * len(days)
secondAdjustmentMean = [0] * len(days)
secondAdjustmentValue = []

for count in range(len(ticketSales_data)):
    currentDay, currentValue = ticketSales_data.iloc[count, [1,2]]
    secondAdjustmentCount[int(currentDay)] += 1
    secondAdjustmentSum[int(currentDay)] += currentValue

print("Season Count:", secondAdjustmentCount)
print("Season Sum:", secondAdjustmentSum)

for count in range(len(secondAdjustmentSum)):
    secondAdjustmentMean[count] = secondAdjustmentSum[count] / secondAdjustmentCount[count]
print("Season Mean:", secondAdjustmentMean)

secondMean = np.mean(ticketSales_data['adjustedTicketSale'].values)
print("Adjusted Ticket Sale Mean:", secondMean)

secondAdjustmentOffsets = [(x - secondMean) for x in secondAdjustmentMean]
print(secondAdjustmentOffsets)

secondAdjustedTickets = []
for count in range(len(ticketSales_data)):
    currentValue, currentDay = ticketSales_data.iloc[count, [2,1]]
    secondAdjustedTickets.append(currentValue - secondAdjustmentOffsets[int(currentDay)])

ticketSales_data = ticketSales_data.assign(second_adjusted_ticket_sales = secondAdjustedTickets)
plt.rcParams['figure.figsize'] = [12, 5]
ticketSales_data.iloc[:, [2,3]].plot()

It is surprising to see that after the first adjustment, we are no longer able to remove more variances, lower the peaks and heighten the troughs, as the offsets list and the mean of the adjusted tickets sold column, are so close to each other.

weeks = ticketSales_data.index.week.unique()
weeks

Int64Index([11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,
            28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43,
            44],
           dtype='int64', name='Datetime')

ticketSales_data = ticketSales_data.assign(week=ticketSales_data.index.isocalendar().week)
ticketSales_data

weekAdjustmentCount = [0] * len(weeks)
weekAdjustmentSum = [0] * len(weeks)
weekAdjustmentMean = [0] * len(weeks)
weekAdjustmentValue = []

for count in range(len(ticketSales_data)):
    currentWeek, currentValue = ticketSales_data.iloc[count, [4,0]]
    weekAdjustmentCount[int(currentWeek)-11] += 1
    weekAdjustmentSum[int(currentWeek)-11] += currentValue

print("Season Count:", weekAdjustmentCount)
print("Season Sum:", weekAdjustmentSum)

for count in range(len(weekAdjustmentSum)):
    weekAdjustmentMean[count] = weekAdjustmentSum[count] / weekAdjustmentCount[count]
print("Season Mean:", weekAdjustmentMean)

weekMean = np.mean(ticketSales_data['adjustedTicketSale'].values)
print("Adjusted Ticket Sale Mean:", secondMean)

weekAdjustmentOffsets = [(x - weekMean) for x in weekAdjustmentMean]
print(weekAdjustmentOffsets)

weekAdjustedTickets = []
for count in range(len(ticketSales_data)):
    currentWeek, currentValue = ticketSales_data.iloc[count, [4,0]]
    weekAdjustedTickets.append(currentValue - weekAdjustmentOffsets[int(currentWeek)-11])

ticketSales_data = ticketSales_data.assign(week_adjusted_ticket_sales = weekAdjustedTickets)
plt.rcParams['figure.figsize'] = [12, 5]
ticketSales_data[['tickets_sold','week_adjusted_ticket_sales']].plot()

plt.rcParams['figure.figsize'] = [12, 5]

ticketSales_data[['tickets_sold','week_adjusted_ticket_sales','adjustedTicketSale']].plot()

As it can be seen, the seasonality was more removed when we are using the day of the week trying adjust according to the mean of each day, rather than the mean of each week. The week adjusted ticket sales seem to more stationary, but the peaks and trough, while decreased, they are still very prominent. Hence, seasonality is still there.

USDCUSDT Tradecount

We can also explore on the USDCUSDT tradecount column which seem to look like that there are seasonality. We will applied the same steps as above.

for no, column in enumerate(usdcusdt_data.columns):
    print(str(no) + " " + column)

0 unix
1 date
2 symbol
3 open
4 high
5 low
6 close
7 Volume USDC
8 Volume USDT
9 tradecount

usdcusdt_data.iloc[2,1].weekday()

3

usdcusdtDayAdjustmentCount = [0] * 31
usdcusdtDayWeekAdjustmentCount = [0] * 7
usdusdtMonthAdjustmentCount = [0] * 12
usdcusdtDayAdjustmentSum = [0] * 31
usdcusdtDayWeekAdjustmentSum = [0] * 7
usdcusdtMonthAdjustmentSum = [0] * 12
usdcusdtDayAdjustmentMean = [0] * 31
usdcusdtDayWeekAdjustmentMean = [0] * 7
usdcusdtMonthAdjustmentMean = [0] * 12
usdcusdtCloseMean = usdcusdt_data['tradecount'].mean()

for rowCount in range(len(usdcusdt_data)):

    usdcusdtDayAdjustmentCount[usdcusdt_data.iloc[rowCount, 1].day-1] += 1
    usdcusdtDayAdjustmentSum[usdcusdt_data.iloc[rowCount, 1].day-1] += usdcusdt_data.iloc[rowCount, 9]

    usdcusdtDayWeekAdjustmentCount[usdcusdt_data.iloc[rowCount, 1].weekday()] += 1
    usdcusdtDayWeekAdjustmentSum[usdcusdt_data.iloc[rowCount, 1].weekday()] += usdcusdt_data.iloc[rowCount, 9]

    usdusdtMonthAdjustmentCount[usdcusdt_data.iloc[rowCount, 1].month-1] += 1
    usdcusdtMonthAdjustmentSum[usdcusdt_data.iloc[rowCount, 1].month-1] += usdcusdt_data.iloc[rowCount, 9]

for count in range(len(usdcusdtDayAdjustmentSum)):
    usdcusdtDayAdjustmentMean[count] = usdcusdtDayAdjustmentSum[count] / usdcusdtDayAdjustmentCount[count]

for count in range(len(usdcusdtDayWeekAdjustmentSum)):
    usdcusdtDayWeekAdjustmentMean[count] = usdcusdtDayWeekAdjustmentSum[count] / usdcusdtDayWeekAdjustmentCount[count]

for count in range(len(usdcusdtMonthAdjustmentSum)):
    usdcusdtMonthAdjustmentMean[count] = usdcusdtMonthAdjustmentSum[count] / usdusdtMonthAdjustmentCount[count]

dayAdjustmentOffset = [(x - usdcusdtCloseMean) for x in usdcusdtDayAdjustmentMean]
dayWeekAdjustmentOffset = [(x - usdcusdtCloseMean) for x in usdcusdtDayWeekAdjustmentMean]
monthAdjustmentOffset = [(x - usdcusdtCloseMean) for x in usdcusdtMonthAdjustmentMean]

dayAdjustedClose = []
dayWeekAdjustedClose = []
monthAdjustedClose = []

for rowCount in range(len(usdcusdt_data)):
    currentValue, currentDatetime = usdcusdt_data.iloc[rowCount, [9,1]]

    dayAdjustedClose.append(currentValue - dayAdjustmentOffset[currentDatetime.day - 1])
    dayWeekAdjustedClose.append(currentValue - dayWeekAdjustmentOffset[currentDatetime.weekday()])
    monthAdjustedClose.append(currentValue - monthAdjustmentOffset[currentDatetime.month - 1])

usdcusdt_data = usdcusdt_data.assign(Adjusted_tradecount_day = dayAdjustedClose)
usdcusdt_data = usdcusdt_data.assign(Adjusted_tradecount_week = dayWeekAdjustedClose)
usdcusdt_data = usdcusdt_data.assign(Adjusted_tradecount_month = monthAdjustedClose)

adjusted_col = ['Adjusted_tradecount_day', 'Adjusted_tradecount_week', 'Adjusted_tradecount_month']
plt.rcParams['figure.figsize'] = [12, 5]

for column in adjusted_col:
    plt.plot(usdcusdt_data['date'], usdcusdt_data['tradecount'])
    plt.plot(usdcusdt_data['date'], usdcusdt_data[column], alpha=0.8)
    plt.title(column)
    plt.show()

As shown above, only when using the month timeframe, where we can see that the graph moved only a little, and the seasonality of the graph was not removed.

Looking for Seasonality using FFT

We will try to use FFT to identify the seasonality in some of the dataset.

pd.Series(np.fft.fft(ticketSalesData)).plot()

pd.Series(np.abs(np.fft.fft(ticketSalesData))).plot()

pd.Series(np.abs(np.fft.fft(ticketSalesData)[10:len(ticketSalesData)])).plot()

Clearly there are seasonality in this dataset

Gold

pd.Series(np.abs(np.fft.fft(goldData['Close'])[10:len(ticketSalesData)])).plot()

pd.Series(np.abs(np.fft.fft(goldData['Close'])[10:len(ticketSalesData)//3])).plot()

For gold, there seem to be no seasonality in the dataset.

Machine Learning Questions

pd.Series(np.abs(np.fft.fft(mlStackoverflow_data['machine-learning'])[0:len(ticketSalesData)])).plot()

pd.Series(np.abs(np.fft.fft(mlStackoverflow_data['machine-learning'])[100:])).plot()

Honestly cannot tell if the final stretch of the values is considered to indicating that this dataset is a seasonal component

USDCUSDT Tradecount

usdcusdt_data['tradecount']

0         277
1       43366
2       58314
3       67437
4       56204
        ...
1046     9876
1047     5308
1048     4360
1049     1185
1050     1054
Name: tradecount, Length: 1051, dtype: int64

pd.Series(np.abs(np.fft.fft(usdcusdt_data['tradecount']))).plot()

Clearly this is not a seasonal dataset.

Forecasting

Naive Forecasting

Taking the previous period as a prediction. Therefore, we only need to shift the column by one, as we have done for the differencing. Some example shown at the bottom.

def naiveForecasting(series):

    temp = pd.DataFrame(series.rename('observed'))
    temp.insert(1, 'predicted', temp['observed'].shift())

    return temp

mlTopicNaiveForecast = naiveForecasting(mlStackoverflow_data['machine-learning'])
mlTopicNaiveForecast.plot()

ticketSoldNaiveForecast = naiveForecasting(ticketSales_data['tickets_sold'])
ticketSoldNaiveForecast.plot()

goldPriceNaiveForecast = naiveForecasting(gold_data['Close'])
goldPriceNaiveForecast.plot()

Seasonal Forecasting

Define a season and then we predict that next season with the previous season's value. It is similar to naive forecasting above, but with more time gap in between

def seasonalForecasting(series, season):
    temp = pd.DataFrame(series.rename('observed'))
    temp.insert(1, 'predicted', temp['observed'].shift(season))

    return temp

Because each data point represents a month, our season could be quarter, half yearly or yearly.

mlTopicSeasonalYearlyForecast = seasonalForecasting(mlStackoverflow_data['machine-learning'], 12)
mlTopicSeasonalYearlyForecast.plot()

mlTopicSeasonalQuarterlyForecast = seasonalForecasting(mlStackoverflow_data['machine-learning'], 3)
mlTopicSeasonalQuarterlyForecast.plot()

ticketSoldWeeklyForecast = seasonalForecasting(ticketSales_data['tickets_sold'], 7)
ticketSoldWeeklyForecast.plot()

ticketSoldMonthlyForecast = seasonalForecasting(ticketSales_data['tickets_sold'], 30)
ticketSoldMonthlyForecast.plot()

goldPriceDailyForecast = seasonalForecasting(gold_data['Close'], 1440)
goldPriceDailyForecast.plot()

goldPriceWeeklyForecast = seasonalForecasting(gold_data['Close'], 10080)
goldPriceWeeklyForecast.plot()

Average Forecasting

Using the average of the previous n of observation to predict.

def averageForecasting(series, n):
    temp = pd.DataFrame(series.rename('observed'))
    temp.insert(1, 'predicted',
                1/n * (temp['observed'].cumsum().shift() - temp['observed'].cumsum().shift(n+1)))

    return temp

def averageForecastPlot(series, list):
    for n in list:
        temp = averageForecasting(series,n)
        plt.plot(temp['predicted'], label = 'n = ' + str(n))
    plt.plot(temp['observed'], label='observed', linewidth=2, alpha=0.3, linestyle='dashed')
    plt.legend()
    plt.show()

averageForecastPlot(mlStackoverflow_data['machine-learning'],[2,6,12,24])

averageForecastPlot(ticketSales_data['tickets_sold'],[2,7,21,60])

averageForecastPlot(gold_data['Close'],[1440,10080,43200])

Based on the result shown, this method seem to be acting as a smoothing method rather than predictor, especially when it is with a longer time period.

Average Differences Forecasting

Now we will explore using average of the difference between timeframe, and use it as a way to predict

def differenceAverage(series, p):
    temp = pd.DataFrame(series.rename('observed'))
    difference = temp['observed'] - temp['observed'].shift()

    prediction_of_differences = (1/p) * (difference.cumsum().shift(1) - difference.cumsum().shift(p+1))
    temp['predicted']= prediction_of_differences.shift(1) + temp['observed'].shift(1)

    return temp

def differenceAveragePlot(series, list):
    plt.plot(series, label='observed', linewidth=2, alpha=0.5, linestyle='dashed')
    for n in list:
        plt.plot(differenceAverage(series, n)['predicted'], label='predicted, n = ' + str(n))

    plt.legend()
    plt.show()

differenceAveragePlot(mlStackoverflow_data['machine-learning'], [2,6,12,24])

differenceAveragePlot(ticketSales_data['tickets_sold'], [2,6,12,24])

differenceAveragePlot(gold_data['Close'][:100000], [1440,10080,43200])

Naive Differences Forecasting

Make list the difference between the current period and previous period. Then, to predict the next value of the next period, take the difference from the current period and the previous period, and use it to predict the next value.

def naiveDifferenceForecasting(series):
    temp = pd.DataFrame(series.rename('observed'))
    differences = temp['observed'] - temp['observed'].shift()
    predictionsOfDifferences = differences.shift()
    temp['predicted'] = predictionsOfDifferences + temp['observed'].shift(1)

    return temp

naiveDifferenceForecasting(mlStackoverflow_data['machine-learning']).plot()

naiveForecasting(ticketSales_data['tickets_sold']).plot()

naiveDifferenceForecasting(gold_data['Close'])[:300].plot()

Ironic, that using a naive method, of use the previous change of value, to make a prediction, seem to be actually good. While based on the three dataset used, the prediction for the machine learning questions and gold prices, the prediction while it seems that it over-predicts the value, it does seem to at least predict the major movement same as the observed. But that might also be because it seems to be the case visually.

Evaluating the forecasts

Will now implement the ways we can use to evaluate the forecasts. We will be implementing MSE, RMSE and MAE, which are typically used for regression predictions.

Define Functions and Setting Table of Results

def mse(df):
    temp = df['observed'] - df['predicted']
    temp = temp**2
    temp = temp.dropna()

    return temp.sum() / temp.count()

def rmse(df):
    return (mse(df))**(1/2)

def mae(df):
    temp = abs(df['observed'] - df['predicted'])
    temp = temp.dropna()

    return temp.sum() / temp.count()

def evaluateErrors(df):
    return [mse(df), rmse(df), mae(df)]

tableOfResults = pd.DataFrame(columns = ['Data', 'Algorithm', 'MSE', 'RMSE', 'MAE'])
tableOfResults

# Function to fill the Table of Results
def appendingTOR(dataframe, dataset, algorithm, list):
    dataframe.loc[len(dataframe)+1] = {'Data': dataset,'Algorithm': algorithm, 'MSE':list[0],'RMSE':list[1], 'MAE':list[2]}

Naive Forecasting

# Re-setup some of the dataset above
sp500Data = sp500_data[['Datetime', 'Close']]
sp500Data = sp500Data.set_index('Datetime')

ftseData = ftse_data[['Datetime', 'Close']]
ftseData = ftseData.set_index('Datetime')

goldData = gold_data[['Datetime', 'Close']]
goldData = goldData.set_index('Datetime')

temp = usdcusdt_data[['date', 'close', 'tradecount']]
temp = temp.set_index(pd.to_datetime(temp['date']))
usdcusdtDataTradeCount = temp[['tradecount']]
usdcusdtDataClose = temp[['close']]

# Create Naive Forcast Predictions
sp500DataNaiveForecast = naiveForecasting(sp500Data['Close'])
ftseDataNaiveForecast = naiveForecasting(ftseData['Close'])
goldDataNaiveForecast = naiveForecasting(goldData['Close'])
usdcusdtDataCloseNaiveForecast = naiveForecasting(usdcusdtDataClose['close'])
usdcusdtTradeCountNaiveForecast = naiveForecasting(usdcusdtDataTradeCount['tradecount'])
pythonNaiveForecast = naiveForecasting(mlStackoverflow_data['python'])

# Appending to Result Table
naiveForecastingString = 'Naive Forecasting'

appendingTOR(tableOfResults, 'SP500', naiveForecastingString, evaluateErrors(sp500DataNaiveForecast))
appendingTOR(tableOfResults, 'FTSE', naiveForecastingString, evaluateErrors(ftseDataNaiveForecast))
appendingTOR(tableOfResults, 'Gold', naiveForecastingString, evaluateErrors(goldDataNaiveForecast))
appendingTOR(tableOfResults, 'USDCUSDT Close Price', naiveForecastingString, evaluateErrors(usdcusdtDataCloseNaiveForecast))
appendingTOR(tableOfResults, 'USDCUSDT Tradecount', naiveForecastingString, evaluateErrors(usdcusdtTradeCountNaiveForecast))
appendingTOR(tableOfResults, 'Python Questions', naiveForecastingString, evaluateErrors(pythonNaiveForecast))
appendingTOR(tableOfResults, 'Machine Learning Questions', naiveForecastingString, evaluateErrors(mlTopicNaiveForecast))
appendingTOR(tableOfResults, 'Ticket Sales', naiveForecastingString, evaluateErrors(ticketSoldNaiveForecast))

tableOfResults

Seasonal Forecasting

# Create Seasonal Forecasting Predictions
sp500SeasonalForecast60 = seasonalForecasting(sp500Data['Close'], 60)
sp500SeasonalForecast1440 = seasonalForecasting(sp500Data['Close'], 1440)
ftseSeasonalForecast60 = seasonalForecasting(ftseData['Close'], 60)
ftseSeasonalForecast1440 = seasonalForecasting(ftseData['Close'], 1440)
goldSeasonalForecast60 = seasonalForecasting(goldData['Close'], 60)
goldSeasonalForecast1440 = seasonalForecasting(goldData['Close'], 1440)
usdcusdtCloseSeasonalForecast = seasonalForecasting(usdcusdtDataClose['close'], 7)
usdcusdtTCSeasonalForecast = seasonalForecasting(usdcusdtDataTradeCount['tradecount'], 7)
pythonSeasonalForecast = seasonalForecasting(mlStackoverflow_data['python'], 12)

# Appending to Result Table
seasonalForecastingString = 'Seasonal Forecasting'

appendingTOR(tableOfResults, 'SP500', seasonalForecastingString + ' - 60', evaluateErrors(sp500SeasonalForecast60))
appendingTOR(tableOfResults, 'SP500', seasonalForecastingString + ' - 1440', evaluateErrors(sp500SeasonalForecast1440))
appendingTOR(tableOfResults, 'FTSE', seasonalForecastingString + ' - 60', evaluateErrors(ftseSeasonalForecast60) )
appendingTOR(tableOfResults, 'FTSE', seasonalForecastingString + ' - 1440', evaluateErrors(ftseSeasonalForecast1440))
appendingTOR(tableOfResults, 'Gold', seasonalForecastingString + ' - 60', evaluateErrors(goldSeasonalForecast60))
appendingTOR(tableOfResults, 'Gold', seasonalForecastingString + ' - 1440', evaluateErrors(goldSeasonalForecast1440))
appendingTOR(tableOfResults, 'USDCUSDT Close Price', seasonalForecastingString, evaluateErrors(usdcusdtCloseSeasonalForecast))
appendingTOR(tableOfResults, 'USDCUSDT Tradecount', seasonalForecastingString, evaluateErrors(usdcusdtTCSeasonalForecast))
appendingTOR(tableOfResults, 'Python Questions', seasonalForecastingString, evaluateErrors(pythonSeasonalForecast))
appendingTOR(tableOfResults, 'Machine Learning Questions', seasonalForecastingString, evaluateErrors(mlTopicSeasonalYearlyForecast))
appendingTOR(tableOfResults, 'Ticket Sales', seasonalForecastingString, evaluateErrors(ticketSoldWeeklyForecast))

tableOfResults

Average Forecasting

# Create Average Forecasting Predictions

sp500AverageForecast60 = averageForecasting(sp500Data['Close'], 60)
sp500AverageForecast1440 = averageForecasting(sp500Data['Close'], 1440)
ftseAverageForecast60 = averageForecasting(ftseData['Close'], 60)
ftseAverageForecast1440 = averageForecasting(ftseData['Close'], 1440)
goldAverageForecast60 = averageForecasting(goldData['Close'], 60)
goldAverageForecast1440 = averageForecasting(goldData['Close'], 1440)
usdcusdtCloseAverageForecast = averageForecasting(usdcusdtDataClose['close'], 7)
usdcusdtTCAverageForecast = averageForecasting(usdcusdtDataTradeCount['tradecount'], 7)
pythonAverageForecast = averageForecasting(mlStackoverflow_data['python'], 12)
mlTopicAverageForecast = averageForecasting(mlStackoverflow_data['machine-learning'], 12)
ticketSalesAverageForecast = averageForecasting(ticketSales_data['tickets_sold'], 7)

# Appending to Result Table
averageForcastingString = 'Average Forecasting'

appendingTOR(tableOfResults, 'SP500', averageForcastingString + ' - 60', evaluateErrors(sp500AverageForecast60))
appendingTOR(tableOfResults, 'SP500', averageForcastingString + ' - 1440', evaluateErrors(sp500AverageForecast1440))
appendingTOR(tableOfResults, 'FTSE', averageForcastingString + ' - 60', evaluateErrors(ftseAverageForecast60))
appendingTOR(tableOfResults, 'FTSE', averageForcastingString + ' - 1440', evaluateErrors(ftseAverageForecast1440))
appendingTOR(tableOfResults, 'Gold', averageForcastingString + ' - 60', evaluateErrors(goldAverageForecast60))
appendingTOR(tableOfResults, 'Gold', averageForcastingString + ' - 1440', evaluateErrors(goldAverageForecast1440))
appendingTOR(tableOfResults, 'USDCUSDT Close Price', averageForcastingString, evaluateErrors(usdcusdtCloseAverageForecast))
appendingTOR(tableOfResults, 'USDCUSDT Tradecount', averageForcastingString, evaluateErrors(usdcusdtTCAverageForecast))
appendingTOR(tableOfResults, 'Python Questions', averageForcastingString, evaluateErrors(pythonAverageForecast))
appendingTOR(tableOfResults, 'Machine Learning Questions', averageForcastingString, evaluateErrors(mlTopicAverageForecast))
appendingTOR(tableOfResults, 'Ticket Sales', averageForcastingString, evaluateErrors(ticketSalesAverageForecast))

tableOfResults

Average Differences

# Create Average Differences Prediction

sp500AverageDifferenceForecast60 = differenceAverage(sp500Data['Close'], 60)
sp500AverageDifferenceForecast1440 = differenceAverage(sp500Data['Close'], 1440)
ftseAverageDifferenceForecast60 = differenceAverage(ftseData['Close'], 60)
ftseAverageDifferenceForecast1440 = differenceAverage(ftseData['Close'], 1440)
goldAverageDifferenceForecast60 = differenceAverage(goldData['Close'], 60)
goldAverageDifferenceForecast1440 = differenceAverage(goldData['Close'], 1440)
usdcusdtCloseAverageDifferenceForecast = differenceAverage(usdcusdtDataClose['close'], 7)
usdcusdtTCAverageDifferenceForecast = differenceAverage(usdcusdtDataTradeCount['tradecount'], 7)
pythonAverageDifferenceForecast = differenceAverage(mlStackoverflow_data['python'], 12)
mlTopicAverageDifferenceForecast = differenceAverage(mlStackoverflow_data['machine-learning'], 12)
ticketSalesAverageDifferenceForecast = differenceAverage(ticketSales_data['tickets_sold'], 7)

# Appending to Result Table
averageDifferenceForecastingString = 'Average Difference Forecasting'

appendingTOR(tableOfResults, 'SP500', averageDifferenceForecastingString + ' - 60', evaluateErrors(sp500AverageDifferenceForecast60))
appendingTOR(tableOfResults, 'SP500', averageDifferenceForecastingString + ' - 1440', evaluateErrors(sp500AverageDifferenceForecast1440))
appendingTOR(tableOfResults, 'FTSE', averageDifferenceForecastingString + ' - 60', evaluateErrors(ftseAverageDifferenceForecast60))
appendingTOR(tableOfResults, 'FTSE', averageDifferenceForecastingString + ' - 1440', evaluateErrors(ftseAverageDifferenceForecast1440))
appendingTOR(tableOfResults, 'Gold', averageDifferenceForecastingString + ' - 60', evaluateErrors(goldAverageDifferenceForecast60))
appendingTOR(tableOfResults, 'Gold', averageDifferenceForecastingString + ' - 1440', evaluateErrors(goldAverageDifferenceForecast1440))
appendingTOR(tableOfResults, 'USDCUSDT Close Price', averageDifferenceForecastingString, evaluateErrors(usdcusdtCloseAverageDifferenceForecast))
appendingTOR(tableOfResults, 'USDCUSDT Tradecount', averageDifferenceForecastingString, evaluateErrors(usdcusdtTCAverageDifferenceForecast))
appendingTOR(tableOfResults, 'Python Questions', averageDifferenceForecastingString, evaluateErrors(pythonAverageDifferenceForecast))
appendingTOR(tableOfResults, 'Machine Learning Questions', averageDifferenceForecastingString, evaluateErrors(mlTopicAverageDifferenceForecast))
appendingTOR(tableOfResults, 'Ticket Sales', averageDifferenceForecastingString, evaluateErrors(ticketSalesAverageDifferenceForecast))

tableOfResults

Naive Difference

# Create Naive Difference Prediction

sp500NaiveDifferenceForecast = naiveDifferenceForecasting(sp500Data['Close'])
ftseNaiveDifferenceForecast = naiveDifferenceForecasting(ftseData['Close'])
goldNaiveDifferenceForecast = naiveDifferenceForecasting(goldData['Close'])
usdcusdtCloseNaiveDifferenceForecast = naiveDifferenceForecasting(usdcusdtDataClose['close'])
usdcusdtTCNaiveDifferenceForecast = naiveDifferenceForecasting(usdcusdtDataTradeCount['tradecount'])
pythonNaiveDifferenceForecast = naiveDifferenceForecasting(mlStackoverflow_data['python'])
mlTopicNaiveDifferenceForecast = naiveDifferenceForecasting(mlStackoverflow_data['machine-learning'])
ticketSalesNaiveDifferenceForecast = naiveDifferenceForecasting(ticketSales_data['tickets_sold'])

# Appending to Result Table
naiveDifferenceForecastingString = 'Naive Difference Forecasting'

appendingTOR(tableOfResults, 'SP500', naiveDifferenceForecastingString, evaluateErrors(sp500NaiveDifferenceForecast))
appendingTOR(tableOfResults, 'FTSE', naiveDifferenceForecastingString, evaluateErrors(ftseNaiveDifferenceForecast))
appendingTOR(tableOfResults, 'Gold', naiveDifferenceForecastingString, evaluateErrors(goldNaiveDifferenceForecast))
appendingTOR(tableOfResults, 'USDCUSDT Close Price', naiveDifferenceForecastingString, evaluateErrors(usdcusdtCloseNaiveDifferenceForecast))
appendingTOR(tableOfResults, 'USDCUSDT Tradecount', naiveDifferenceForecastingString, evaluateErrors(usdcusdtTCNaiveDifferenceForecast))
appendingTOR(tableOfResults, 'Python Questions', naiveDifferenceForecastingString, evaluateErrors(pythonNaiveDifferenceForecast))
appendingTOR(tableOfResults, 'Machine Learning Questions', naiveDifferenceForecastingString, evaluateErrors(mlTopicNaiveDifferenceForecast))
appendingTOR(tableOfResults, 'Ticket Sales', naiveDifferenceForecastingString, evaluateErrors(ticketSalesNaiveDifferenceForecast))

tableOfResults

Analysis

Because the evaluation metrics does not produce a standardize output, we cannot use the MSE, RMSE, and MAE value directly without context. It needs to be compared within the same dataset used. For example, the evaluations on USDCUSDT dataset are the lowest, but that does not mean that average difference algorithm on usdcusdt is better than naive difference forecasting on Ticket Sales. So let's pick S&P500, Python Questions and Ticket Sales, to compare first and see what observations can we find.

display(tableOfResults[tableOfResults['Data'] == 'SP500'].sort_values('MSE'),
        tableOfResults[tableOfResults['Data'] == 'Python Questions'].sort_values('MSE'),
        tableOfResults[tableOfResults['Data'] == 'Ticket Sales'].sort_values('MSE'),
)

We can see that all of there evaluation value are very different in terms of size and magnitude. Then, currently we can see that naive forecasting performing the best. Let's list all the best algorithm for the given dataset.

for dataset in tableOfResults['Data'].unique():
    display(tableOfResults[tableOfResults['Data'] == dataset].sort_values('MSE').head(2))

We can observe that Naive Forecasting seem to be the best choice at the moment, and the Average Difference in second. Only TicketSales and USDCUSDT Tradecount do not have both of the algorithm. This could be because the ticket sales and usdcusdt datasets, have more prominent repeatable pattern that other forecasting methods work better, while because the rest of the dataset are more random in nature.

Additionally, something to take note is that Naive Forecasting and Average Differences are currently the best algorithm only in the context of technical evaluation of MSE, RMSE and MAE. Because if we were to set trading/investment rules and different way to evaluate performance, the result might be different.

Autoregression Prediction

Before we can use autoregression prediction, first we will need to make the datasets stationary, as well as using autocorrelation and partial autocorrelation.

Choosing Dataset

We have a lot of datasets that we can use. We will use ticket sales (because it consists of more obvious seasonal changes), machine learning questions from stackoverflow (for its multiplicativeness), SP500 and Gold dataset as they are real world financial data. But because the SP500 and Gold data are very large, we will only use 3 months of data.

We will need need to also split the dataset into training and testing sets. We will use 70% as training and 30% training.

Autocorrelation and Partial Autocorrelation

# Importing plot_acf
from statsmodels.graphics.tsaplots import plot_acf, plot_pacf

Gold Data

Making it stationary

stationaryGoldData = goldData
stationaryGoldData['observed'] = goldData['Close'] - goldData['Close'].shift()
stationaryGoldData = stationaryGoldData.dropna().drop('Close', axis=1)
stationaryGoldData = stationaryGoldData[stationaryGoldData.index.dayofweek != 5]
stationaryGoldData = stationaryGoldData[:108000]

kpss_test(stationaryGoldData)
print()
adftest(stationaryGoldData)
stationaryGoldData.plot()

Setting the training and testing set

stationaryGoldDataLength = len(stationaryGoldData)
traniningStationaryGoldDataLength = int(len(stationaryGoldData) * 0.7)
testingStationaryGoldDataLength = stationaryGoldDataLength - traniningStationaryGoldDataLength

print(stationaryGoldDataLength, traniningStationaryGoldDataLength, testingStationaryGoldDataLength)

108000 75600 32400

ACF

plot_acf(stationaryGoldData[:traniningStationaryGoldDataLength],lags=60)

PACF

plot_pacf(stationaryGoldData[:traniningStationaryGoldDataLength], lags = 60)

S&P500

Making it stationary

stationarySP500Data = sp500Data
stationarySP500Data['observed'] = sp500Data['Close'] - sp500Data['Close'].shift()
stationarySP500Data = stationarySP500Data.dropna().drop('Close', axis=1)
stationarySP500Data = stationarySP500Data[stationarySP500Data.index.dayofweek != 5].sort_index()
stationarySP500Data = stationarySP500Data[:36000]

kpss_test(stationarySP500Data)
print()
adftest(stationarySP500Data)
stationarySP500Data.plot()

Setting the training and testing set

stationarySP500DataLength = len(stationarySP500Data)
trainingStationarySP500Data = int(0.7 * stationarySP500DataLength)
testingStationarySP500Data = stationarySP500DataLength - trainingStationarySP500Data

print(stationarySP500DataLength, trainingStationarySP500Data, testingStationarySP500Data)

36000 25200 10800

ACF

plot_acf(stationarySP500Data[:trainingStationarySP500Data], lags=60)
plt.xticks([x for x in range(0,61,2)])

PACF

plot_pacf(stationarySP500Data[:trainingStationarySP500Data], lags=60)
plt.xticks([x for x in range(0,61,2)])

Machine Learning Questions

Because the dataset is multiplicative, we will need to log and get the difference of it.