This is the final part of my series of articles on my dissertation from the University of York. The first article introduced the basic concepts for this study while the second article described the experimentation process and approaches taken. This final article explains the results and evaluation.

Results and Evaluation

The Base Cases

The first base case consisted of just predicting the most common direction, while the second base consisted of a feedforward network which took the closing prices of the FTSE for the last five days.

The data used for training and testing these experiments spanned from 1st January 2013 to 1st January 2018. This translates to 1256 days of closing prices, thus splitting the data 80 / 20 will result in 1004 daily closing prices for training and 252 for testing. The direction of the FTSE index in the testing dataset rose 51.98% of the time, therefore 48.02%. Therefore, the predictive accuracy of the first base case is 51.98%.

As for all feedforward networks, an exercise for finding the most suitable learning rate and number of hidden nodes had to be performed. This consisted of running models with different combinations of parameters. For each combination, the model was run 20 times and the average accuracy was reported.  The graph below shows the results from this exercise for the second base case.

The optimal parameters for the second base case were 9 hidden nodes with a learning rate of 0.0016. These parameters achieved an average of 55.36%.

The results of both base cases were compared with the other experiments.

Hypothesis 1: Incorporating Other Stock Market Indices

Performing the same exercise as before, the optimal parameters for the first hypothesis were 17 hidden nodes and a learning rate of 0.0003, which achieved an average accuracy of 66.51%. Compared to the first and second base cases, an increase of +14.52% and +11.15% accuracy was achieved respectively.

Hypothesis 2: Incorporating Macroeconomic Parameters

The best accuracy was achieved with a learning rate of 0.0011 and 47 hidden nodes, with an average accuracy of 53.45%.   Compared to the first and second base cases, an increase of +1.47% and a decrease of  -0.0191% in accuracy was achieved respectively.

Hypothesis 3: Incorporating Macroeconomic Parameters

The best accuracy was achieved with a learning rate of 0.00001 and 56 hidden nodes, with an average accuracy of 63.47%.   Compared to the first and second base cases, an increase of +11.49% and  +8.11% in accuracy was achieved respectively.

Statistical Significance Tests

A summary of the results can be found in the following table.

 It is clear that the best accuracy was achieved by hypothesis 1, where closing prices from other stock market indices were incorporated with FTSE data.

However, recall the null hypotheses mentioned in the second article:

The method in question makes no significant difference in prediction accuracy when compared to underlying base cases

Using the Kruskal-Wallis and Wilcoxon Signed-rank statistical tests, we could verify that no significant improvement in accuracy was achieved from:

1. The second base case with respect to the first base case.
2. The second hypothesis with respect to the second base case.
3. The third hypothesis with respect to the first hypothesis.

Therefore, the null hypothesis for hypothesis 2 can be accepted, while it can be rejected for hypothesis 1 and 2.

Discussion

The following remarks can be made from the results:

• Incorporating macroeconomic parameters into models does not help increase predictive performance.
  • The worse performing model was hypothesis 2. Hypothesis 3, which just like hypothesis 2 incorporated macroeconomic parameters, also did not achieve any significant increase in accuracy over hypothesis 1.
  • While these two observations seem to suggest that incorporating macroeconomic parameters into models does not help increase predictive performance, there could be an alternative reason for these results.
  •  While the macroeconomic parameters chosen are regarded as important parameters for describing the state of an economy, the possibility remains that these parameters were not the most suitable macroeconomic parameters for stock market prediction. This study, given more time, could have performed a more informed exercise for determining the macroeconomic parameters with the highest correlation with stock market indices.
  • Another possibility is that the chosen ensemble approach, with two feedforward network stages, was not ideal for incorporating such data.
  • There is also the possibility that the way the macroeconomic parameters were preprocessed and fed into the meta-model as lagged quarterly data was not ideal.
  • Clearly, there is plenty of room for improvement with regards to incorporating macroeconomic data.
• Incorporating closing stock prices from global markets was a success, with hypothesis 1 achieving the best accuracy and a statistically significant improvement over the base cases. 
  • Interestingly, the ensemble approach in hypothesis 3 did not outperform the single staged feedforward model, which suggests that less complex architectures do perform best.
  • This also supports the earlier argument about the chosen ensemble approach not being suitable for incorporating macroeconomic data in such a way.
  • While the chosen ensemble approach may not have been the ideal choice of architecture, given more time, it would have been interesting to compare hypothesis 1 with a similar ensemble that naively only takes the most common prediction from the pre-trained models.
• While this study showed that external data does not help improve the prediction accuracy of the model, it also showed that it is not always obvious which data is best, and thus a lot more effort must be made into finding the most influential data for improving a model. 

Final Remarks

I’ve really enjoyed working on this dissertation. It was a fruitful experience that has exposed me to machine learning with TensorFlow, basic economics, stock markets, financial market theory and some level of statistics. I was rather surprised to see that macroeconomic parameters did not make an improvement in accuracy performance, and wish I had more than four months to try alternative approaches.

If you’ve got an idea of how to improve this study, or interested in building upon what I’ve done, feel free to drop me a message. Feedback is also very welcome! I hope you enjoyed reading these articles. Thank you for reading!