Idiosyncratic volatility estimation on 50 years of stock market data

Financial markets require a great deal of decision making from the investors and market makers. One metric that can help ease the process of decision making is investment risk which can be measured in two parts; systematic risk and idiosyncratic risk. Clear understanding of the volatilities in each risk component can be a powerful signal in recognizing the right assets to maximize the investment returns. In this project, we focus on the idiosyncratic volatility values and provide an easy to use source code implementation that can pave the road for future studies on the relation between the idiosyncratic volatility and the chosen asset return values. Using our implemented source code, we pre-calculated the idiosyncratic volatility values for 31,198 members of NYSE, Amex and Nasdaq markets for the trade dates occurring between January 1963 and December 2019, and release this dataset along with our implemented source code. Additionally, we consider the application of machine learning techniques in predicting the idiosyncratic volatility values using the raw trade data to explore a data extension option for the future market trade records that have not yet occurred. We offer a deep learning based regression model and compare it with traditional tree-based methods on a small subset of our per-calculated idiosyncratic volatility dataset. Our analytical results show that the performance of the deep learning techniques is much more robust in comparison to that of the traditional tree-based baselines. However, more work needs to be done to be able to use a machine learning based model to reliably predict the idiosyncratic volatility values using the raw trade records.