This project aims to develop a comprehensive big data system from start to finish, encompassing key components such as data extraction, transformation, and loading (ETL), along with database management and computational tasks using Data Science methodologies. Technologies employed will include Python (utilizing libraries such as NumPy, Pandas, Scipy, Scikit Learn, and Dask), Docker, MongoDB, and Airflow. The primary focus will be on utilizing historical data sourced from Alpha Vintage to analyze stock price trends, presenting easily interpretable visualizations of observed patterns. Additionally, anomaly detection algorithms will be applied to identify irregularities in stock prices.
Alpha Vantage provides financial data APIs catering to real-time and historical stock, forex, cryptocurrency, and technical indicators. Their API offers developers a comprehensive toolkit for accessing financial data and constructing applications. For this project, the focus will be on utilizing intraday data from the S&P500 index provided by Alpha Vantage.