Walmart-end-to-end-with-Azure-Databricks

The data is extracted in batches using Azure Data Factory and transformed using Spark in Azure Databricks. The transformed data is stored in Synapse Analytics in columnar format. Finally, a dashboard is created using Looker Studio or Power BI, As a result of my project, fresh food has become their top category! This will help with the decision to promote fresh food by placing them in more visible locations and on the website or application. Their most popular fresh food is MorningStar Farms Veggie Burgers, with customers picking up a total of 184 units daily. With this information, the company is able to contact MorningStar Farm's to order more products and discuss pricing options. For more information about data source.Here

Project Overview

Contents

Create Storage Account.

• Azure Blob Storage.
  
• Azure Data Lake Storage Gen2.
  

Create Azure Data Factory.

• Create Linked Services.
  

  • Source Linked Services.
    
  • Sink Linked Services.
    
  • Databricks Linked Services.
    

• Create Dataset.
  

  • Dataset of Azure Blob Storage.
    
  • Dataset of Azure Data Lake Storage Gen2.
    

• Extract data using Azure Data Factory.
  

Create Azure Databricks Services.

• Create Cluster.
  
• Mounting Data Lake Storage.
  
• Using Databricks to transform data.
  
• Run Databricks Notebook using Azure Data Factory.
  

Create Synapse Analytics workspace.

• Create Lake database.
  
• Create Table from data lake.
  
• Load data into Azure Synapse Analytics.
  

Import Azure Synapse Analytics (SQL DW) Data into the Power BI.

• Connect to a data source in Power BI.
  
• Creating Data visualizations.