Provide data consolidation process

[tomkerkhove]

Provide data consolidation process that is represented by an Azure Data Factory pipeline.

Process will:

• Get all stored user information from our SQL database
• Process StackExchange data set stored in Azure Data Lake Store, via U-SQL script ran by Azure Data Lake Analytics
• Consolidate data from above provides in Azure Storage and compress it (Move & compress consolidated U-SQL output #20 )
• Publish event on Event Grid with following information: (Publish Event Grid event when data is consolidated #19)
  - Type: DataConsolidationCompleted
  - Data: EmailAddress, First Name, Last Name

Input parameters

• Email address of customer

Endpoint in API Management

• Verb: Post
• Uri: /users/{email}/data-consolidation/archive

[tomkerkhove]

Given this is not available in Data Factory yet, the trigger will go through API Management

[tomkerkhove]

This is done with known limitations:

• Output from ADLA is not copied to Blob storage yet
• No event grid event since this is not there yet