Event driven Architecture with S3, SNS, SQS, and Lambda. Uses Events to decouple an application's components.
- loosely coupled applications
- add new features without changing existing applications
- Scale and fail components independently.
I. Create the s3 bucket
![Screen Shot 2023-12-06 at 8 54 44 PM](https://private-user-images.githubusercontent.com/74584964/288434580-9cc01c6a-4a92-414b-901c-9db2c5245de2.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjEyNjQ4NzcsIm5iZiI6MTcyMTI2NDU3NywicGF0aCI6Ii83NDU4NDk2NC8yODg0MzQ1ODAtOWNjMDFjNmEtNGE5Mi00MTRiLTkwMWMtOWRiMmM1MjQ1ZGUyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzE4VDAxMDI1N1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWVjZmQwY2RhNjRkYjQxN2Y5NzlhMzk1ZTk1MDY5MGEyOGE2YjAzNTkxM2M5M2ViZWM0YjkxYzQyNDUzZDJiYjMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.QfS_OCciZ1mCz6EIGdD1y1lCPCNiVQgOPaBTpBV1oAo)
II. Create an SNS topic
- create the topic and change the access policy: copy your account number, s3 bucket name, and SQS queues.
![Screen Shot 2023-12-06 at 3 42 37 PM](https://private-user-images.githubusercontent.com/74584964/288433280-bfcbaa39-aa10-447d-9376-c58fbd17d13c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjEyNjQ4NzcsIm5iZiI6MTcyMTI2NDU3NywicGF0aCI6Ii83NDU4NDk2NC8yODg0MzMyODAtYmZjYmFhMzktYWExMC00NDdkLTkzNzYtYzU4ZmJkMTdkMTNjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzE4VDAxMDI1N1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTAyNmIwNjcyMzIwNTUwN2Y0ZjViY2EyYmQwYmIxNTRiNGY3MDFmNzRmM2JlYTlhMzNiMTVhYmUzMzgyN2UzZWMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.DUXc37ncTSs9PVX1VD0ZkzVyksyaDRnSH_B1iX8r3yM)
{
"Version": "2008-10-17",
"Id": "__default_policy_ID",
"Statement": [
{
"Sid": "__default_statement_ID",
"Effect": "Allow",
"Principal": {
"Service": "s3.amazonaws.com"
},
"Action": "SNS:Publish",
"Resource": "arn:aws:sns:<region>:<accountId>:<snsTopicName>",
"Condition": {
"StringEquals": {
"aws:SourceAccount": "<accountId>"
},
"ArnLike": {
"aws:SourceArn": "arn:aws:s3:*:*:<s3BucketName>"
}
}
},
{
"Sid": "sqs_statement",
"Effect": "Allow",
"Principal": {
"Service": "sqs.amazonaws.com"
},
"Action": "sns:Subscribe",
"Resource": "arn:aws:sns:<region>:<accountId>:<snsTopicName>",
"Condition": {
"ArnEquals": {
"aws:SourceArn": [
"arn:aws:sqs:<region>:<accountId>:<sqsQueueName>",
"arn:aws:sqs:<region>:<accountId>:<sqsQueueName>"
]
}
}
}
]
}
III. Create SQS Quere
![Screen Shot 2023-12-06 at 3 53 08 PM](https://private-user-images.githubusercontent.com/74584964/288433535-5b4eb5a8-4072-459a-a062-1e6d1b59c053.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjEyNjQ4NzcsIm5iZiI6MTcyMTI2NDU3NywicGF0aCI6Ii83NDU4NDk2NC8yODg0MzM1MzUtNWI0ZWI1YTgtNDA3Mi00NTlhLWEwNjItMWU2ZDFiNTljMDUzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzE4VDAxMDI1N1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTliNjI4Njk3NjNkZTY1ODlkM2RiYTg2YzVlMTc2NTI1ZTg1Yzg3NTg4ZGFhOWJjMDhlNDNhOWZlZjcxY2ZmYzkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.2Ynm9yRCmF-Yr8VuFMGhDsleAFCp_GR4yQtPObCzA38)
-
event stream: create the queue and change the access policy:
{ "Version": "2008-10-17", "Id": "__default_policy_ID", "Statement": [ { "Sid": "Stmt1234", "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": [ "sqs:ReceiveMessage", "sqs:sendMessage" ], "Resource": "arn:aws:sqs:<region>:<accountId>:<sqsQueueName>", "Condition": { "ArnEquals": { "aws:SourceArn": "arn:aws:lambda:<region>:<accountId>:<lambdaName>" } } }, { "Sid": "Stmt12345", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "sqs:SendMessage", "Resource": "arn:aws:sqs:<region>:<accountId>:<sqsQueueName>", "Condition": { "ArnLike": { "aws:SourceArn": "arn:aws:sns:<region>:<accountId>:<snsTopicName>" } } } ]
IV. Create the Lambda Functions
-
create two lambda policies to allow access to SQS and CloudWatch
{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "sqs:DeleteMessage", "logs:CreateLogStream", "sqs:ReceiveMessage", "sqs:GetQueueAttributes", "logs:PutLogEvents" ], "Resource": [ "arn:aws:sqs:<region>:<accountId>:q1", "arn:aws:logs:<region>:<accountId>:log-group:/aws/lambda/<lambdaName>:*" ] }, { "Sid": "VisualEditor1", "Effect": "Allow", "Action": [ "sqs:ReceiveMessage", "logs:CreateLogGroup" ], "Resource": [ "arn:aws:logs:<region>:<accountId>:*", "arn:aws:sqs:<region>:<accountId>:<sqsQueueName>" ] } ] }
-
Create two roles for each lambda function and then attach each policy to the corresponding lambda role.
![Screen Shot 2023-12-06 at 4 29 11 PM](https://private-user-images.githubusercontent.com/74584964/288434098-a6e149d1-7e77-4935-9152-ff114d7c5234.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjEyNjQ4NzcsIm5iZiI6MTcyMTI2NDU3NywicGF0aCI6Ii83NDU4NDk2NC8yODg0MzQwOTgtYTZlMTQ5ZDEtN2U3Ny00OTM1LTkxNTItZmYxMTRkN2M1MjM0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzE4VDAxMDI1N1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTcxZWJjZTg0ODExMGE4Y2NlYjQ1OGUwZDY4NTgxNDFhNzE5ZjA3NTBhOWNlZTQwMWRmMDQ3ZmQwMDgxYTE0NjEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Ooo3nIl0msohlqeYYWBm7g-AmeeWi90niz-2PsFb5fI)
- Create the two lambda functions, and let's print the data
![Screen Shot 2023-12-06 at 8 48 06 PM](https://private-user-images.githubusercontent.com/74584964/288434296-a37e1d04-8196-4c4b-95e8-a62a153bd526.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjEyNjQ4NzcsIm5iZiI6MTcyMTI2NDU3NywicGF0aCI6Ii83NDU4NDk2NC8yODg0MzQyOTYtYTM3ZTFkMDQtODE5Ni00YzRiLTk1ZTgtYTYyYTE1M2JkNTI2LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzE4VDAxMDI1N1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTJkOGQzMWFmOWFmYjEwZjNiYjM5NjI0NWVlMGEyMGY1MzQzNjJmMmZkMzcxYzcxMGY1MGJiOTVjNjRhYzVlMDgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.KGEnbGWiUipbzf0I_td9BnbG92fQ4RHvJSXC95bSPxM)
![Screen Shot 2023-12-06 at 8 48 35 PM](https://private-user-images.githubusercontent.com/74584964/288434477-5dd24113-914e-4c34-a4b3-079e12677d55.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjEyNjQ4NzcsIm5iZiI6MTcyMTI2NDU3NywicGF0aCI6Ii83NDU4NDk2NC8yODg0MzQ0NzctNWRkMjQxMTMtOTE0ZS00YzM0LWE0YjMtMDc5ZTEyNjc3ZDU1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzE4VDAxMDI1N1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWU4MGFlNTZiNmQyN2E0OThmNTgyOTYzMmEyMzMyYzUyMDdkMDFkODNjY2M2ODI0OWJlZTI4OGIyZmNhZDA2MmQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.L1jYsUGQk3VY3s1H5HvL6tciNqF3kMPNFUgfKxwHdy4)
V. Connect the four components
- connect the s3 bucket to the SNS by creating an event notification at the s3 bucket level: select "All object create events" as the event type and the SNS topic as a destination
![Screen Shot 2023-12-06 at 8 55 40 PM](https://private-user-images.githubusercontent.com/74584964/288435208-95171796-8e87-41db-86cc-a38a4d3bf5c0.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjEyNjQ4NzcsIm5iZiI6MTcyMTI2NDU3NywicGF0aCI6Ii83NDU4NDk2NC8yODg0MzUyMDgtOTUxNzE3OTYtOGU4Ny00MWRiLTg2Y2MtYTM4YTRkM2JmNWMwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzE4VDAxMDI1N1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWM1NTgxNmVmOWZiNjg0Y2E5NTdiYmI2YmY0ZTFkMTMwNTFkN2YwYzcwOTQ3MTVhNzFjMTZiZTZkZmM0OGZiNGEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.NC6lyqKuQWcedm_6pDU2yG4WFVNIeGhcxACf4Wz3N-M)
-
connect SQS and the SNS by creating a subscription at the SNS level (with a filter policy). NB: create two subscriptions for each queue. The filter policy: we filter the message based on the event name: The put event will be forwarded to queue 1
{ "Records": { "eventName": [ "ObjectCreated:Put" ] } }
The copy event will be forwarded to queue 2
![Screen Shot 2023-12-06 at 9 00 56 PM](https://private-user-images.githubusercontent.com/74584964/288435339-8d68de81-e951-4de5-a12b-bb311815848b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjEyNjQ4NzcsIm5iZiI6MTcyMTI2NDU3NywicGF0aCI6Ii83NDU4NDk2NC8yODg0MzUzMzktOGQ2OGRlODEtZTk1MS00ZGU1LWExMmItYmIzMTE4MTU4NDhiLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzE4VDAxMDI1N1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTMyM2ExOTkzYzQxOGU0Y2ZhM2ZhYTVhNTQwODI1NmNlNzA5YmI1YmQ4MjZlN2UxNWU3MGZmZmNiMDE2OGE0ZDMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.v6z5fzn7wpRcavFPlEyXe7zJ-DF_UVEFnDmv7UqW4nE)
{
"Records": {
"eventName": [
"ObjectCreated:Copy"
]
}
}
- Finally, connect SQS queues to the lambda functions at the SQS level.
![Screen Shot 2023-12-06 at 9 05 49 PM](https://private-user-images.githubusercontent.com/74584964/288435475-16005b50-eeb8-4122-adb3-389d27fa5017.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjEyNjQ4NzcsIm5iZiI6MTcyMTI2NDU3NywicGF0aCI6Ii83NDU4NDk2NC8yODg0MzU0NzUtMTYwMDViNTAtZWViOC00MTIyLWFkYjMtMzg5ZDI3ZmE1MDE3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzE4VDAxMDI1N1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTQxZTJlZjk0MjdmODIxMGU4ZDlhODU2ZTMwMzA2NzczZjc5YTE4ZmMzMmY0Mzc1MTU5ZTQxZWViY2Y5NmUxMzgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.TuwYWPBqncsFXfgf2_u-yMX1ix3ju11v05W4o2iYMXw)
VI. Test Setup
![Screen Shot 2023-12-06 at 10 07 19 PM](https://private-user-images.githubusercontent.com/74584964/288435592-993846ee-7d5c-4116-8f44-8a3be1241515.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjEyNjQ4NzcsIm5iZiI6MTcyMTI2NDU3NywicGF0aCI6Ii83NDU4NDk2NC8yODg0MzU1OTItOTkzODQ2ZWUtN2Q1Yy00MTE2LThmNDQtOGEzYmUxMjQxNTE1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzE4VDAxMDI1N1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTZkODg0MjdmODBlMjAwMDA3NjA4MDkyNThlYTFmZTMxYWFiNjgyOWUzNWRkNjQyMTgxODYxNmRhY2UxNjE3MmEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Lk0Wfdrb9_M6zmh5MdgppvkT8WrUFxNcWxWVCaEAW40)
- copy the same file to check the second lambda log
![Screen Shot 2023-12-06 at 10 07 19 PM](https://private-user-images.githubusercontent.com/74584964/288442483-b0d97f74-035d-4ad4-af4d-ebe5ef06b882.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjEyNjQ4NzcsIm5iZiI6MTcyMTI2NDU3NywicGF0aCI6Ii83NDU4NDk2NC8yODg0NDI0ODMtYjBkOTdmNzQtMDM1ZC00YWQ0LWFmNGQtZWJlNWVmMDZiODgyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzE4VDAxMDI1N1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTdjMDFkZDIwM2RkNmU0Nzg3MjZjNTE4OTIwNTAxOTQ1NjdkOWMzODI4ZjdmZmM3ZDc2YjIwOTE2ODRmZjVmZmImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.D5BG0h52DK7YRsYglH5Eot8c872x8ryegAQJ_YR5g_Y)