AWS DynamoDB Streams to Lambda Tutorial in Python | Step by Step Guide

AWS DynamoDB Streams to Lambda Tutorial in Python | Step by Step Guide

Be A Better Dev

4 года назад

59,726 Просмотров

DynamoDB Streams are a powerful feature that allow applications to respond to change on your table's records. Combining this feature with Lambda functions creates a highly scalable and fault tolerant application that can easily respond to record modification. In this video, I show you how to create a Lambda function and define a handler function (in python) that will respond to DynamoDB change events. From there, I define a required IAM role and attach it to the Lambda Function. Afterwards, I create a DynamoDB table and hook up the Lambda trigger. I test the connectivity by making manual updates on my DynamoDB table and confirm the Lambda function is responding appropriately.

Looking to get hands on experience building on AWS with a REAL project? Check out my course - The AWS Learning Accelerator! https://courses.beabetterdev.com/courses/aws-learning-accelerator

📚 MY RECOMMENDED READING LIST FOR SOFTWARE DEVELOPERS📚
Clean Code - https://amzn.to/37T7xdP
Clean Architecture - https://amzn.to/3sCEGCe
Head First Design Patterns - https://amzn.to/37WXAMy
Domain Driver Design - https://amzn.to/3aWSW2W
Code Complete - https://amzn.to/3ksQDrB
The Pragmatic Programmer - https://amzn.to/3uH4kaQ
Algorithms - https://amzn.to/3syvyP5
Working Effectively with Legacy Code - https://amzn.to/3kvMza7
Refactoring - https://amzn.to/3r6FQ8U

🎙 MY RECORDING EQUIPMENT 🎙
Shure SM58 Microphone - https://amzn.to/3r5Hrf9
Behringer UM2 Audio Interface - https://amzn.to/2MuEllM
XLR Cable - https://amzn.to/3uGyZFx
Acoustic Sound Absorbing Foam Panels - https://amzn.to/3ktIrY6
Desk Microphone Mount - https://amzn.to/3qXMVIO
Logitech C920s Webcam - https://amzn.to/303zGu9
Fujilm XS10 Camera - https://amzn.to/3uGa30E
Fujifilm XF 35mm F2 Lens - https://amzn.to/3rentPe
Neewer 2 Piece Studio Lights - https://amzn.to/3uyoa8p

💻 MY DESKTOP EQUIPMENT 💻
Dell 34 inch Ultrawide Monitor - https://amzn.to/2NJwph6
Autonomous ErgoChair 2 - https://bit.ly/2YzomEm
Autonomous SmartDesk 2 Standing Desk - https://bit.ly/2YzomEm
MX Master 3 Productivity Mouse - https://amzn.to/3aYwKVZ
Das Keyboard Prime 13 MX Brown Mechanical- https://amzn.to/3uH6VBF
Veikk A15 Drawing Tablet - https://amzn.to/3uBRWsN
 

📚 References / Resources:
Getting started with AWS: https://youtu.be/lTyqzyk86f8

Lambda function code (python): https://gist.github.com/djg07/2bcf216822c9ab664d8286924776999f

☁Topics covered include:
- Lambda Function Creation
- Lambda handler using python
- Lambda DynamoDB Streams IAM role
- DynamoDB Table Setup
- DynamoDB Streams
- DynamoDB NewImage
- DynamoDB OldImage
- DynamoDB Stream "Change" Events
- Lambda Cloudwatch Monitoring

🌎 Find me here:
Twitter - https://twitter.com/BeABetterDevv
Instagram - https://www.instagram.com/beabetterdevv/
Patreon - Donations help fund additional content - https://www.patreon.com/beabetterdev

#DynamoDB
#DynamoDBStreams
#Lambda
#AWS
#Serverless

Тэги:

#AWS_DynamoDB_Streams #DynamoDB #DynamoDB_Stream #DynamoDB_Streams #DynamoDB_Stream_Lambda #dynamodb_streams_python #walkthrough #dynamodb_streams_lambda #dynamodb_stream_lambda #dynamodb_old_image #dynamodb_new_image #aws_dynamodb_stream #aws_dynamodb_streams #aws_dynamodb_stream_python #aws_dynamodb_streams_python #dynamo_lambda #dynamo_stream_lambda #dynamodb_trigger #dynamodb #aws #amazon_web_services #dynamodb_trigger_lambda #serverless #serverless_nodejs #aws_developer
Ссылки и html тэги не поддерживаются


Комментарии:

@estebanrodriguez6307
@estebanrodriguez6307 - 23.01.2024 11:23

Great content! just wanted to know where did you get the eventsSample.json?

Ответить
@malikahmed7045
@malikahmed7045 - 09.12.2023 14:59

You are amazing and extremely intelligent, how come I never found your channel before.
Why did you stop posting new videos?
You could very easily be an inspiration my friend for millions of people out there seeking knowledge

Ответить
@MCAMarshallS
@MCAMarshallS - 28.08.2023 07:59

hii bro , can you provide aws rds , dynamo db lab videos link ?

Ответить
@Ckbagchi
@Ckbagchi - 12.06.2023 19:29

Loved the way u have explained the topic with example.

Ответить
@JesusAlfredoHernandezOrozco
@JesusAlfredoHernandezOrozco - 14.02.2023 21:39

Great explanation. Thank you very much!

Ответить
@knandi73
@knandi73 - 11.08.2022 21:35

To make your videos carry better quality and professionalism, please avoid the points below.
1. Do not make sounds with lips after speaking few sentences as a pause-maker.
2. Avoid musical pronunciations.

Ответить
@joeaabb
@joeaabb - 21.02.2022 10:05

Terrific series. Could you do a series on S53

Ответить
@pratikgupta9692
@pratikgupta9692 - 21.01.2022 07:47

Hi, such a great video! I just have one question. Can i do something so that it doesn't trigger the lambda for every event but does it on an hourly basis or based on the number of records inserted?

Ответить
@soniauppal
@soniauppal - 04.01.2022 17:56

Excellent very usful video but i have a question when you said left side we have references insert on our DynamoDB table so this reference code is made by us only and where we are keeping it please help!!

Ответить
@ting-yuhsu4229
@ting-yuhsu4229 - 11.12.2021 18:38

would be interested, if you also show the handling when lambda failed to execute and send message to DLQ

Ответить
@GuitarreroDaniel
@GuitarreroDaniel - 06.12.2021 08:01

I love you man. This video was so easy to follow and I was able to made my own implementation at the first try. Thanks for everything!

Ответить
@salmanshaik452
@salmanshaik452 - 10.11.2021 19:08

Excellent demonstration Kudos! How will you write the Modify Function if there are changes multiple columns in single record? Please let me know

Ответить
@SamuelSantana1000
@SamuelSantana1000 - 25.10.2021 02:52

Nice, Man! I was able to learn in few minutes! Thk!

Ответить
@brucedeo1981
@brucedeo1981 - 24.10.2021 03:02

my Tables, how no option to set up triggering from there to the Lambda function. New console experience, contains no such option (unless i dont see it). Switching back to the old console i can see it.

Ответить
@khilielbullock8509
@khilielbullock8509 - 18.10.2021 17:52

I wonder is it possible to do like a lambda to check to see if a name (or something else) already exists within a dynamoDB database. If it does, then... x, if not then... y

Ответить
@oerickmuller
@oerickmuller - 10.09.2021 16:42

Great video, thanks a lot.

Ответить
@yelloverz5537
@yelloverz5537 - 17.08.2021 21:00

But where you put the code for inserting into DB

Ответить
@CptSupermrkt
@CptSupermrkt - 27.07.2021 15:05

Love this channel.

It's kind of interesting to me that CloudWatch Metrics shows up as a single invocation, but the CloudWatch Logs show three very distinct invocations with unique IDs, execution times, and memory usage. I was thinking that maybe with batching, the way Lambda works might be different than I'm used to, but according to the documentation, the RequestId that shows up in the CloudWatch Logs refers to "The unique request ID for the invocation," and your log shows 3 unique IDs, yet the CloudWatch Metrics invocation count is 1. Not really a question I suppose, just wondering how this is working under the hood.

Ответить
@nikitasharma5087
@nikitasharma5087 - 21.07.2021 08:31

I did the same but I am facing an issue that some of my records are missing from the lambda event, i.e. if 100 records are being processed in dynamo then the lambda receives 97 records ... some records get missed, I can't find them in the event, anything I can do in this matter please suggest.

Ответить
@SuperAyan98
@SuperAyan98 - 06.07.2021 16:17

Hi , I had a question . Can you load each stream 'event' that you're iteratively parsing into a buffer and load that buffer into s3 ? If so then how ? Thanks a lot for the great content !

Ответить
@priyankapatel9461
@priyankapatel9461 - 18.06.2021 16:01

Useful video! How can I achieve this requirement?

Collect and store information regarding creation and deletion of S3 buckets and also creation and termination of EC2 instances in the AWS account


1. Create a CloudWatch Rule to listen to the below AWS services event sources and event types:


a) S3 - Create and Delete bucket operations
b) EC2 - Create and terminate instance states


2. The CloudWatch rule from #1 should trigger a Lambda function, the lambda function should parse the event to log the following details about the event in a DynamoDB table:


Hint: Use AWS SDK for Python (boto3) to store the information in DynamoDB table.


a) Event time
b) Event source
c) Event name
d) Resource name (Bucket name or instance ID)
e) AWS region
f) Username

Ответить
@mr-oe2kd
@mr-oe2kd - 16.06.2021 13:38

Can we pipe dynamodb stream directly to aws eventbridge without a lambda?

Ответить
@rajmishra6190
@rajmishra6190 - 06.06.2021 14:43

Excellent !! could you please update this code to interact with Lex bot ?

Ответить
@konstantinlitvin8071
@konstantinlitvin8071 - 22.04.2021 10:32

Very useful. Thank you very much!

Ответить
@nishitrathi4730
@nishitrathi4730 - 05.04.2021 07:02

Great explanation. I have one question, How do we send the inserted/updated/deleted rows to Elastic search ?

Ответить
@tanmayrane858
@tanmayrane858 - 04.04.2021 09:52

Great tutorial !!! I want to display the lastest/newest row in my Table to the S3 Website, please guide.

Ответить
@cipher6042
@cipher6042 - 27.03.2021 02:38

This video was dope and helped me alot with something Im doing for work, many thanks!!!

Ответить
@devilangel036
@devilangel036 - 16.01.2021 01:15

Do you have a video on how to read data from dynamodb stream using python?

Ответить
@venkataseshapyeddanapudi320
@venkataseshapyeddanapudi320 - 27.12.2020 16:59

great video. how about adding an attribute to an entry/row in the table and then to all the entries/rows at a single instance? Can you please make that video?

Ответить
@kelvintailor2448
@kelvintailor2448 - 15.12.2020 23:22

Great codding. Thanks, its really helpful to me also.

Ответить
@kiranmahesh93
@kiranmahesh93 - 04.12.2020 17:16

Where is the video to check why try catch block necessary for lambda function in aws, please let me know as I was searching it from yesterday

Ответить
@javiasilis
@javiasilis - 29.11.2020 18:47

Thank you for the explanation. Thanks for taking the time and energy to perform a very simple and understanding tutorial.

Now talking directly into a real application. What if I designed DynamoDB in a single table instance? From what I'm seeing DynamoDB streams do not differentiate the type of object that was inserted, unless I add a special key that identifies the type, which I can then use to filter it out.

Without knowing I've been doing what DynamoDB streams offers in a manual fashion, by implementing Domain Events directly in the application. What I like about Streams is that your application becomes a little less susceptible to out of sync because you guarantee that an item is inserted.

What I'm still thinking is that if your application needs to do a multi-step insertion, (User creates an account, then you need to contact a third party service, after that send an email... what if step 2 fails, and I can't let the user continue without proper account creation?) and one of those data fails down the line, you'd still need to look for a way either to recover or to rollback the operation (Dead Letter Queues I suppose, or letting the user know about the failed operation).

Ответить
@N1NJ4ASSASIN1
@N1NJ4ASSASIN1 - 17.11.2020 00:59

Hey thanks a bunch for your videos, they're very helpful man! There's a new messaging system for CloudWatch, and it doesn't show the print statements from your code, rather REPORT, START, and END RequestId: followed by a unique id. Is there any way we can go back to the previous console to check our print statements and details within our lamdba functions? This will help out a bunch when I'm error handling my own functions

Ответить
@knaraya936
@knaraya936 - 20.10.2020 06:45

Thanks for a good intro to dynamodb streams. I noticed that you did not enable Streams when you created the dynamodb GameScore table, yet you enabled streams roles in IAM which you attached to the Lambda function, and you used triggers in the dynamodb table and everything worked. So I am still confused about not enabling Streams - is that really ok?

Ответить
@jongschneider
@jongschneider - 19.10.2020 02:55

I love your videos. They have helped me better understand a number of AWS concepts.

Do you use Terraform to manage your infrastructure as code? If so, have you considered doing a Terraform series?

Ответить
@ViniciusFeitosa
@ViniciusFeitosa - 02.10.2020 20:57

Thanks again. Your videos are much better than the official AWS documentation

Ответить
@harjos78
@harjos78 - 26.08.2020 18:40

Awesome!. Your videos are really very helpful.. Appreciate the efforts in making this video

Ответить
@SammyTvMan
@SammyTvMan - 15.08.2020 02:29

Great video

Ответить
@mabrouk2005
@mabrouk2005 - 06.08.2020 18:58

can you do a video on lambda and AWS load balancers. thanks

Ответить
@PoojaSumann
@PoojaSumann - 31.07.2020 01:41

Thanks for the video. Please upload these videos with code in Java too.

Ответить
@makhus3478
@makhus3478 - 27.07.2020 06:21

Can you explain how to push to elastic search (insert/modify) from dynamodb. Please take care of index while pushing as ES will automatically give index. Push index also from dynamodb (id)

Ответить
@rickfarina9835
@rickfarina9835 - 14.06.2020 21:16

Clean and to the point... nicely done!!

Ответить
@piyushmajgawali1611
@piyushmajgawali1611 - 16.05.2020 08:39

can I control event generation? I don't care about insert and remove. I only care about modifications in certain fields.

Ответить
@renianx7610
@renianx7610 - 24.04.2020 14:03

Thanks a lot.
I believe they are excellent videos.

Ответить
@krishnavenkatachalam985
@krishnavenkatachalam985 - 12.04.2020 16:34

Excellent video.. I have a question - I see that in your cloudwatch logs then entire record or event got printed before every operation even though you did not explicitly print the complete record. How did it get printed?

Ответить
@shoebmoin10
@shoebmoin10 - 01.04.2020 20:28

What is the software that you use for making these videos?

Ответить