Architecting an AWS Big Data Pipeline with File Tracking, Audit Logging, & Real-Time Monitoring!

Hi, it's a greate vedio. I do have some real task to do such similar data type convertion. I just wonder for 10 concurrency setting, do you know how long will it take for such job to convert 1TB data?

Thanks for the great video. I am submitting a Spark job to EMR using Lambda. Lambda is getting success irrespective of the job status of the Spark job. How can I get the status of the EMR job to kick off the rest of the Lambda functions?

Underrated

Hi,
I get this following error when i try to run the pipeline. Trying to overcome this from past two days.
''''when calling the StartJobRun operation: Failed to start job run due to missing metadata'''..

Please note: the job name produced in Lambda is right. the region is set right.

Finally I found Exactly what iam looking for ...

zabardast

How to implement the same use case with emr without using glue. Can we implement with emr ?

Great explanation