How to run Spark Job on EMR Serverless

What is EMR Serverless?

EMR Stands for Elastic Map Reduce.

This is a handy option for running big data jobs on the fly, POC without worrying about setting up infrastructure.

Here are quick steps to utilize this :

You can possibly create different Studios for different teams / purposes within your account.

Head over to EMR Studio and click “Create Studio”

On the new screen, give an appropriate name and description. More important setting here is to select a proper service role(This is the role your virtual clusters will assume) so make sure to attach any policies that your cluster may need. e.g S3 access, DB access etc.

That’s it. Once Studio is created, it’s extremely simple to submit the spark job:

Click “Create application”

Once an application is created, click on “Submit job”

On new screen, give an appropriate name, provide jar file (at S3) EMR role should have access to this bucket.

Provide main class name , any optional param for your job and submit.

That’s it, AWS will take care of creating infrastructure and running job for you like Magic !!!

Related posts:

Walking around the bustling hallways, I look back at the program to confirm the room number and speaker. I squeeze through a small crowd, reach my destination, and found an empty seat. I sat next to…

Last weekend my family and I spent time at a resort about three hours from home to celebrate the 40th anniversary of my parents. This trip was a mental experiment of sorts. Typically, the role that I…

Is it likely that you are a tennis elbow casualty yet not happy with the tennis elbow treatment you where first given by your family subject matter expert? Then, you will very much love to find that…