Lead AWS Data Engineer

JOB_REQUIREMENTS

Hires in

Not specified

Employment Type

Not specified

Company Location

Not specified

Salary

Not specified

Job Title: Lead AWS Data Engineer

Experience: 10+ Years

Location: Pune, Kalyani Nagar (work from office)

Work Time zone: US Mountain Time (MT) (Starting from 9 PM/10 PM IST), Transport will be provided

Job Summary: We are looking for an experienced PySpark Developer with strong knowledge of AWS services, particularly AWS Lambda, to design and implement scalable data processing workflows for large datasets. The ideal candidate will have hands-on experience orchestrating Spark-based ETL jobs using AWS Glue, EMR, and Lambda, along with querying and integrating Athena and DynamoDB for analytical and operational use cases.

Key Responsibilities:

Design and develop data transformation pipelines using PySpark on AWS Glue or Amazon EMR.
Use AWS Lambda to trigger and orchestrate PySpark jobs as part of scalable data workflows.
Build event-driven architectures using S3 events, CloudWatch, and Step Functions.
Integrate Amazon Athena for query-based transformations and reporting pipelines.
Read/write and process structured/unstructured data using DynamoDB, S3, and Athena.
Optimize PySpark code for performance, scalability, and cost-efficiency on cloud-based data platforms.
Collaborate with cross-functional teams to translate data requirements into technical implementations.
Monitor, debug, and tune data pipelines using CloudWatch, Glue/EMR logs, and Athena query metrics.

Required Skills:

10+ years of experience with Python, PySpark, and distributed data processing.
Strong hands-on experience with AWS Glue, AWS Lambda, Amazon EMR, and S3.
Proven experience integrating with Amazon Athena and DynamoDB.
Proficient in building and deploying serverless solutions using Step Functions, EventBridge, and Lambda.
Solid understanding of data formats (Parquet, Avro, JSON, etc.) and transformation logic.
Experience working in US Mountain Time Zone.

Good to Have:

Exposure to CI/CD pipelines, Infrastructure as Code (Terraform/CDK).
Knowledge of data lake and data governance architectures.
Prior experience with on-premises to cloud data migration.
Familiarity with Athena performance tuning and DynamoDB Streams for event-driven ingestion.

Similar jobs