AWS Data Engineer

Location: Remote
Duration: 4 months
Bidget: 70-80k

NOTE : Only contractors or freelancers should apply

Description:

This job description focuses on the design, development, and maintenance of data pipelines and ETL (Extract, Transform, Load) processes within the AWS ecosystem. The role heavily leverages AWS Glue for data transformation and Amazon S3 for data storage.

Key Responsibilities:

• Data Pipeline Design and Development: Design, build, and maintain scalable and efficient data pipelines using AWS Glue, integrating with other AWS services like Lambda, EMR, Redshift, Kinesis, and Athena.

• ETL Job Creation and Management: Develop and optimize AWS Glue ETL jobs (using Spark/Python/Scala) to clean, transform, and enrich raw data for analytics or other downstream applications.

• Data Cataloging and Discovery: Use AWS Glue Data Catalog and crawlers to automatically discover, catalog, and manage metadata for various data sources, including those stored in Amazon S3

. • Amazon S3 Data Management: Manage data storage in Amazon S3, including setting up appropriate storage classes, managing data lifecycle policies, and ensuring data security and accessibility.

• Data Quality and Validation: Implement data quality checks and validation rules within the ETL process to ensure data accuracy and reliability

. • Monitoring and Troubleshooting: Monitor the performance and health of Glue jobs and data pipelines, troubleshoot issues, and optimize for efficiency and cost-effectiveness.

• Collaboration and Communication: Work with data engineers, data scientists, and other stakeholders to understand data requirements and deliver robust data solutions.

Job Types: Contractual / Temporary, Freelance
Contract length: 4 months

Pay: ₹70,000.00 - ₹80,000.00 per month

Work Location: Remote