AWS Data Engineer (Senior)

About the Role

We are seeking a highly skilled Lead Data Engineer to define and drive the data migration strategy from legacy RDBMS platforms to PostgreSQL for a mission-critical billing and invoicing system. This role requires a hands-on technical leader who can design the migration architecture, oversee implementation, and ensure seamless transition without disruption to core business functions.

The ideal candidate will have deep expertise in data migration frameworks, large-scale distributed processing (Spark preferred), and experience with both open-source and AWS ecosystem tools. They will work closely with business stakeholders and customers to manage expectations, deliver with precision, and ensure system reliability and performance post-migration.

Key Responsibilities

• Define and Own Migration Strategy
• Architect end-to-end data migration solution from RDBMS to PostgreSQL.
• Evaluate open-source and AWS-native tools (e.g., AWS DMS, Glue, EMR, Schema Conversion Tool).
• Design frameworks for schema conversion, bulk data loads, CDC (Change Data Capture), and reconciliation.

• Technical Leadership
• Lead a team of data engineers to execute migration plans.
• Establish coding, testing, and validation standards to ensure accuracy and performance.
• Leverage Spark (preferred compute engine) for high-volume transformation and migration workloads.

• Customer & Stakeholder Management
• Engage with business and customer teams to align migration timelines with billing cycle dependencies.
• Manage customer expectations through transparent planning, reporting, and risk management.
• Serve as the escalation point for migration challenges.

• Performance, Reliability & Compliance
• Ensure zero or minimal downtime for the billing and invoicing systems during migration.
• Implement data validation and reconciliation frameworks to guarantee accuracy.
• Adhere to compliance and security requirements for sensitive customer and financial data.

Required Qualifications

• 8+ years of experience in Data Engineering, with at least 3 years in a lead or architect role.
• Proven expertise in RDBMS-to-PostgreSQL migration projects.
• Strong hands-on experience with Apache Spark for large-scale data transformations.
• Solid knowledge of AWS Data Services (Glue, EMR, RDS, S3, IAM).
• Strong SQL/PLSQL experience, knowledge of data validation frameworks and reconciliation techniques.
• Ability to design and implement highly available, secure, and scalable data pipelines.
• Excellent communication skills to manage stakeholders and set customer expectations.

Preferred Skills

• Experience with billing/invoicing or financial systems.
• Familiarity with Terraform or Infrastructure-as-Code (IaC).
• Strong SQL and performance tuning expertise in PostgreSQL.
• Knowledge of DevOps/DataOps best practices.