Lead I - Airflow Migration Developer

6 - 8 Years
1 Opening
Bangalore, Kochi, Trivandrum

Job Summary:

We are seeking an experienced Airflow Migration Developer to lead the migration of job scheduling processes from Autosys to Apache Airflow. The ideal candidate will have a strong background in batch processing, workflow orchestration, and cloud deployment, particularly on AWS. This role involves close collaboration with application, data, and cloud engineering teams to ensure a seamless transition, robust scheduling, and high availability in production environments.

Key Responsibilities:

• Analyze existing Autosys job definitions, dependencies, and workflows.

• Design and implement equivalent Apache Airflow DAGs that maintain or improve scheduling efficiency and SLAs.

• Integrate Airflow with MarkitEDM and other downstream systems.

• Ensure proper logging, monitoring, and ing aligned with SLA and operational standards.

• Collaborate with cloud engineering and application teams to deploy workflows in AWS environments (e.g., ECS, Lambda).

• Create detailed documentation for migration logic, configurations, and operational runbooks.

• Support User Acceptance Testing (UAT) and manage production rollout activities.

Mandatory Skills:

• Strong hands-on experience with Apache Airflow, including DAG design and scheduling.

• Proficient with Autosys job scheduling and batch job management.

• Proficiency in Python for scripting and workflow orchestration.

• Solid experience with AWS services: ECS, S3, Lambda, CloudWatch.

• Familiarity with MarkitEDM or similar data management platforms.

• Knowledge of CI/CD pipelines and DevOps practices.

• Strong debugging, problem-solving, and performance tuning skills.

• Excellent communication and documentation capabilities.

Good-to-Have Skills:

• Experience in financial services or enterprise-scale data environments.

• Knowledge of Airflow deployment in containerized environments (e.g., Docker, Kubernetes).

• Awareness of audit, compliance, and security requirements in batch processing.

• Familiarity with infrastructure automation and Infrastructure as Code (IaC) tools.

Apache Airflow,Autosys job scheduling ,Python,Markit Edm