Qureos Mentor

Rapsys Technologies Pte. Ltd

We are seeking a highly skilled Senior PySpark Data Engineer to design, build, and optimize large-scale data pipelines and distributed systems. Beyond deep expertise in Apache Spark (PySpark) and automation, this role requires the ability to manage stakeholders, ensure timely delivery, and assess requirements. You will play a critical role in bridging business needs with technical execution, ensuring high-quality, scalable, and reliable data solutions.KEY RESPONSIBLITIESTechnical Delivery<ul><li>Data Pipeline Development: Design, develop, and optimize scalable ETL/ELT pipelines using PySpark.</li><li>System Integration &amp;amp; Automation: Build automated data ingestion and transformation frameworks integrating with APIs, databases, enterprise applications, and cloud platforms.</li><li>Performance Optimization: Tune Spark jobs for efficiency, scalability, and cost-effectiveness.</li><li>Data Quality &amp;amp; Governance: Implement validation, monitoring, and controls to ensure data reliability and compliance.</li><li>DevOps &amp;amp; Orchestration: Deploy pipelines through CI/CD frameworks and manage workflows using Airflow, Prefect, or similar tools.</li></ul>Stakeholder Management<ul><li>Requirements Assessment: Engage with business stakeholders, analysts, and application owners to gather and translate requirements into technical deliverables.</li><li>Deliverables &amp;amp; Timelines: Define scope, create technical plans, and manage progress to ensure on-time delivery of data solutions.</li><li>Communication: Provide regular updates on progress, risks, and dependencies to technical and non-technical stakeholders.</li><li>Cross-Functional Collaboration: Work closely with data scientists, product managers, and business teams to align solutions with organizational goals.</li><li>Risk &amp;amp; Issue Management: Identify delivery risks early and proactively recommend mitigation strategies.</li><li>Mentorship &amp;amp; Leadership: Guide junior engineers and champion best practices for data engineering and delivery management.</li></ul>EducationalQualification<ul><li>Bachelor&rsquo;s or Master&rsquo;s degree in Computer Science, Data Engineering, Information Systems, or related field.</li><li>Experience Minimum 7+ years&rsquo; experience</li><li>Required skills: Pyspark, Python, Airflow, SQL</li><li>Domain: Data warehousing, Banking, Telecom</li></ul>Job Type: Contractual / Temporary Contract length: 12 monthsEducation:<ul><li>Secondary(10th Pass) (Preferred)</li></ul>Experience:<ul><li>Pyspark: 4 years (Required)</li><li>Data Engineer: 6 years (Required)</li><li>GCP/Azure/AWS: 1 year (Preferred)</li><li>Airflow: 1 year (Required)</li><li>SQL: 1 year (Required)</li></ul>Work Location: In person

Senior PySpark Data Engineer

SummaryWe are seeking a highly skilled Python / Airflow Data Engineer with 7 to 8 years of experience, specifically with a strong background in AWS technologies. The ideal candidate will have a deep understanding of Apache Airflow and its integration within AWS ecosystem, enabling efficient data pipeline orchestration and management.Responsibilities<ul><li>Design, develop, and maintain complex data pipelines using Python for efficient data processing and orchestration.</li><li>Collaborate with cross-functional teams to understand data requirements and architect robust solutions within the AWS environment.</li><li>Implement data integration and transformation processes to ensure optimal performance and reliability of data pipelines.</li><li>Optimize and fine-tune existing data pipelines / Airflow to improve efficiency, scalability, and maintainability.</li><li>Troubleshoot and resolve issues related to data pipelines, ensuring smooth operation and minimal downtime.</li><li>Work closely with AWS services like S3, Glue, EMR, Redshift, and other related technologies to design and optimize data infrastructure.</li><li>Develop and maintain documentation for data pipelines, processes, and system architecture.</li><li>Stay updated with the latest industry trends and best practices related to data engineering and AWS services.</li></ul>Requirements<ul><li>Bachelor&rsquo;s degree in Computer Science, Engineering, or a related field.</li><li>Proficiency in Python, PySpark and SQL for data processing and manipulation.</li><li>Min 5 years of experience in data engineering, specifically working with Apache Airflow and AWS technologies.</li><li>Strong knowledge of AWS services, particularly S3, Glue, EMR, Redshift, and AWS Lambda.</li><li>Understanding of Snowflake Data Lake is preferred.</li><li>Experience with optimizing and scaling data pipelines for performance and efficiency.</li><li>Good understanding of data modeling, ETL processes, and data warehousing concepts.</li><li>Excellent problem-solving skills and ability to work in a fast-paced, collaborative environment.</li><li>Effective communication skills and the ability to articulate technical concepts to non-technical stakeholders.</li></ul>Preferred Qualifications:<ul><li>AWS certification(s) related to data engineering or big data.</li><li>Experience working with big data technologies like Snowflake, Spark, Hadoop, or related frameworks.</li><li>Familiarity with other data orchestration tools in addition to Apache Airflow.</li><li>Knowledge of version control systems like Bitbucket, Git.</li></ul>Job Type: Contractual / Temporary Contract length: 12 monthsExperience:<ul><li>Data engineer: 6 years (Required)</li><li>Python: 1 year (Required)</li><li>Airflow: 1 year (Required)</li><li>AWS service S3, Glue, EMR, Redshift, and AWS Lambda.: 1 year (Required)</li><li>PySpark: 1 year (Required)</li><li>SQL: 1 year (Required)</li></ul>Work Location: Remote

Data Engineer(Python + Airflow + Snowflake)

Join Rapsys Technologies Pte. Ltd and explore 2   exciting career opportunities. They're hiring for various roles with options for remote or on-site work. Apply today and become part of their innovative team.

Rapsys Technologies Pte. Ltd

ABOUT_COMPANY

Senior PySpark Data Engineer

Data Engineer(Python + Airflow + Snowflake)