Secondary skills: Snowflake, Databricks, and Python

Mode of work: Work from Office

Experience: 7-10 Years

About the job:

We are seeking a highly skilled and experienced Data Engineer with expertise in Python and PySpark to join our team. The ideal candidate will have 7 to 10 years of experience in designing, developing, and maintaining scalable data pipelines. You will work with large datasets, both structured and unstructured, and develop data models for data scientists and analysts. Your role will involve optimizing Spark jobs, designing data integration workflows, and ensuring the reliability and stability of data pipelines..

Know your team:

At ValueMomentum’s Engineering Center, we are a team of passionate engineers who thrive on tackling complex business challenges with innovative solutions while transforming the P&C insurance value chain. We achieve this through strong engineering foundation and continuously refining our processes, methodologies, tools, agile delivery teams, and core engineering archetypes. Our core expertise lies in six key areas: Cloud Engineering, Application Engineering, Data Engineering, Core Engineering, Quality Engineering, and Domain expertise.

Join a team that invests in your growth. Our Infinity Program empowers you to build your career with role-specific skill development, leveraging immersive learning platforms. You'll have the opportunity to showcase your talents by contributing to impactful projects.

Responsibilities:

• Design, develop, and maintain scalable data pipelines using PySpark and related big data technologies.
• Work with large datasets and develop data models for consumption by data scientists and analysts.
• Optimize Spark jobs for better performance and resource management.
• Design and implement data integration workflows between various data sources.
• Troubleshoot and resolve issues related to data pipelines.
• Collaborate with cross-functional teams to understand business requirements and deliver solutions.
• Ensure data quality and cleanliness using validation and transformation techniques.
• Write and maintain efficient, scalable code in Python and PySpark.
• Manage data storage, computation, and scaling on cloud platforms like AWS or Azure.

Requirements:

• Bachelor’s degree in Computer Science, Engineering, or a related field.
• 2 to 4 years of experience with PySpark for data processing on large-scale datasets.
• Solid understanding of Spark architecture, including RDDs, DataFrames, and Datasets.
• Strong programming experience in Python, including libraries such as pandas, numpy, and matplotlib.
• Experience with Hadoop, Hive, and NoSQL databases (e.g., Cassandra, MongoDB).
• Working knowledge of cloud computing services (e.g., AWS, Azure, or Google Cloud).
• Familiarity with batch and stream processing (using Kafka, Flink, Spark Streaming).
• Strong problem-solving skills and attention to detail.
• Excellent communication and teamwork skills.

Good To Have:

• Experience with Apache Airflow or other orchestration tools.
• Familiarity with Docker or Kubernetes for containerized data environments.
• Experience in implementing and managing CI/CD pipelines, focusing on automating, testing, and deploying code.