Senior Data Engineer

Company Description

Help us to achieve our goal to be the global leader in total quality services.

With your help we will achieve this by delivering Total Quality 360, a comprehensive suite of cutting-edge services which combine quality engineering, cyber security, conformance & interoperability and content quality to deliver end-to-end total quality solutions.

Here at Resillion, our culture is based on an ‘if you see something, say something’ attitude where we take responsibility. It’s one where we expect to adapt and embrace change as the company grows. It’s based on recognising the individual worth of every one of our employees and developing their skills to keep us all at the forefront of our industry.

Above all, it’s a culture where we’re passionate about what we do, and we’re committed to the greater good of the company.

If you would like to be part of our journey, then this role may be the one for you.

Job Description

Senior Data Engineer - Quality Engineering

Experience Range: 7-10 Years

Location: Bangalore (Hybrid Mode)

Resillion, a leading quality engineering company with offices around the world, is seeking a talented Data Engineer to join our growing India team. In this role, you will play a critical part in building and maintaining the data infrastructure that supports our AI-powered testing tools and analytics solutions. You will have technical responsibility for the entire data lifecycle, from data acquisition and ingestion to storage, processing, and analysis.

Responsibilities:

• Collaborate with GenAI engineers, Software Engineers, Test automation engineers and other stakeholders within Resillion to understand data needs and translate them into technical solutions, including designing data pipelines for training & deploying models and data pre-processing for AI, including generative AI applications.

• Design, implement, and advise on data migration testing strategies and data quality assurance strategies for Resillion customers, ensuring a smooth transition of data to new customer systems.

• Design, develop, and implement scalable data pipelines using cloud-based data engineering tools and technologies, with a focus on both Microsoft Azure solutions (e.g., Azure Data Factory, Azure Databricks) and Google Cloud Platform (GCP) solutions (e.g., Google Cloud Dataflow, Google Cloud Dataproc).

• Write efficient and maintainable code to extract, transform, and load data from various sources, leveraging your expertise in Azure Data Lake Storage and other relevant Azure services, as well as Google Cloud Storage and other relevant GCP services.

• Build and manage data warehouses and data lakes for quality engineering data, utilizing your knowledge of Azure Synapse Analytics or similar technologies, and Google BigQuery or similar technologies.

• Develop and implement data quality checks and monitoring procedures to ensure data integrity using Azure Data Catalog or other appropriate tools, as well as Google Cloud Data Catalog or other appropriate tools.

• Automate data engineering tasks and workflows using Azure automation tools and GCP automation tools (e.g., Cloud Functions, Cloud Composer).

• Set up quality intelligence dashboards for quality assurance data using Microsoft Power BI to provide stakeholders with clear and actionable insights.

• Stay up-to-date with the latest data engineering tools and technologies, including advancements in AI, Machine Learning (MLOps), and generative AI for data processing in both the Azure and GCP environments.

• Advise on and implement test data generation strategies and solutions for various testing needs.

Qualifications

• Minimum 7+ years of experience in data engineering or a related field

• Proven experience in designing, developing, and deploying data pipelines

• Strong understanding of data warehousing, data lakes, and data modelling concepts

• Expertise in SQL and scripting languages (e.g., Python)

• Experience with cloud platforms (specifically Microsoft Azure and Google Cloud Platform) and their data engineering services (Azure Data Factory, Databricks, Data Lake Storage, Synapse Analytics and Google Cloud Dataflow, Dataproc, Cloud Storage, BigQuery, etc.)

• Experience working with data formats commonly used in software development (e.g., logs, code repositories) is a plus

• Experience with data migration strategies and data migration testing

• Experience in defining and implementing data testing strategies and data quality assurance processes, particularly in data migration projects.

• Experience in advising on and implementing test data generation techniques and tools.

• Experience with MLOps practices and tools for deploying and monitoring machine learning models.

• Experience with generative AI concepts and their application in data processing and analysis.

• Experience with Microsoft Power BI for data visualization

• Excellent communication and collaboration skills

• Ability to work independently and as part of a team

Additional Information

Why Join Us?

• Work in a dynamic and rapidly evolving field.
• Competitive compensation, benefits, and growth opportunities.
• Make a significant impact on the organization’s overall posture.