Data Test Engineer

Information Technology

Job Title:

Job Description

Key Responsibilities:

Test Strategy and Planning: Develop and implement comprehensive test strategies for data processing and machine learning pipelines, ensuring alignment with business requirements.

Test Automation: Design, build, and maintain automated test frameworks using Python to validate Apache Spark jobs, Iceberg table operations, and complex data pipelines.

Data Validation and Quality Assurance: Implement robust data validation checks to ensure data integrity and quality in Iceberg tables and across data pipelines.

Performance Testing: Conduct performance testing of data pipelines and machine learning models, identifying bottlenecks and optimizing Apache Spark jobs for scalability and efficiency.

Cloud Infrastructure Testing: Validate the deployment and functionality of data solutions on Google Cloud Platform, ensuring efficient and secure use of cloud resources.

Continuous Integration and Deployment: Integrate automated tests into the CI/CD pipeline, collaborating with DevOps teams to enhance continuous testing and delivery processes.

Debugging and Issue Resolution: Investigate and resolve issues related to data processing, model performance, and pipeline failures, working closely with developers and data engineers.

Documentation and Reporting: Document test plans, test cases, and test results, providing regular reports on test coverage, defects, and quality metrics to stakeholders.

Qualifications:

Bachelor’s degree in Computer Science, Data Science, or a related field.
Proven experience as an SDET or in a similar role.
Strong proficiency in Python and test automation frameworks.
Experience with Apache Spark, Apache Iceberg, and Google Cloud Platform.
Expertise in data validation and quality assurance practices.
Excellent analytical skills and problem-solving abilities.
Ability to work in a team-oriented and collaborative environment.
Preferred Skills:

Experience with performance testing tools and techniques.
Familiarity with machine learning frameworks and data science concepts.
Strong communication skills and the ability to explain complex technical concepts to non-technical stakeholders.
This role is crucial in maintaining the quality and reliability of our data-driven solutions, ensuring they meet the highest standards of performance and accuracy.

Location:

Language Requirements:

Time Type:

If you are a California resident, by submitting your information, you acknowledge that you have read and have access to the Job Applicant Privacy Notice for California Residents