Machine Learning QA

Job Title: Machine Learning QA

About the Role

The ML QA will perform quality checks on work done by developers who design, build, and maintain machine learning models and data pipelines powering core business use cases. The role is hands-on with Python for model development, feature engineering,

and pipeline automation, leveraging Azure ML, and Azure DevOps. Success means robust, production grade models with proven business impact, traceable lineage, and operational excellence at scale.

Key Responsibilities

• Feature Engineering & Model Development

o Translate model prototypes from Data Scientists into Azure ML production pipelines,

including data ingestion, training, inference, and retraining.

o Build and iterate on ML models (forecasting/classification/regression) using modern ML

frameworks (scikit-learn, XGBoost, LightGBM, PyTorch/TensorFlow).

o Develop robust feature pipelines (deterministic code, modular definitions, reusability)

using Pandas and orchestrate in AML Pipelines Jobs.

o Design experiments with proper sampling, train-test splits, cross-validation, and metrics

selection (e.g., RMSE, AUC, MAPE).

o Implement model selection, champion/challenger promotion, and versioning strategies.

o Document experiment results for reproducibility and regulatory compliance.

• Model Operationalization & Monitoring

o Productionize models as batch or real-time endpoints via Azure ML.

o Implement model validation gates (drift/shift, prediction distribution checks, champion

vs. challenger results).

o Set up model monitoring dashboards for latency, prediction freshness, data drift, and

feature importance tracking.

o Integrate model deployment/test harnesses with Azure DevOps pipelines for CI/CD.

o Develop FastAPIs to invoke and consume ML models.

• Data Engineering & Quality

o Profile, clean, and transform raw data from Snowflake, SQL, and third-party sources.

o Implement checks for data quality (nulls, schema validation, outlier handling, time

alignment, duplicate detection).

o Automate feature extraction and maintain feature store consistency.

• Collaboration & Quality Ops

o Work with Product, Data, and QA teams to agree on model acceptance criteria and

experiment reviews.

o Contribute to defect taxonomy (data/model/serving), pipeline observability, and SLO dashboards.

o Publish model performance reports and SLI/SLO summaries for stakeholders.

Required Qualifications

• 4+ years developing data-focused solutions (3+ years in ML modeling and operations).
• Advanced proficiency in Python, including writing high-performance, optimized code for:

– Exploratory Data Analysis (efficient pandas/NumPy usage, vectorization, memory

optimization)

• _ Well versed with pandas, NumPy, ML frameworks, scikit-learn

– Data transformation pipelines (scalable ETL/ELT patterns, modular reusable code)

– Model building and tuning (scikit-learn, XGBoost/LightGBM, PyTorch/TensorFlow)

– Parallel and distributed processing using tools such as multiprocessing, joblib

• Understanding of model validation, drift detection, and online monitoring.
• Experience with feature stores.
• Bachelor’s/Master’s degree in Computer Science, Statistics, Information Technology or related field.
• Certification in Azure Data or ML Engineer Associate is a plus

Job Types: Full-time, Contractual / Temporary
Contract length: 6-12 months

Pay: ₹1,000,000.00 - ₹1,800,000.00 per year

Work Location: Remote