Senior Data Engineer

Profile: Data Engineer

Skills: Azure Data Factory, SQL, Data Migration

Experience: 7+ Years

Time: 3 Hrs Per Day (Between 7 PM IST to 1 AM IST)

Location: Remote

Role Overview

We are seeking an experienced Data Engineer to lead end-to-end migration of enterprise data from SQL-based systems into Microsoft Dataverse using Azure Data Factory (ADF), ensuring data quality, integrity, governance, and smooth transition for business applications built on the Power Platform.

Key Responsibilities

1. Data Migration Planning & Strategy

• Assess existing SQL data sources, schemas, relationships, and dependencies.
• Develop detailed migration plans, mapping documents, and transformation rules.
• Define migration approach (incremental, bulk load, delta loads, cutover strategy).

2. Azure Data Factory Development

• Design and build ADF pipelines, dataflows, activities, and triggers for migration.
• Implement ETL/ELT logic using ADF, Azure Functions, Logic Apps, or Dataverse APIs.
• Configure staging layers in Azure SQL, Data Lake, or other storage as needed.
• Optimize pipelines for performance, cost, and reliability.

3. Dataverse Integration

• Configure Dataverse tables, columns, relationships, alternate keys, and business rules.
• Perform data transformations that align SQL structures to Dataverse schema.
• Execute data loads into Dataverse using ADF connectors, Power Platform connectors, or Azure Dataflows.
• Resolve constraints, lookups, and referential integrity issues during loads.

4. Data Quality, Validation & Testing

• Run data profiling, cleansing, deduplication, and quality checks before migration.
• Validate data using SQL queries, Dataverse views, and Power Apps/Power BI.
• Perform reconciliation and sign-off with business stakeholders.
• Document defects and support remediation activities.

5. Deployment & Cutover

• Automate migration through CI/CD (Azure DevOps).
• Execute dry runs, mock migrations, and production cutover.
• Monitor pipeline runs, logs, exceptions, and performance metrics.
• Provide hypercare support post go-live.

6. Documentation & Compliance

• Maintain detailed technical documentation (mapping sheets, migration strategy, runbooks).
• Ensure compliance with governance, security, and data protection standards.
• Collaborate with functional teams, architects, and Power Platform developers.

Required Skills:

• Strong expertise in SQL Server (queries, stored procedures, T-SQL, schema design).
• Hands-on experience with Azure Data Factory (ADF pipelines, Dataflows, integration runtimes).
• Experience loading data into Microsoft Dataverse / CDS.
• Knowledge of Power Platform (Power Apps, Power Automate) is a plus.
• Experience with Azure SQL, Azure Data Lake, Azure Functions, Logic Apps.
• Understanding of REST APIs, Dataverse Web API, OData.
• Familiarity with ETL/ELT frameworks and best practices.
• Understanding of data warehousing, metadata management, and data governance.

Job Type: Part-time

Pay: From ₹30,000.00 per month

Expected hours: 15 per week

Benefits:

• Work from home

Experience:

• Data Engineer: 7 years (Required)
• Azure Data factory: 4 years (Required)
• SQL: 4 years (Required)

Work Location: Remote