Data Engineer

Key Tasks:

• Build robust pipelines across bronze, silver, and gold layers using Databricks, Spark, dbt, and Azure orchestration.
• Implement data contracts, tests, and schema validation to protect quality and trust.
• Apply lineage, catalog, RBAC, and metadata to meet governance and audit needs.
• Prepare curated gold datasets, features, and embeddings for AI and BI consumption
• Monitor health and cost, meet SLAs, and keep pipelines observable and reliable

Key Responsibilities: Architectural Alignment

• Follow the enterprise reference architecture for a Lakehouse on Azure with on prem S3 compatible storage.
• Ingest from core systems using batch and streaming. Use Goldengate for CDC where available and Air byte or ADF for scheduled loads.
• Land data in bronze as faithful copies, transform to silver with conformed models, and publish gold data marts and semantic entities.
• Use Single Store or similar where low latency serving is required as guided by architecture.

Governance and Metadata Integration

• Enforce ownership and stewardship through data contracts and clear domain boundaries.
• Register assets in the catalog with business and technical metadata and document lineage end to end.
• Apply privacy by design with masking, tokenization, and least privilege access in line with PHI and PII requirements and FHIR or MDR mappings where required.
• Track quality KPIs such as freshness, completeness, and validity with automated checks.

Platform and Operational Alignment

• Use CI/CD for pipelines with environment promotion and rollback.
• Implement runbooks, dashboards, and alerts for jobs, latency, and data freshness.
• Optimize compute and storage including partitioning, file sizing, and caching to improve cost and performance.
• Collaborate with Platform Engineering on identity, secrets, network, and storage guardrails.

Cross Functional Collaboration

• Work with Lead Data Architect and AI Architect to translate patterns into concrete implementations.
• Partner with BI to publish gold entities to the semantic layer and support Power BI or Fine BI models.
• Enable AI by delivering feature ready tables, embedding jobs for retrieval augmented generation, and inference friendly data shapes with MLflow integration where appropriate.
• Align with Delivery Governance on intake, prioritization, and release readiness.

Critical Datasets and Domains: Own and operate pipelines for tier one domains under guidance from Data Architecture. Examples include Patient, Encounter, Provider, Orders and Results, Claims and Billing, Master Data such as MDM entities, Genomics such as VCF and lab omics feeds, and operational telemetry such as identity and access logs.

Academic/Skill Requirements:

• Bachelor s degree in computer science, Engineering, or related field.
• Strong SQL and dimensional or data vault modeling fundamentals.
• Proficiency in Python for data processing and automation.
• Hands on experience e.g. Databricks, Spark, and Delta Lake.
• ELT with dbt including tests and documentation.
• Working knowledge of Azure Data Factory or Databricks Workflows, Azure Storage and ADLS, Azure Active Directory, and Key Vault.
• Familiarity with CI/CD in Azure DevOps or GitHub and container basics with Docker and Kubernetes.