Data Engineer

Job Description
Cloud-Native Data Engineering on AWS

• Strong, hands-on expertise in AWS native data services: S3, Glue (Schema Registry, Data Catalog), Step Functions, Lambda, Lake Formation, Athena, MSK/Kinesis, EMR (Spark), SageMaker (inc. Feature Store)
• Comfort designing and optimizing pipelines for both batch (Step Functions) and streaming (Kinesis/MSK) ingestion.
• Data Mesh & Distributed Architectures
• Deep understanding of data mesh principles: including domain-oriented ownership, treating data as a product, and the use of federated governance models
• Experience enabling self-service platforms, decentralized ingestion, and transformation workflows.
• Data Contracts & Schema Management
• Advanced knowledge of schema enforcement, evolution, and validation (preferably AWS Glue Schema Registry/JSON/Avro)
• Data Transformation & Modelling
• Proficiency with modern ELT/ETL stack: Spark (EMR), dbt, AWS Glue, and Python (pandas)

AI/ML Data Enablement

• Designing and supporting vector stores (OpenSearch), feature stores (SageMaker Feature Store), and integrating with MLOps/data pipelines for AI/semantic search and RAG-type workloads
• Metadata, Catalog, and Lineage
• Familiarity with central cataloging, lineage solutions, and data discovery (Glue Data Catalog, Collibra, Atlan, Amundsen, etc.)
• Implementing end-to-end lineage, auditability, and governance processes.
• Security, Compliance, and Data Governance
• Design and implementation of data security: row/column-level security (Lake Formation), KMS encryption, role-based access using AuthN/AuthZ standards (JWT/OIDC), GDPR/SOC2/ISO 27001-aligned policies
• Orchestration & Observability
• Experience with pipeline orchestration (AWS Step Functions, Apache Airflow/MWAA) and monitoring (CloudWatch, X-Ray) in large-scale environments.

APIs & Integration

• API design for both batch and real-time data delivery (REST, GraphQL endpoints for AI/reporting/BI consumption)

Job Responsibilities

• Design, build, and maintain ETL/ELT pipelines to extract, transform, and load data from various sources into cloud-based data platforms.
• Develop and manage data architectures, data lakes, and data warehouses on AWS (e.g., S3, Redshift, Glue, Athena).
• Collaborate with data scientists, analysts, and business stakeholders to ensure data accessibility, quality, and security.
• Optimize performance of large-scale data systems and implement monitoring, logging, and alerting for pipelines.
• Work with both structured and unstructured data, ensuring reliability and scalability.
• Implement data governance, security, and compliance standards.
• Continuously improve data workflows by leveraging automation, CI/CD, and Infrastructure-as-Code (IaC)