Senior Data Engineer

Job Summary

As a Senior Data Engineer, you will be a foundational architect, responsible for designing, building, and maintaining the robust, scalable, secure, and high-performance data infrastructure that powers our AI/ML initiatives. You will work closely with Data Scientists, AI Engineers, AIOps/MLOps Engineers, and Software Engineers to ensure the seamless flow of data from diverse government sources into our AI/ML pipelines, enabling the development and deployment of impactful intelligent applications across Abu Dhabi.

What You'll Do:

● Design, build, and maintain scalable and reliable data pipelines for ingesting, transforming, and storing large and complex datasets from various government sources (structured, semi-structured, and unstructured).

● Architect and implement data warehousing and data lake solutions on cloud platforms (primarily Azure) to provide a centralized and accessible data foundation for AI/ML and analytics.

● Develop and optimize ETL/ELT processes using industry-standard tools and technologies (e.g., Azure Data Factory, Databricks, Kafka) to ensure data quality, integrity, and efficient processing.

● Implement and manage data governance policies and procedures, ensuring data security, compliance with government regulations, and adherence to data quality standards.

● Collaborate with Data Scientists and AI Engineers to understand their data requirements and build data pipelines that meet their specific needs for model development and evaluation.

● Work closely with AIOps/MLOps Engineers to ensure seamless integration of data pipelines with AI/ML workflows and deployment processes.

● Design and implement data monitoring and alerting systems to proactively identify and resolve data quality issues and pipeline failures.

● Optimize data storage and retrieval systems for performance and cost-efficiency, leveraging appropriate database technologies (relational, NoSQL, vector, graph).

● Implement data security measures including access control, encryption, and data masking to protect sensitive government data.

● Develop and maintain data catalogs and data lineage solutions to ensure data discoverability, understand data flow, and facilitate data governance.

● Evaluate and adopt new data engineering tools and technologies to continuously improve our data infrastructure and processes.

● Troubleshoot and resolve data-related issues across the data lifecycle, ensuring data availability and reliability for AI/ML and analytics.

● Document data pipelines, data models, and data engineering processes clearly and concisely for both technical and non-technical stakeholders.

What You'll Bring:

● Bachelor's or Master's degree in Computer Science, Data Engineering, or a related technical field with a strong focus on data management and infrastructure.

● At least 3 years of hands-on experience in designing, building, and maintaining data pipelines and data infrastructure in a production environment.

● Deep understanding of data warehousing concepts, data modeling techniques (e.g., Kimball, Inmon), and data lake architectures.

● Proven experience with cloud platforms, especially Microsoft Azure, and their data engineering services (e.g., Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage, Azure Databricks).

● Extensive experience with ETL/ELT tools and technologies (e.g., Apache Spark, Apache Kafka, SQL).

● Strong proficiency in SQL and experience with various database systems, including:

○ Relational Databases: (e.g., PostgreSQL, SQL Server, Azure SQL Database).

○ NoSQL Databases: (e.g., MongoDB, Cassandra, Azure Cosmos DB).

○ Vector Databases: (e.g., Milvus, Pinecone, ChromaDB - understanding their data ingestion and retrieval aspects).

○ Graph Databases: (e.g., Neo4j, ArangoDB - understanding their data modeling and querying aspects).

● Strong scripting and programming skills in languages such as Python or Scala for data processing and automation.

● Experience with data governance frameworks, data quality management, and data security principles.

● Experience with data monitoring and alerting tools.

● Excellent problem-solving and analytical skills with the ability to diagnose and resolve complex data-related issues.

● Strong collaboration and communication skills, with the ability to work effectively with Data Scientists, AI Engineers, AIOps/MLOps Engineers, and other stakeholders.

● A proactive and detail-oriented approach with a strong focus on building robust and reliable data infrastructure.

Bonus Points For:

● Experience with data virtualization technologies.

● Experience with building real-time data pipelines.

● Knowledge of data lineage and data catalog tools (e.g., Azure Purview).

● Experience with implementing data mesh concepts.

● Familiarity with infrastructure-as-code (IaC) tools (e.g., Terraform, ARM templates) for data infrastructure provisioning.

● Certifications in relevant cloud platforms or data engineering technologies.

● Experience working with government data regulations and compliance standards.

Job Type: Full-time

Pay: From AED20,000.00 per month

Ability to commute/relocate:

• Abu Dhabi: Reliably commute or planning to relocate before starting work (Required)

Experience:

• data pipelines and data infra in a production env: 5 years (Required)
• ETL/ELT tools and tech, Apache Spark, Apache Kafka, SQL: 5 years (Required)
• AI/ML development : 3 years (Required)

License/Certification:

• Emirates ID (Required)