AI Engineer

Role Overview

We are seeking an AI Data Engineer who thrives at the intersection of Data Engineering

and Autonomous AI. You will move beyond traditional ETL to build "AI-Ready" data pipelines

and Agentic systems. Your role is two-fold:

1. CoE Accelerator Development: Architect and build internal frameworks and

autonomous agents that automate complex data lifecycle tasks.

2. Client Delivery: Partner with clients to design and deploy sophisticated RAG

(Retrieval-Augmented Generation) systems and Agentic workflows that can reason,

plan, and execute data operations independently.

Key Responsibilities

● Agentic Workflow Development: Design and deploy autonomous agents (using

LangGraph, AutoGen, CrewAI) capable of orchestrating complex, multi-step data

tasks, such as self-healing pipelines, automated data quality remediation, or

autonomous SQL generation and execution.

● AI-Ready Data Pipelines: Architect robust pipelines using PySpark and Databricks

to transform data into high-quality vectors and knowledge graphs optimized for

Agentic memory and reasoning.

● Accelerators & Frameworks: Develop and maintain modular, reusable "Data

Accelerators" that standardize Agentic orchestration, evaluation, and cost-monitoring

for our CoE.

● Vector Database Management: Engineer, deploy, and manage vector indices (e.g.,

Databricks Vector Search, Pinecone) to serve as the long-term memory for AI

agents.

● LLMOps & Monitoring: Implement observability frameworks to track agent

performance, reasoning accuracy, and token costs. Integrate MLflow for experiment

tracking.

● Strategic Collaboration: Act as a subject matter expert for the Data Practice CoE,

contributing to technical whitepapers and the adoption of cutting-edge Agentic

architectures.

Technical Requirements

● Core Engineering: Expert-level proficiency in Python, PySpark, and SQL.

● Databricks Mastery: Hands-on expertise with the full Databricks ecosystem: Unity

Catalog, Delta Live Tables (DLT), Workflows, and Serverless compute.

● Agentic & AI Orchestration: Strong experience building RAG pipelines and

Agentic workflows using LangGraph, CrewAI, AutoGen, or LlamaIndex. This is

the key differentiator for this role.

● Vectorization & Embeddings: Understanding of embedding models, chunking

strategies, and the lifecycle of managing vector datasets for enterprise AI.

● Cloud Architecture: Familiarity with deploying AI-driven data solutions on AWS,

Azure, or GCP.

● Tools & Methodologies: Experience with CI/CD (Git/GitHub Actions),

containerization (Docker), and test-driven development.

Preferred Qualifications

● Agentic Expertise (Huge Plus): Demonstrable experience in building autonomous

agents that can troubleshoot, reason, or perform complex analytical tasks with

minimal human intervention.

● Certifications: Databricks Certified Data Engineer Professional, Azure/AWS AI

Engineer associate certifications.

● Full-Stack GenAI: Experience with frontend frameworks (Streamlit/Flask) to build

rapid prototypes/PoCs of data accelerators.

● Governance: Familiarity with data security, PII masking, and access control models

within an AI/Agentic context.