Senior AI Data Engineer

Project description

We are looking for AI Engineers for a strategic initiative to build Agentic AI use cases around Trading Surveillance. Ideal candidate is someone with strong pyspark background coupled with proven experience build Agentic AI pipelines for code conversion from Q KDB to Pyspark.

Responsibilities

Design, develop, and maintain full-stack Python applications with modern frontend frameworks

Build and optimize RAG (Retrieval-Augmented Generation) systems for AI applications

Create and implement efficient vector databases and knowledge stores

Develop APIs that connect frontend interfaces with backend AI services

Implement and maintain CI/CD pipelines for AI applications

Monitor application performance and troubleshoot issues in production

Design and maintain data pipelines, ETL/ELT workflows, and data infrastructure to support AI and analytics workloads

Skills

Must have

Minimum of 5 years of experience in AI/ML solution implementation (mandatory)

Minimum of 3 years of experience in Pyspark processing high volume data (mandatory)

AI & ML Skills:

Multi-Agent System Design & Orchestration

Full-stack AI Integration (Frontend to Backend)

Advanced Prompt Engineering

LLM Observability, Evaluation & Monitoring (LLMOps)

Data Engineering Skills:

Proficient in Pyspark coding and concepts. Example: Lazy evaluation, window functions, spark architecture and performance finetuning (ex: data skew reduction (salting etc), joins (broadcast, merge sort joins ect)

SQL proficiency and query optimization

Main Technologies:

Pyspark

Git hub Copilot

Nice to have

-

Other

Languages

English: C2 Proficient

Seniority

Senior

Chennai, India

Req. VR-121109

AI/ML

BCM Industry

31/08/2026

Req. VR-121109