Senior Data Engineer

Job overview:

We are seeking a Senior Data Engineer with strong expertise in building and optimizing large-scale data platforms to support advanced analytics, AI, and Generative AI (GenAI) use cases. The ideal candidate will have hands-on experience with Snowflake and Databricks, combined with the ability to design scalable data pipelines and enable AI/ML integration. This role requires strong data engineering fundamentals, cloud expertise, and collaboration with data scientists, AI architects, and business stakeholders.

Key Responsibilities:

• Design, build, and maintain large-scale data pipelines for batch and real-time processing.
• Develop and optimize data models, ETL/ELT processes, and data warehouses/lakes on Snowflake and Databricks.
• Ensure enterprise data quality, lineage, governance, and security.
• Apply best practices for cost efficiency, scalability, and observability in cloud-native platforms.
• Collaborate with AI/ML teams to prepare datasets for training, fine-tuning, and evaluation.
• Integrate pipelines with AI/GenAI workflows including LLMs, embeddings, vector databases, and RAG.
• Enable feature engineering and feature store integration for ML applications.
• Support responsible AI by ensuring datasets are accurate, diverse, and compliant.
• Work with architects, data scientists, and product owners on end-to-end solutions.
• Mentor junior engineers, enforce coding standards, and create reusable frameworks.
• Contribute to design sessions, architecture reviews, and sprint planning.

Key Skills:

• 7+ years of experience in data engineering with large-scale, enterprise-grade systems.
• Strong expertise with Snowflake (data modeling, query optimization, performance tuning).
• Hands-on experience with Databricks (PySpark, Delta Lake, Unity Catalog).
• Proficiency in SQL, Python, and Spark for data processing and transformation.
• Experience with cloud platforms (AWS, Azure, GCP) and associated data services.
• Knowledge of data governance, lineage, metadata management, and security best practices.
• Familiarity with CI/CD pipelines, orchestration tools (Airflow, Dagster, Prefect), and DevOps practices.

Preferred Qualifications

• Exposure to AI/GenAI use cases, including RAG pipelines, embeddings, and model-ready data pipelines.
• Experience with vector databases (Pinecone, FAISS, Weaviate, Milvus) and feature stores.
• Familiarity with data quality frameworks (Great Expectations, dbt tests, Monte Carlo).
• Prior experience supporting real-time streaming data (Kafka, Kinesis, EventHub).
• Certifications in Snowflake, Databricks, or cloud platforms (AWS, Azure, GCP).