Qureos Mentor

Trantor Software Pvt. Ltd

Title - Lead Data EngineerExperience - 10+ yearsLocation - Bangalore (Hybrid)About Role:This role also requires a solid understanding of Large Language Models (LLMs) and GenAI data ecosystems, enabling the development of high-quality datasets and retrieval-ready pipelines for AI-powered applications. As a technical leader, you will mentor engineering teams, establish best practices, and collaborate closely with architects, data scientists, and business stakeholders to deliver robust data solutions.Key Responsibilities1. Lead the architecture, development, and optimization of scalable data platforms supporting both batch and streaming workloads.2. Design, implement, and maintain ETL/ELT pipelines for data ingestion, transformation, and curation using the Databricks Lakehouse Platform and medallion architecture.3. Define data models, schemas, and storage strategies for data lakes and data warehouses to support analytics, reporting, machine learning, and GenAI initiatives.4. Develop and curate high-quality datasets, feature engineering pipelines, and retrieval-ready data stores, including embeddings and vector-based data structures, for LLM-powered applications.5. Establish engineering standards, coding best practices, code review processes, and CI/CD pipelines to ensure maintainable and reliable solutions.6. Build and automate end-to-end data workflows using orchestration tools such as Apache Airflow or equivalent platforms.7. Lead migrations from legacy on-premises or cloud-based data warehouses to modern cloud-native and lakehouse architectures.8. Optimize performance and cost by implementing effective partitioning, caching, compute tuning, and distributed processing strategies.9.Implement robust data governance, security, lineage, and access control frameworks aligned with organizational compliance requirements.10. Build monitoring, logging, alerting, and data quality frameworks to ensure reliability and proactive issue resolution.11. Mentor and guide data engineers while collaborating with architects, data scientists, and business stakeholders to translate business requirements into scalable technical solutions.12. Participate in and lead Agile ceremonies, including sprint planning, stand-ups, retrospectives, and technical reviews.Required Qualifications10+ years of hands-on experience in data engineering, including leadership of enterprise-scale data platform initiatives.Strong expertise with the Databricks Lakehouse Platform, including Delta Lake, Delta Live Tables, Databricks Workflows, and Unity Catalog.Proven experience implementing the medallion (Bronze/Silver/Gold) architecture for enterprise data platforms.Deep knowledge of distributed data processing using Apache Spark, including PySpark and Spark SQL.Extensive experience building scalable ETL/ELT pipelines for both batch and streaming data processing.Expert proficiency in Python and SQL for data engineering, transformation, validation, and pipeline development.Strong experience designing and managing data lakes and data warehouses using dimensional and lakehouse modeling techniques.Practical understanding of Large Language Models (LLMs) and GenAI concepts, including prompts, embeddings, vector databases, Retrieval-Augmented Generation (RAG), and supporting data pipelines.Hands-on experience with at least one major cloud platform (AWS, Azure, or GCP) and its core data services.Demonstrated success leading migrations from legacy platforms such as Hadoop or traditional data warehouses to modern cloud and lakehouse environments.Strong expertise in distributed computing, data partitioning, and performance optimization techniques.Experience implementing data security, governance, lineage, encryption, IAM, and metadata management.Solid understanding of object-oriented programming principles, software design patterns, and CI/CD practices.Experience working within Agile/Scrum environments and mentoring engineering teams.Excellent analytical, problem-solving, stakeholder management, and communication skills.Preferred QualificationsExperience developing LLM-powered applications or AI data pipelines using frameworks such as LangChain, LlamaIndex, or similar technologies.Hands-on experience with vector databases, including Pinecone, Weaviate, FAISS, or pgvector.Industry certifications in Databricks, AWS, Azure, or Google Cloud Platform.Experience with streaming technologies such as Spark Structured Streaming or Apache Kafka.Familiarity with Infrastructure as Code (Terraform) and DevOps tools such as Git, Jenkins, or Azure DevOps.Exposure to MLOps and LLMOps practices, including model deployment and lifecycle management.Experience working with business intelligence and visualization tools such as Power BI, Tableau, or Amazon QuickSight.Pay: ₹3,000,000.00 - ₹3,500,000.00 per yearApplication Question(s):<ul><li>What is your notice period?</li></ul>Experience:<ul><li>Data Engineering: 10 years (Required)</li><li>Large language Model: 4 years (Required)</li><li>cloud technology: 4 years (Required)</li></ul>Work Location: Hybrid remote in Bangalore City, Bengaluru, Karnataka

Lead Data Engineer

Join Trantor Software Pvt. Ltd and explore 1   exciting career opportunities. They're hiring for various roles with options for remote or on-site work. Apply today and become part of their innovative team.

Trantor Software Pvt. Ltd

About Company

Lead Data Engineer