data engineer (pyspark) Jobs in Bangalore

data engineer (pyspark)

lead data  engineer

lead data scientist

lead data engineer

MakeMyTrip

<div>
 
 
 Role:
 
 Lead Data Engineer
 
 
 
 Location:
 
 Bangalore
 
 
 
 Years of Experience:
 
 6 -8 Years
 
 
 
 About the function:
 
 
 
 At MakeMyTrip (MMT), technology is at the heart of everything we do. As a leading player in the travel industry, we leverage cutting-edge solutions like AI, machine learning, and cloud infrastructure to create seamless and innovative travel experiences. We prioritize a tech-first mindset, where our engineering teams lead the charge in transforming the way people travel.
 
 
 Our culture encourages collaboration, creativity, and thought leadership, empowering engineers to challenge the status quo and develop impactful, game-changing solutions. At MMT, you'll work with the latest technologies, continually push boundaries, and have the chance to shape the future of travel.
 
 
 Join us and become part of a forward-thinking, tech-driven organization that thrives on innovation and continuous learning.
 
 
 
 
 
 
 About the team:
 
 
 
 MakeMyTrip as India&rsquo;s leading online travel company and provides petabytes of raw data which is helpful for business growth, analytical and machine learning needs.
 
 
 
 
 
 Data Platform Team is a horizontal function at MakeMyTrip to support various LOBs (Flights, Hotels, Holidays, Ground) and works heavily on streaming datasets which powers personalized experiences for every customer from recommendations to in-location engagement.
 
 
 
 
 
 Our team's key responsibilities are:
 
 <ol>
 <li>
 Design, construct, and maintain robust data systems and architectures
 </li>
 <li>
 Develop and optimize data capture, storage, processing, serving, and querying platforms
 </li>
 <li>
 Create data products for personalization, recommendation, customer segmentation, and intelligence
 </li>
 <li>
 Enhance our Measurement platform for A/B experimentation
 </li>
 <li>
 Contribute to our Feature Store, an internal unified data analytics platform
 </li>
 <li>
 Participate in the development of our next-generation Travel Planner using Generative AI and Multi-Agent frameworks.
 </li>
 <li>
 Implement and optimize data solutions for travel-specific use cases, such as:
 </li>
 <li>
 Analyzing cross-city travel patterns to extend trip recommendations
 </li>
 <li>
 Identifying correlations between hotel bookings in different areas to suggest complementary destinations
 </li>
 </ol>
 
 
 
 
 Required Skills and Experience:
 
 <ul>
 <li>
 Extensive experience working with large data sets with hands-on technology skills to design and build robust data architecture
 </li>
 <li>
 At least 6+ years of hands-on experience in Spark/BigData Tech stack
 </li>
 <li>
 Expertise in stream processing engines like Spark Structured Streaming or Apache Flink
 </li>
 <li>
 Analytical processing on Big Data using Spark
 </li>
 <li>
 At least 4+ years of experience in Scala. Experience with Python will be a plus.
 </li>
 <li>
 Hands-on administration, configuration management, monitoring, performance tuning of Spark workloads, Distributed platforms, and JVM based systems
 </li>
 <li>
 At least 2+ years of cloud deployment experience &ndash; AWS | Azure | Google Cloud Platform
 </li>
 <li>
 2 or more product deployments of big data technologies &ndash; Business Data Lake, NoSQL databases etc
 </li>
 <li>
 Awareness and decision-making ability to choose among various big data, NoSQL, analytics tools and technologies
 </li>
 <li>
 Should have experience in architecting and implementing domain centric big data solutions
 </li>
 <li>
 Ability to frame architectural decisions and provide technology leadership &amp; direction
 </li>
 <li>
 Excellent problem solving, hands-on engineering, and communication skills
 </li>
 </ul>
 
 
 
 
 
 
 
 At MakeMyTrip, we're committed to innovation and excellence in the travel industry. Join us in shaping the future of travel through data-driven solutions and advanced technologies. If you're passionate about leveraging data to create exceptional travel experiences, we want to hear from you!
 
</div>

Lead Data  Engineer

Virtusa

<div>
 
 About The Role
 
 
 
 We are seeking a highly skilled Data Engineer with deep expertise in PySpark and the Cloudera Data Platform (CDP) to join our data engineering team. As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines that ensure high data quality and availability across the organization. This role requires a strong background in big data ecosystems, cloud-native tools, and advanced data processing techniques.
 
 
 The ideal candidate has hands-on experience with data ingestion, transformation, and optimization on the Cloudera Data Platform, along with a proven track record of implementing data engineering best practices. You will work closely with other data engineers to build solutions that drive impactful business insights.
 
 
 
 Responsibilities
 
 
 
 Data Pipeline Development: Design, develop, and maintain highly scalable and optimized ETL pipelines using PySpark on the Cloudera Data Platform, ensuring data integrity and accuracy.
 
 
 Data Ingestion: Implement and manage data ingestion processes from a variety of sources (e.g., relational databases, APIs, file systems) to the data lake or data warehouse on CDP.
 
 
 Data Transformation and Processing: Use PySpark to process, cleanse, and transform large datasets into meaningful formats that support analytical needs and business requirements.
 
 
 Performance Optimization: Conduct performance tuning of PySpark code and Cloudera components, optimizing resource utilization and reducing runtime of ETL processes.
 
 
 Data Quality and Validation: Implement data quality checks, monitoring, and validation routines to ensure data accuracy and reliability throughout the pipeline.
 
 
 Automation and Orchestration: Automate data workflows using tools like Apache Oozie, Airflow, or similar orchestration tools within the Cloudera ecosystem.
 
 
 Education and Experience
 
 
 Bachelors or Masters degree in Computer Science, Data Engineering, Information Systems, or a related field.
 
 
 3+ years of experience as a Data Engineer, with a strong focus on PySpark and the Cloudera Data Platform.
 
 
 Technical Skills
 
 
 PySpark: Advanced proficiency in PySpark, including working with RDDs, DataFrames, and optimization techniques.
 
 
 Cloudera Data Platform: Strong experience with Cloudera Data Platform (CDP) components, including Cloudera Manager, Hive, Impala, HDFS, and HBase.
 
 
 Data Warehousing: Knowledge of data warehousing concepts, ETL best practices, and experience with SQL-based tools (e.g., Hive, Impala).
 
 
 Big Data Technologies: Familiarity with Hadoop, Kafka, and other distributed computing tools.
 
 
 Orchestration and Scheduling: Experience with Apache Oozie, Airflow, or similar orchestration frameworks.
 
 
 Scripting and Automation: Strong scripting skills in Linux.
</div>

Data Engineer (Pyspark)

Pocket FM

<div>
 
 Senior Software Engineer-Backend
 
 
 Location: Bengaluru
 
 
 Experience: 3&ndash;5 years
 
 
 
 
 
 About Pocket FM
 
 
 Pocket FM, founded in 2018, is India&rsquo;s leading audio storytelling platform, transforming the way millions consume stories. Offering high-quality serialized content across genres such as Romance, Drama, Thriller, Fantasy, Sci-Fi, and Mythology in eight languages, Pocket FM has built a strong global presence with over 200 million listeners worldwide. With users spending an average of 120 minutes daily on the platform, it has emerged as one of the fastest-growing audio platforms, rapidly expanding its reach across the US, Europe, LATAM, and Southeast Asia.
 
 
 
 
 
 Our unique model combines free listening with micropayments for premium content, powering strong business growth. In FY25, we reached an ARR of INR 2,000 crore, with over 100,000 hours of content on the platform. We're also at the forefront of innovation, leveraging AI-generated content to scale efficiently.
 
 
 
 
 
 Role Overview:
 
 
 We are hiring for a Senior Software Engineer &ndash; Backend who will be responsible for building high-performance APIs, scalable microservices, and robust infrastructure powering our audio platform.
 
 
 This is a high-ownership role that requires strong technical skills, problem-solving abilities, and a mindset to work in a fast-paced product environment.
 
 
 
 
 
 Key Responsibilities:
 
 <ul>
 <li>
 Design, build, and maintain scalable backend services using Python or Golang
 </li>
 <li>
 Develop and manage RESTful APIs for mobile/web platforms and internal tools
 </li>
 <li>
 Collaborate closely with product, design, and analytics teams to deliver features at scale
 </li>
 <li>
 Optimize performance, ensure high availability, and handle large-scale traffic
 </li>
 <li>
 Write clean, modular, and testable code adhering to software best practices
 </li>
 <li>
 Lead debugging efforts, production issue resolution, and post-mortem analysis
 </li>
 <li>
 Participate in code reviews and continuously drive technical improvements
 </li>
 </ul>
 
 
 
 
 Skills &amp; Qualifications:
 
 <ul>
 <li>
 3&ndash;5 years of backend development experience in Python or Golang
 </li>
 <li>
 Strong understanding of data structures, algorithms, OOP, operating systems, and networking
 </li>
 <li>
 Proven experience developing and managing REST APIs
 </li>
 <li>
 Proficient in working with databases like PostgreSQL, MongoDB, and Redis
 </li>
 <li>
 Hands-on experience with cloud platforms &ndash; AWS, GCP, or Azure
 </li>
 <li>
 Experience with Docker or other container technologies
 </li>
 <li>
 Familiarity with CI/CD pipelines and deployment best practices
 </li>
 <li>
 Experience with monitoring and logging tools like Prometheus, Grafana, or ELK stack
 </li>
 <li>
 Experience with Git and version control workflows
 </li>
 <li>
 Contributions to open-source projects or a strong personal backend project portfolio
 </li>
 <li>
 Bachelor&rsquo;s degree in Computer Science, Engineering, or a related technical discipline
 </li>
 <li>
 Strong communication skills and a proactive, ownership-driven mindset
 </li>
 </ul>
 
 
 
 
 You can get more updates, insights and everything behind the scenes at Pocket FM here -
 
 Pocket FM
 
 
</div>

Senior Software Engineer - Backend

<div>
 
 Introduction
 
 
 
 In this role, you'll work in one of our IBM Consulting Client Innovation Centers (Delivery Centers), where we deliver deep technical and industry expertise to a wide range of public and private sector clients around the world. Our delivery centers offer our clients locally based skills and technical expertise to drive innovation and adoption of new technology.
 
 
 
 Your Role And Responsibilities
 
 
 
 <ul>
 <li>
 As an Data Engineer at IBM you will harness the power of data to unveil captivating stories and intricate patterns. You'll contribute to data gathering, storage, and both batch and real-time processing.
 </li>
 <li>
 Collaborating closely with diverse teams, you'll play an important role in deciding the most suitable data management systems and identifying the crucial data required for insightful analysis. As a Data Engineer, you'll tackle obstacles related to database integration and untangle complex, unstructured data sets.
 </li>
 <li>
 In this role, your responsibilities may include:
 </li>
 <li>
 Implementing and validating predictive models as well as creating and maintain statistical models with a focus on big data, incorporating a variety of statistical and machine learning techniques
 </li>
 <li>
 Designing and implementing various enterprise search applications such as Elasticsearch and Splunk for client requirements
 </li>
 <li>
 Work in an Agile, collaborative environment, partnering with other scientists, engineers, consultants and database administrators of all backgrounds and disciplines to bring analytical rigor and statistical methods to the challenges of predicting behaviours.
 </li>
 <li>
 Build teams or writing programs to cleanse and integrate data in an efficient and reusable manner, developing predictive or prescriptive models, and evaluating modelling results
 
 
 </li>
 </ul>
 
 Preferred Education
 
 
 
 Master's Degree
 
 
 
 Required Technical And Professional Expertise
 
 
 
 <ul>
 <li>
 Proof of Concept (POC) Development: Develop POCs to validate and showcase the feasibility and effectiveness of the proposed AI solutions.
 </li>
 <li>
 Help in showcasing the ability of Gen AI code assistant to refactor/rewrite and document code from one language to another
 </li>
 <li>
 Document solution architectures, design decisions, implementation details, and lessons learned.
 </li>
 <li>
 Stay up to date with the latest trends and advancements in AI, foundation models, and large language models.
 </li>
 <li>
 Evaluate emerging technologies, tools, and frameworks to assess their potential impact on solution design and implementation
 
 
 </li>
 </ul>
 
 Preferred Technical And Professional Experience
 
 
 
 <ul>
 <li>
 Experience and working knowledge in COBOL &amp; JAVA would be preferred
 </li>
 <li>
 Having experience in Code generation, code matching &amp; code translation leveraging LLM capabilities would be a Big plus
 </li>
 <li>
 Demonstrate a growth mindset to understand clients' business processes and challenges
 </li>
 </ul>
</div>

Data Scientist-Advanced Analytics

MKS Inc.

<div>
 
 A Day in Your Life at MKS:
 
 
 
 As a
 
 Technical Data Compliance Engineer/Analyst
 
 at MKS Global Capability Center Bangalore, you will partner with R&amp;D teams across MKS PSD Business Units to manage technical data of complex high-precision positioning systems to ensure design documentation in compliance with MKS standards and international regulations.
 
 
 
 You Will Make an Impact By:
 
 
 
 <ul>
 <li>
 Technical data management of electromechanical parts and assemblies, including regular communication with suppliers and R&amp;D teams.
 </li>
 <li>
 Design quality and regulatory compliance (RoHS, REACH, etc.)
 </li>
 <li>
 Manage, generate, and implement ECO documentation changes. ERP system of record data entry and maintenance.
 </li>
 <li>
 CAD/manufacturing drawing review.
 
 
 </li>
 </ul>
 
 Skills You Bring:
 
 
 
 <ul>
 <li>
 3D CAD experience with SolidWorks and/or AutoCAD with fundamental drafting proficiency and knowledge of ANSI Y14.5 GD&amp;T
 </li>
 <li>
 Working knowledge of safety and regulatory compliance for mechanical and electrical hardware
 </li>
 <li>
 Experience with mechanical and electrical hardware designs, product BOMs, and Engineering Change Order process, and new product introduction.
 </li>
 <li>
 Experience with ERP systems and data entry (SAP preferred)
 </li>
 <li>
 Excellent verbal and written communication skills (French speaking preferred)
 
 
 </li>
 </ul>
</div>

Mechanical Engineer Analyst

Coursera

<div>
 
 About Coursera
 
 
 
 Coursera was founded in 2012 by Stanford professors Andrew Ng and Daphne Koller to make world-class learning accessible to everyone, everywhere. Today, over 190 million learners and 375+ university and industry partners use our platform to gain skills in fields like AI, data science, technology, and business. As a Delaware public benefit corporation and Certified B Corp, we&rsquo;re driven by the belief that learning can transform lives through learning.
 
 
 
 Why Join Us
 
 
 
 At Coursera, we&rsquo;re looking for inventors, innovators, and lifelong learners ready to shape the future of education. You&rsquo;ll help build global programs and tools that power online learning for millions turning bold ideas into real impact. People who thrive here are customer-first builders who move fast, simplify ruthlessly, and iterate relentlessly on the metrics that matter.
 
 
 We&rsquo;re a globally distributed team and let you choose the best way you work, whether it's from home, a Coursera hub, or a co-working space near you. Our virtual hiring and onboarding make it easy to join us and start making an impact from anywhere. If you&rsquo;re ready to make a global impact, scale unique products exclusive to Coursera, and expand your career horizons, apply below.
 
 
 
 Job Overview:
 
 
 
 We are looking for an experienced and strategic Director of Content Optimization &amp; Design who will stand up and lead the India-based Content Optimization, Design Studio, Learning Analytics, and Quality functions to accelerate our Content production and optimization engine. This role will be responsible for massively scaling content throughput, reducing unit costs, accelerating time-to-launch, and safeguarding quality at scale. You will operationalize a hybrid resourcing model (in-house, vendors, surge contractors, AI tooling) to deliver &gt;50% of planned launches while improving learner engagement and completion outcomes.
 
 
 The ideal candidate will have a proven track record in content design and optimization, with the ability to drive efficiencies, ensure adherence to quality standards, and foster strategic partnerships.
 
 
 
 Responsibilities
 
 
 
 <ul>
 <li>
 Build and lead an India organization across Content Optimization &amp; Design (Instructional Designers, Solutions Engineer etc. ), Learning Analytics and Quality. Own end-to-end production and optimization throughput targets for short/long courses and specializations; deploy 24/5 workflows and tight vendor alignment to pull revenue forward.
 </li>
 <li>
 Stand up and run a mission-critical QA system that safeguards quality and velocity across all launch types. Drive catalog-scale optimizations targeting monthly throughput at scale; prioritize work via ticketing and initiatives in partnership with global stakeholders.
 </li>
 <li>
 Reduce unit costs and cycle times via process engineering, vendor strategy, playbooks, automation, and AI-assisted tooling. Lead learning analytics with LXDs to run experiments, codify best practices, and inform roadmap decisions.
 </li>
 <li>
 Oversee systems and business process infrastructure in tools such as Salesforce, ServiceNow, and Airtable; ensure data integrity and operational reporting. Partner closely with Vendor Management, Learning/Production, and Content &amp; Credentials leadership to maintain strategic alignment and regional relationship health.
 </li>
 <li>
 Manage capacity planning, headcount, budgets, and OKRs; ensure engagement, development, and retention of India-based teams. Establish clear SLAs, SOPs, and governance for intake, triage, QA, launches, optimizations, and incident response.
 </li>
 <li>
 Track market trends and conduct benchmarking to identify opportunities for innovation and cost savings.
 
 
 </li>
 </ul>
 
 Basic Qualifications
 
 
 
 <ul>
 <li>
 15+ years in content production, instructional design operations, or scaled program delivery; 5+ years leading multi-disciplinary teams. Proven track record scaling high-volume content operations with vendors and contractors; strong vendor management and negotiation.
 </li>
 <li>
 Builder mindset with expertise in learning design/production pipelines, QA systems, and catalog optimization at scale. Data-driven operator with experimentation mindset; comfortable with analytics and A/B testing frameworks.
 </li>
 <li>
 Strong process design and change management skills; ability to implement SOPs, SLAs, and continuous improvement and experience building or administering operational systems (e.g., Salesforce, ServiceNow, Airtable) and workflow automation.
 </li>
 <li>
 Familiarity with labs/tech-enabled learning and the boundary between ID and engineering; fluency in collaborating with solutions engineers.
 </li>
 <li>
 Excellent stakeholder management across product, content, analytics, engineering, and regional partner teams.
 </li>
 <li>
 Comfortable leading in a hybrid, fast-paced environment with global time zones
 
 
 </li>
 </ul>
 
 Preferred Qualifications
 
 
 
 <ul>
 <li>
 MBA or advanced degree preferredExperience in edtech or large-scale marketplace/content ecosystems.Background in AI-enabled content workflows and post-production automation.Prior leadership of India-based delivery teams serving global stakeholders.
 
 
 </li>
 </ul>
 
 
 If this opportunity interests you, you might like these courses on Coursera:
 
 
 
 
 <ul>
 <li>
 Strategic Leadership and Management
 
 
 </li>
 </ul>
 Coursera is an Equal Opportunity Employer committed to building a welcoming and inclusive workplace. We consider all qualified applicants without regard to legally protected characteristics and provide reasonable accommodations upon request at accommodations@coursera.org. Learn more in our CCPA Applicant Notice and GDPR Recruitment Notice.
 
 
</div>

Director, Content Optimization & Design

<div>
 We are seeking a skilled Python Engineer to join our dynamic team. The ideal candidate will be responsible for developing robust web frameworks, building backend code to efficiently serve datasets, and performing data transformations to support our data-driven applications. This role requires a deep understanding of Python, web development frameworks, data processing, and database management
 
 
 Core Python: Be proficient in the concepts and features of the Python language , experienced with web frameworks like Flask and FastAPI for developing RESTful APIs, and skilled in asynchronous programming using asyncio. Experience with handling concurrent connections and optimizing IO operations. Should have a deep understanding of microservices architecture and be capable of writing clean, efficient, and maintainable code.
 
 
 Security Best Practices and Requirements
 
 
 Understanding of security practices for microservices, including authentication, authorization, and encryption. Experience with securing APIs using OAuth, JWT, and API gateways.
 
 
 Web Framework Development
 
 
 Design, develop, and maintain scalable web frameworks using Python.
 
 
 Collaborate with front-end developers to integrate user-facing elements with server-side logic.
 
 
 Ensure the web frameworks are secure, performant, and maintainable.
 
 
 Have a sound knowledge of one or more web frameworks that are based on Python, such as Django, Flask, Pyramid, etc. Should have the know how to use the framework's features, such as routing, templating, database integration, authentication, testing.
 
 
 Object Relational Mappers (ORMs): Develop backend systems to serve datasets through APIs and other interfaces.
 
 
 Implement data access layers, integrating with various databases and data storage solutions.
 
 
 Optimize backend code for performance, scalability, and reliability.
 
 
 Be able to use ORMs to interact with databases using Python objects. ORMs abstract away the low-level SQL queries and provide a high-level interface to manipulate data. Some popular ORMs for Python are SQL Alchemy, Django ORM, etc. Should know how to define models, perform CRUD operations, query data, handle transactions.
 
 
 Data Transformations and Database Management: Design and maintain database schemas to support application requirements.
 
 
 Optimize database performance and ensure efficient data retrieval.
 
 
 Implement and manage data storage solutions, both relational and NoSQL.
 
 
 Design and implement data transformation pipelines to process and prepare data for analysis.
 
 
 Utilize ETL (Extract, Transform, Load) processes to clean, aggregate, and enrich data.
 
 
 Ensure data integrity, quality, and consistency across all transformation processes.
 
 
 Other Skills:Should also have general skills that are fundamental to any developer, such as:
 
 
 Debugging and testing: Should be able to use tools and techniques to find and fix errors in code, such as logging, breakpoints, unit testing, integration testing
</div>

Python Developer

Explore 80+ Lead data engineer in Bangalore this November. Find roles across top companies and apply now to boost your career.