data engineer (pyspark) Jobs in Bangalore

data engineer (pyspark)

principal data engineer

Virtusa

<div>
 
 About The Role
 
 
 
 We are seeking a highly skilled Data Engineer with deep expertise in PySpark and the Cloudera Data Platform (CDP) to join our data engineering team. As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines that ensure high data quality and availability across the organization. This role requires a strong background in big data ecosystems, cloud-native tools, and advanced data processing techniques.
 
 
 The ideal candidate has hands-on experience with data ingestion, transformation, and optimization on the Cloudera Data Platform, along with a proven track record of implementing data engineering best practices. You will work closely with other data engineers to build solutions that drive impactful business insights.
 
 
 
 Responsibilities
 
 
 
 Data Pipeline Development: Design, develop, and maintain highly scalable and optimized ETL pipelines using PySpark on the Cloudera Data Platform, ensuring data integrity and accuracy.
 
 
 Data Ingestion: Implement and manage data ingestion processes from a variety of sources (e.g., relational databases, APIs, file systems) to the data lake or data warehouse on CDP.
 
 
 Data Transformation and Processing: Use PySpark to process, cleanse, and transform large datasets into meaningful formats that support analytical needs and business requirements.
 
 
 Performance Optimization: Conduct performance tuning of PySpark code and Cloudera components, optimizing resource utilization and reducing runtime of ETL processes.
 
 
 Data Quality and Validation: Implement data quality checks, monitoring, and validation routines to ensure data accuracy and reliability throughout the pipeline.
 
 
 Automation and Orchestration: Automate data workflows using tools like Apache Oozie, Airflow, or similar orchestration tools within the Cloudera ecosystem.
 
 
 Education and Experience
 
 
 Bachelors or Masters degree in Computer Science, Data Engineering, Information Systems, or a related field.
 
 
 3+ years of experience as a Data Engineer, with a strong focus on PySpark and the Cloudera Data Platform.
 
 
 Technical Skills
 
 
 PySpark: Advanced proficiency in PySpark, including working with RDDs, DataFrames, and optimization techniques.
 
 
 Cloudera Data Platform: Strong experience with Cloudera Data Platform (CDP) components, including Cloudera Manager, Hive, Impala, HDFS, and HBase.
 
 
 Data Warehousing: Knowledge of data warehousing concepts, ETL best practices, and experience with SQL-based tools (e.g., Hive, Impala).
 
 
 Big Data Technologies: Familiarity with Hadoop, Kafka, and other distributed computing tools.
 
 
 Orchestration and Scheduling: Experience with Apache Oozie, Airflow, or similar orchestration frameworks.
 
 
 Scripting and Automation: Strong scripting skills in Linux.
</div>

Data Engineer (Pyspark)

MKS Inc.

<div>
 
 A Day in Your Life at MKS:
 
 
 
 As a
 
 Technical Data Compliance Engineer/Analyst
 
 at MKS Global Capability Center Bangalore, you will partner with R&amp;D teams across MKS PSD Business Units to manage technical data of complex high-precision positioning systems to ensure design documentation in compliance with MKS standards and international regulations.
 
 
 
 You Will Make an Impact By:
 
 
 
 <ul>
 <li>
 Technical data management of electromechanical parts and assemblies, including regular communication with suppliers and R&amp;D teams.
 </li>
 <li>
 Design quality and regulatory compliance (RoHS, REACH, etc.)
 </li>
 <li>
 Manage, generate, and implement ECO documentation changes. ERP system of record data entry and maintenance.
 </li>
 <li>
 CAD/manufacturing drawing review.
 
 
 </li>
 </ul>
 
 Skills You Bring:
 
 
 
 <ul>
 <li>
 3D CAD experience with SolidWorks and/or AutoCAD with fundamental drafting proficiency and knowledge of ANSI Y14.5 GD&amp;T
 </li>
 <li>
 Working knowledge of safety and regulatory compliance for mechanical and electrical hardware
 </li>
 <li>
 Experience with mechanical and electrical hardware designs, product BOMs, and Engineering Change Order process, and new product introduction.
 </li>
 <li>
 Experience with ERP systems and data entry (SAP preferred)
 </li>
 <li>
 Excellent verbal and written communication skills (French speaking preferred)
 
 
 </li>
 </ul>
</div>

Mechanical Engineer Analyst

<div>
 We are seeking a skilled Python Engineer to join our dynamic team. The ideal candidate will be responsible for developing robust web frameworks, building backend code to efficiently serve datasets, and performing data transformations to support our data-driven applications. This role requires a deep understanding of Python, web development frameworks, data processing, and database management
 
 
 Core Python: Be proficient in the concepts and features of the Python language , experienced with web frameworks like Flask and FastAPI for developing RESTful APIs, and skilled in asynchronous programming using asyncio. Experience with handling concurrent connections and optimizing IO operations. Should have a deep understanding of microservices architecture and be capable of writing clean, efficient, and maintainable code.
 
 
 Security Best Practices and Requirements
 
 
 Understanding of security practices for microservices, including authentication, authorization, and encryption. Experience with securing APIs using OAuth, JWT, and API gateways.
 
 
 Web Framework Development
 
 
 Design, develop, and maintain scalable web frameworks using Python.
 
 
 Collaborate with front-end developers to integrate user-facing elements with server-side logic.
 
 
 Ensure the web frameworks are secure, performant, and maintainable.
 
 
 Have a sound knowledge of one or more web frameworks that are based on Python, such as Django, Flask, Pyramid, etc. Should have the know how to use the framework's features, such as routing, templating, database integration, authentication, testing.
 
 
 Object Relational Mappers (ORMs): Develop backend systems to serve datasets through APIs and other interfaces.
 
 
 Implement data access layers, integrating with various databases and data storage solutions.
 
 
 Optimize backend code for performance, scalability, and reliability.
 
 
 Be able to use ORMs to interact with databases using Python objects. ORMs abstract away the low-level SQL queries and provide a high-level interface to manipulate data. Some popular ORMs for Python are SQL Alchemy, Django ORM, etc. Should know how to define models, perform CRUD operations, query data, handle transactions.
 
 
 Data Transformations and Database Management: Design and maintain database schemas to support application requirements.
 
 
 Optimize database performance and ensure efficient data retrieval.
 
 
 Implement and manage data storage solutions, both relational and NoSQL.
 
 
 Design and implement data transformation pipelines to process and prepare data for analysis.
 
 
 Utilize ETL (Extract, Transform, Load) processes to clean, aggregate, and enrich data.
 
 
 Ensure data integrity, quality, and consistency across all transformation processes.
 
 
 Other Skills:Should also have general skills that are fundamental to any developer, such as:
 
 
 Debugging and testing: Should be able to use tools and techniques to find and fix errors in code, such as logging, breakpoints, unit testing, integration testing
</div>

Python Developer

Straive

<div>
 
 
 About the Role
 
 
 
 Straive is looking for a talented and driven
 
 Consultant / Data Scientist / GenAI Engineer
 
 to join our
 
 Analytics &amp; GenAI delivery team
 
 . In this role, you will work under the guidance of the Senior Project Manager / Engagement Manager to design, develop, and deploy advanced AI/ML and Generative AI solutions for global enterprise clients. You will be part of a high-performing team, collaborating with both onshore and offshore members to build scalable, production-grade AI systems.
 
 
 
 
 
 This role is ideal for candidates from
 
 premier engineering institutes
 
 with
 
 2&ndash;3 years of relevant experience
 
 in Python development, LLM integration, and RAG workflows, along with a passion for solving complex problems in real-world business contexts.
 
 
 
 
 
 
 Key Responsibilities
 
 
 <ul>
 <li>
 Develop and maintain
 
 Python-based
 
 applications, AI/ML models, and data processing pipelines for GenAI projects.
 </li>
 <li>
 Implement
 
 Large Language Model (LLM)
 
 integrations, including
 
 Retrieval-Augmented Generation (RAG)
 
 pipelines and embedding-based search solutions.
 </li>
 <li>
 Build data ingestion and transformation workflows, working with structured and unstructured datasets.
 </li>
 <li>
 Optimize AI model performance through
 
 prompt engineering
 
 , fine-tuning, and evaluation techniques.
 </li>
 <li>
 Collaborate closely with senior team members to translate business requirements into technical solutions.
 </li>
 <li>
 Integrate AI solutions with
 
 vector databases
 
 (e.g., Cosmos DB, Pinecone, ChromaDB) and API-driven applications.
 </li>
 <li>
 (Optional) Contribute to
 
 cloud-native deployments
 
 and
 
 Azure architecture
 
 &ndash;based solutions, including containerization, CI/CD, and basic MLOps workflows.
 </li>
 <li>
 Document workflows, maintain code repositories, and follow Agile development practices.
 </li>
 </ul>
 
 
 
 
 
 Required Qualifications
 
 
 <ul>
 <li>
 
 2&ndash;3 years
 
 of relevant experience in AI/ML development, preferably in enterprise projects.
 </li>
 <li>
 Bachelor&rsquo;s or Master&rsquo;s degree in Computer Science, Data Science, AI/ML, or related field from a
 
 premier engineering institute
 
 .
 </li>
 <li>
 Proficiency in
 
 Python
 
 programming and familiarity with relevant libraries (e.g., LangChain, Hugging Face, Pandas, NumPy).
 </li>
 <li>
 Hands-on experience implementing
 
 RAG pipelines
 
 , embeddings, and vector search solutions.
 </li>
 <li>
 Understanding of
 
 LLM architectures
 
 and integration patterns.
 </li>
 <li>
 Working knowledge of
 
 SQL
 
 and data processing best practices.
 </li>
 <li>
 Basic knowledge of
 
 cloud DevOps concepts
 
 , preferably with
 
 Azure
 
 (AWS/GCP experience is also acceptable).
 </li>
 </ul>
 
 
 
 
 Preferred Skills
 
 
 <ul>
 <li>
 Exposure to
 
 agentic AI frameworks
 
 such as LangGraph, Semantic Kernel, or similar.
 </li>
 <li>
 Familiarity with ML model lifecycle management and deployment workflows.
 </li>
 <li>
 Prior experience working with cross-border teams and Agile environments.
 </li>
 </ul>
 
 
</div>

Data Scientist

SPAN

<div>
 
 Our Mission
 
 
 
 
 SPAN is enabling electrification for all ⚡
 
 
 
 We are a mission-driven company designing, building, and deploying products that electrify the built environment, reduce carbon emissions, and slow the effects of climate change.
 
 
 <ul>
 <li>
 Decarbonization is the process to reduce or remove greenhouse gas emissions, especially carbon dioxide, from entering our atmosphere.
 </li>
 <li>
 Electrification is the process of replacing fossil fuel appliances that run on gas or oil with all-electric upgrades for a cleaner way to power our lives.
 
 
 </li>
 </ul>
 At SPAN, we believe in:
 
 
 <ul>
 <li>
 Enabling homes and vehicles powered by clean energy
 </li>
 <li>
 Making electrification upgrades possible
 </li>
 <li>
 Building more resilient homes with reliable backup
 </li>
 <li>
 Designing a flexible and distributed electrical grid
 
 
 </li>
 </ul>
 
 The Role
 
 
 
 As an Software Engineer, you'll play a crucial role in ensuring the quality and reliability of our products through a combination of automated and manual testing methodologies. This is a high impact role that will work closely with SPAN&rsquo;s internal Software, Device Software, Hardware Engineering, Systems, and QA teams.
 
 
 
 Responsibilities
 
 
 
 As a Software Engineer, you will be expected to:
 
 
 <ul>
 <li>
 Develop and maintain automation framework, driver library, and consumer facing dashboards.
 </li>
 <ul>
 <li>
 Supporting DSW with HITL testing, Reliability, QA, and Systems testing
 </li>
 </ul>
 <li>
 Document defects, issues, and enhancement requests clearly and concisely.
 </li>
 <li>
 Participating in requirements review and systems engineering planning to determine high-leverage opportunities for automation.
 </li>
 <li>
 Working with the firmware team to ensure that engineering SDKs to the hardware have required capabilities for automated tests (e.g., structured log capture). Working with the firmware, systems, and hardware teams to efficiently triage issues.
 </li>
 <li>
 Commissioning, standing up, and maintaining a worker pool of hardware test cells (&lsquo;device under test&rsquo;) for DVT and production hardware. Worker pool capacity planning and commissioning of new test cells as usage and wait time increases.
 </li>
 <li>
 Reliable on-premise workflow scheduling and orchestration of test sequences on hardware, and the integration of these workflows to engineering CI/CD systems
 </li>
 <li>
 Machine specification of test scenarios and acceptance criteria, and traceability with external requirements management tools (Jama)
 </li>
 <li>
 Storage of raw log data to external object storage (S3), and structured test result data to Postgres (Currently done with Chronos).
 </li>
 <li>
 Reporting web dashboards that provide clarity around pass/fail criteria for software releases, as well as other leading, non-deterministic reliability indicators (e.g., &lsquo;crashiness&rsquo;). Ensuring that these dashboards surface critical failing cases to the engineering team and that underlying issues can be efficiently identified.
 </li>
 <li>
 Monitoring of both physical test cell and web infrastructure uptime, capacity, and reliability
 </li>
 <li>
 Identify areas for process improvement and actively contribute to enhancing software development and testing practices pertaining to automation.
 
 
 </li>
 </ul>
 
 About You
 
 
 
 
 Required Qualifications
 
 
 
 <ul>
 <li>
 Proven experience as an Associate Engineer
 </li>
 <li>
 Proficiency building reliable and maintainable tools and services in Python
 </li>
 <li>
 Proficiency in one of SPAN&rsquo;s coding language: Python and frameworks such as ReactJS, and Kotlin
 </li>
 <li>
 Strong knowledge of software testing methodologies, test case design and automated testing.
 </li>
 <li>
 Experience with CI/CD tools and Github (i.e, CircleCI, Docker).
 </li>
 <li>
 Effective communication and collaboration skills.
 </li>
 <li>
 Ability to work independently and as part of a team.
 </li>
 <li>
 Comfort building and working in greenfield automated testing frameworks and infrastructure
 </li>
 <li>
 Experience with CAN, RS485 protocols
 </li>
 <li>
 Knowledge of embedded systems
 
 
 </li>
 </ul>
 
 Bonus Qualifications
 
 
 
 <ul>
 <li>
 Experience working in a startup environment
 </li>
 <li>
 Experience with or preference for a rapid or continuous release cadence
 </li>
 <li>
 Familiarity with Agile/Scrum development processes.
 </li>
 <li>
 Experience w/ GraphQL or GRPC, RDBMS/SQL, S3 and AWS.
 </li>
 <li>
 Experience working in the Renewable Energy industry
 
 
 </li>
 </ul>
 
 Life at SPAN
 
 
 
 Our
 
 Bengaluru
 
 team plays a pivotal role in SPAN&rsquo;s continued growth and expansion. Together, we&rsquo;re driving
 
 engineering
 
 ,
 
 product development
 
 , and
 
 operational excellence
 
 to shape the future of home energy solutions.
 
 
 As part of our team in India, you&rsquo;ll have the opportunity to collaborate closely with our teams in the US and across the globe. This international collaboration fosters innovation, learning, and growth, while helping us achieve our bold mission of electrifying homes and advancing clean energy solutions worldwide.
 
 
 Our in-office culture offers the chance for dynamic interactions and hands-on teamwork, making SPAN a truly collaborative environment where every team member&rsquo;s contribution matters.
 
 
 Our climate-focused culture is driven by a team of forward-thinkers, engineers, and problem-solvers who push boundaries every day.
 
 
 <ul>
 <li>
 Do mission-driven work: Every role at SPAN directly advances clean energy adoption.
 </li>
 <li>
 Bring powerful ideas to life: We encourage diverse ideas and perspectives to drive stronger products.
 </li>
 <li>
 Nurture an innovation-first mindset: We encourage big thinking and bold action.
 </li>
 <li>
 Deliver exceptional customer value: We value hard work, and the ability to deliver exceptional customer value.
 
 
 </li>
 </ul>
 
 Benefits At SPAN India
 
 
 
 ⚡
 
 Generous paid leave
 
 
 
 
 ⚡Comprehensive
 
 
 Insurance &amp; Health Benefits
 
 
 
 ⚡
 
 Centrally located office in Bengaluru with easy access to public transit, dining, and city amenities
 
 
 
 
 Interested in joining our team? Apply today and we&rsquo;ll be in touch with the next steps!
 
 
 
</div>

Software Engineer

Pocket FM

<div>
 
 Senior Software Engineer-Backend
 
 
 Location: Bengaluru
 
 
 Experience: 3&ndash;5 years
 
 
 
 
 
 About Pocket FM
 
 
 Pocket FM, founded in 2018, is India&rsquo;s leading audio storytelling platform, transforming the way millions consume stories. Offering high-quality serialized content across genres such as Romance, Drama, Thriller, Fantasy, Sci-Fi, and Mythology in eight languages, Pocket FM has built a strong global presence with over 200 million listeners worldwide. With users spending an average of 120 minutes daily on the platform, it has emerged as one of the fastest-growing audio platforms, rapidly expanding its reach across the US, Europe, LATAM, and Southeast Asia.
 
 
 
 
 
 Our unique model combines free listening with micropayments for premium content, powering strong business growth. In FY25, we reached an ARR of INR 2,000 crore, with over 100,000 hours of content on the platform. We're also at the forefront of innovation, leveraging AI-generated content to scale efficiently.
 
 
 
 
 
 Role Overview:
 
 
 We are hiring for a Senior Software Engineer &ndash; Backend who will be responsible for building high-performance APIs, scalable microservices, and robust infrastructure powering our audio platform.
 
 
 This is a high-ownership role that requires strong technical skills, problem-solving abilities, and a mindset to work in a fast-paced product environment.
 
 
 
 
 
 Key Responsibilities:
 
 <ul>
 <li>
 Design, build, and maintain scalable backend services using Python or Golang
 </li>
 <li>
 Develop and manage RESTful APIs for mobile/web platforms and internal tools
 </li>
 <li>
 Collaborate closely with product, design, and analytics teams to deliver features at scale
 </li>
 <li>
 Optimize performance, ensure high availability, and handle large-scale traffic
 </li>
 <li>
 Write clean, modular, and testable code adhering to software best practices
 </li>
 <li>
 Lead debugging efforts, production issue resolution, and post-mortem analysis
 </li>
 <li>
 Participate in code reviews and continuously drive technical improvements
 </li>
 </ul>
 
 
 
 
 Skills &amp; Qualifications:
 
 <ul>
 <li>
 3&ndash;5 years of backend development experience in Python or Golang
 </li>
 <li>
 Strong understanding of data structures, algorithms, OOP, operating systems, and networking
 </li>
 <li>
 Proven experience developing and managing REST APIs
 </li>
 <li>
 Proficient in working with databases like PostgreSQL, MongoDB, and Redis
 </li>
 <li>
 Hands-on experience with cloud platforms &ndash; AWS, GCP, or Azure
 </li>
 <li>
 Experience with Docker or other container technologies
 </li>
 <li>
 Familiarity with CI/CD pipelines and deployment best practices
 </li>
 <li>
 Experience with monitoring and logging tools like Prometheus, Grafana, or ELK stack
 </li>
 <li>
 Experience with Git and version control workflows
 </li>
 <li>
 Contributions to open-source projects or a strong personal backend project portfolio
 </li>
 <li>
 Bachelor&rsquo;s degree in Computer Science, Engineering, or a related technical discipline
 </li>
 <li>
 Strong communication skills and a proactive, ownership-driven mindset
 </li>
 </ul>
 
 
 
 
 You can get more updates, insights and everything behind the scenes at Pocket FM here -
 
 Pocket FM
 
 
</div>

Senior Software Engineer - Backend

izmo ltd

<div>
 
 Job Title
 
 
 
 Software Engineer - Mid-Level Backend (.NET)
 
 
 
 Minimum Experience
 
 
 
 3-5 years
 
 
 
 Employment Type
 
 
 
 Full-time (Office Based &ndash; UK Hours)
 
 
 
 About The Role
 
 
 
 We are seeking a skilled
 
 Mid-Level Backend Software Engineer
 
 with
 
 strong expertise in .NET (C#)
 
 to join our high-performing engineering team. Your primary focus will be on building and maintaining robust, scalable backend systems and web applications using the latest .NET technologies.
 
 
 While backend development will be your main responsibility, experience with React is highly desirable, as you may occasionally work across our full technology stack to deliver end-to-end solutions.
 
 
 This is an exciting opportunity to contribute to products used globally in
 
 20+ countries
 
 
 and 15+ languages
 
 . We are looking for someone who can work autonomously, take ownership of their deliverables, and thrive in a collaborative environment.
 
 
 
 Key Responsibilities
 
 
 
 <ul>
 <li>
 Design, develop, and maintain backend services, web applications and APIs using .NET (C#).
 </li>
 <li>
 Collaborate with Product Owners, Designers, Engineers, and QA teams to deliver high-quality solutions.
 </li>
 <li>
 Integrate with third-party APIs and services, including handling large-scale data imports.
 </li>
 <li>
 Write clean, efficient, and maintainable code, following best practices and coding standards.
 </li>
 <li>
 Contribute to a collaborative, supportive, and high-performing engineering culture.
 </li>
 <li>
 Work independently on tasks while proactively communicating progress and challenges.
 </li>
 <li>
 Ensure a strong focus on detail and maintain a proactive approach to work and problem-solving.
 </li>
 <li>
 Continuously learn and adapt to new tools, frameworks, and engineering best practices.
 
 
 </li>
 </ul>
 
 Required Skills &amp; Experience
 
 
 
 <ul>
 <li>
 3+ years of commercial software development experience, with a strong focus on backend engineering.
 </li>
 <li>
 High proficiency in .NET (C#), including .NET Framework and .NET 8/9/10.
 </li>
 <li>
 Experience building RESTful APIs.
 </li>
 <li>
 Strong knowledge of ASP.NET and ASP.NET Core knowledge, along with core HTML &amp; CSS understanding.
 </li>
 <li>
 Experience working with MSSQL Server and Entity Framework Core.
 </li>
 <li>
 Solid understanding of software engineering principles (SOLID, design patterns)
 </li>
 <li>
 Familiarity with CI/CD pipelines and Git-based workflows.
 </li>
 <li>
 Strong problem-solving skills, attention to detail, and a commitment to taking responsibility for delivering clean, maintainable, and well-tested code.
 </li>
 <li>
 Ability work both independently and as a part of a high-performing, collaborative team.
 </li>
 <li>
 Strong communication skills.
 </li>
 <li>
 Experience working in an Agile environment and using tools / services such as Azure DevOps.
 
 
 </li>
 </ul>
 
 Nice-to-Have Skills
 
 
 
 <ul>
 <li>
 React (React Query, hooks, component-driven development)
 </li>
 <li>
 Experience with containerisation (Docker &amp; Kubernetes).
 </li>
 <li>
 Knowledge of CI/CD tools such as Azure DevOps, GitHub Actions, and Octopus Deploy.
 </li>
 <li>
 Familiarity of Microsoft Azure for cloud hosting.
 </li>
 <li>
 Unit &amp; integration testing frameworks &ndash; xUnit or similar
 </li>
 </ul>
</div>

Software Engineer .Net

Explore 57+ Principal data engineer in Bangalore this November. Find roles across top companies and apply now to boost your career.