big data engineer

Firminiq Systems

<p><b>Role Overview</b></p><p>As a Big Data Engineer, you will be responsible for architecting, developing, and maintaining scalable data platforms. You will work on complex data migration projects, particularly transitioning from <b>MySQL to NoSQL</b>, while leveraging AWS cloud-based big data tools.</p><p><b>Key Responsibilities</b></p><ul><li>Design, develop, and maintain large-scale data processing pipelines using <b>AWS Glue, Spark, and Scala</b>.</li><li>Implement scalable and efficient workflows with <b>AWS Step Functions and EMR</b>.</li><li>Lead and execute data migration projects from <b>MySQL to NoSQL databases</b>.</li><li>Collaborate with data scientists, analysts, and business teams to deliver clean and reliable data solutions.</li><li>Optimize big data processes for performance, scalability, and cost-efficiency.</li><li>Ensure data security, compliance, and governance best practices are followed.</li></ul><p><b>Required Skills &amp; Qualifications</b></p><ul><li><b>5+ years of proven experience</b> as a Big Data Engineer.</li><li>Strong expertise in <b>AWS Glue, Spark, Scala, AWS Step Functions, and EMR</b>.</li><li>Solid hands-on experience with <b>data migration from MySQL to NoSQL</b> (e.g., DynamoDB, MongoDB, Cassandra).</li><li>Proficiency in designing and maintaining ETL pipelines and data integration solutions.</li><li>Strong problem-solving and debugging skills in large-scale data environments.</li><li>Familiarity with modern data architectures and best practices.</li></ul><p><b>Good to Have (Optional)</b></p><ul><li>Experience with Python or Java for big data applications.</li><li>Knowledge of CI/CD practices for data workflows.</li><li>Exposure to data warehousing and analytics tools.</li></ul><p><b>Why Join Us?</b></p><ul><li>Opportunity to work on <b>cutting-edge big data projects</b>.</li><li>Collaborative and innovative work culture.</li><li>Career growth and continuous learning environment.</li><li>Competitive salary and benefits package</li></ul><p>Job Types: Full-time, Permanent</p><p>Pay: Up to &#8377;2,000,000.00 per year</p><p>Benefits:</p><ul><li>Health insurance</li><li>Provident Fund</li></ul><p>Work Location: In person</p>

Senior Big Data Engineer

Apple

<div>Apple is where individual imaginations gather together, committing to the values that lead to great work. Every new product we build, service we create, or Apple Store experience we deliver is the result of us making each other&rsquo;s ideas stronger. That happens because every one of us shares a belief that we can make something wonderful and share it with the world, changing lives for the better. It&rsquo;s the diversity of our people and their thinking that inspires the innovation that runs through everything we do. When we bring everybody in, we can do the best work of our lives. Here, you&rsquo;ll do more than join something - you&rsquo;ll add something. Apple Pay brought mobile payment to millions of customers, and it&rsquo;s just the beginning. We are looking for engineers who enjoy both hands-on technical work and designing thoughtful, scalable services for Wallet and Apple Pay. Our team&rsquo;s vision is to be the engine of intelligent transformation, leveraging a unified, reliable data platform to build and deploy innovative and solutions that drive significant business impact and enable data-driven decision-making throughout the organization. We are seeking pragmatic AI Big Data Engineers to join our dynamic team to build and optimize data and analytics solutions as well as perform ML enablement and participate in generative AI initiatives to craft the future of Wallet and Apple Pay. You will collaborate with cross-functional teams across different time zones and deliver impactful and scalable data architectures.
<br>
<br>
<b>Description</b><br>
<br>
- Instrument APIs, user journey and interaction flows to systematically collect behavioral, transactional, and operational data, enabling robust analytics and insightful reporting - Build robust data architectures for Wallet, Payments &amp; Commerce products. - Build robust data architectures supporting large-scale Wallet Payments &amp; Commerce (WPC) applications. - Optimize ETL workflows to enhance data processing efficiency and reliability. - Develop tools and frameworks to optimize data processing performance. - Ensure data quality and integrity across all data systems and platforms. - Collaborate closely with diverse set of partners to gather requirements, prioritize use cases, and ensure high-quality data products delivery. - Integrate data pipelines into the broader ML Operations (MLOps) process. This includes automating the data flow for feature engineering, model retraining, performance monitoring models in production, drift detection and ensuring scalability. - Construct and maintain data pipelines for Gen AI/RAG solutions, including processes for data extraction, chunking, embedding, and grounding to prepare data for models and perform continuous quality and performance measurement.
<br>
<br>
<b>Minimum Qualifications</b><br>
<ul><li>
Bachelor&rsquo;s or Master&rsquo;s degree in Computer Science or a related technical field or equivalent experience
</li><li>4+ years of experience in designing, developing and deploying data engineering for analytics or ML &amp; AI pipelines.
</li><li>Strong proficiency in SQL, Scala, Python, or Java, with hands-on experience in data pipeline tools (e.g., Apache Spark, Kafka, Airflow), CI/CD practices, and version control.
</li><li>Familiarity with cloud platforms (AWS, Azure, GCP) and data management and analytics tools like Snowflake, Databricks and Tableau.
</li><li>Strong understanding of data warehousing, data modeling (dimensional/star schemas), and metric standardization.
</li><li>Strong problem-solving skills and the ability to work in an agile environment.
</li></ul><br>
<b>Preferred Qualifications</b><br>
<ul><li>
Ability to create technical Specs, instrumentation specs and posses the ability to understand APIs, MSDs etc..
</li><li>Expertise in building and refining large-scale data pipelines, as well as developing tools and frameworks for data platforms.
</li><li>Hands-on experience with big data technologies such as distributed querying(Trino), real-time analytics(OLAP), near-real-time data processing (NRT), and decentralized data architecture (Apache Mesh).
</li><li>Familiarity with data governance, security protocols, and compliance in financial data systems.
</li><li>Experience enabling ML pipelines including automating the data flow for feature engineering, model retraining, performance monitoring models in production, drift detection and ensuring scalability.
</li><li>Familiarity with GenAI concepts like Retrieval-Augmented Generation (RAG), Large Language Models (LLMs), prompt engineering, vector embeddings, and LLM fine-tuning
</li><li>Works independently with minimal oversight, actively builds relationships, and contributes to a positive team environment.
</li><li>Demonstrates sound judgment, applies technical principles to complex projects, evaluates solutions, and proposes new ideas and process improvements.
</li><li>Seeks new opportunities for growth, demonstrates a thorough understanding of technical concepts, exercises independence in problem-solving, and delivers impactful results at the team level.
</li>Familiarity with Fintech, Wallet domain, digital commerce etc..<br>
<li>
</li></ul><br>
<br>
Submit CV</div>

AI Big Data Engineer

Citi

<div><p>Responsible for designing, developing, and optimizing data processing solutions using a combination of Big Data technologies. Focus on building scalable and efficient data pipelines for handling large datasets and enabling batch &amp; real-time data streaming and processing.<br>
<br>
Responsibilities:<br>
&gt; Develop Spark applications using Scala or Python (Pyspark) for data transformation, aggregation, and analysis.<br>
&gt; Develop and maintain Kafka-based data pipelines: This includes designing Kafka Streams, setting up Kafka Clusters, and ensuring efficient data flow.<br>
&gt; Create and optimize Spark applications using Scala and PySpark: They leverage these languages to process large datasets and implement data transformations and aggregations.<br>
&gt; Integrate Kafka with Spark for real-time processing: They build systems that ingest real-time data from Kafka and process it using Spark Streaming or Structured Streaming.<br>
&gt; Collaborate with data teams: This includes data engineers, data scientists, and DevOps, to design and implement data solutions.<br>
&gt; Tune and optimize Spark and Kafka clusters: Ensuring high performance, scalability, and efficiency of data processing workflows.<br>
&gt; Write clean, functional, and optimized code: Adhering to coding standards and best practices.<br>
&gt; Troubleshoot and resolve issues: Identifying and addressing any problems related to Kafka and Spark applications.<br>
&gt; Maintain documentation: Creating and maintaining documentation for Kafka configurations, Spark jobs, and other processes.<br>
&gt; Stay updated on technology trends: Continuously learning and applying new advancements in functional programming, big data, and related technologies.<br>
<br>
Proficiency in:<br>
Hadoop ecosystem big data tech stack(HDFS, YARN, MapReduce, Hive, Impala).<br>
Spark (Scala, Python) for data processing and analysis.<br>
Kafka for real-time data ingestion and processing.<br>
ETL processes and data ingestion tools<br>
Deep hands-on expertise in Pyspark, Scala, Kafka<br>
<br>
Programming Languages:<br>
Scala, Python, or Java for developing Spark applications.<br>
SQL for data querying and analysis.<br>
<br>
Other Skills:<br>
Data warehousing concepts.<br>
Linux/Unix operating systems.<br>
Problem-solving and analytical skills.<br>
Version control systems</p><p>
-</p><p><b>
Job Family Group:
</b></p>Technology<p>
-</p><p><b>
Job Family:</b></p>
Applications Development<p>
-</p><p><b>
Time Type:</b></p>
Full time<p>
-</p><p><b>
Most Relevant Skills
</b></p>Please see the requirements listed above.<p>
-</p><p><b>
Other Relevant Skills
</b></p>For complementary skills, please see above and/or contact the recruiter.<p>
-</p><p><i>
Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.</i></p><p></p><p><i><br>
If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review </i><i>Accessibility at Citi</i><i>.</i><br>
<br>
<i>View Citi&rsquo;s </i><i>EEO Policy Statement</i><i> and the </i><i>Know Your Rights</i><i> poster.</i></p></div>

Big Data Engineer - Hybrid - Officer

<div></div><div><div><div><div><ul><div>5 - 7 Years</div>
<div></div><div>1 Opening</div>
 <div>Hyderabad</div></ul></div><p></p><div><br>
<div><div>
</div></div></div></div><div></div><div><h3 class="jobSectionHeader"><b>Role description</b></h3>
<div><p>We are seeking a skilled Big Data Engineer with 3+ years of experience in developing, optimizing, and maintaining large-scale data applications on AWS Cloud. The ideal candidate will have expertise in Java, Scala, Python, and Big Data technologies such as Hadoop, Spark, and Cassandra. Proficiency in AWS services like EMR, EC2, S3, Lambda, and Step Functions is essential. Experience with REST APIs, SOAP web services, and Linux scripting is required. The role involves designing and implementing scalable data pipelines, optimizing performance, conducting root cause analysis for defects, and ensuring adherence to coding standards and best practices. Strong communication skills and the ability to collaborate with cross-functional teams are crucial. A proactive approach to learning and applying new technologies is expected</p></div>
<h3 class="jobSectionHeader"><b>Skills</b></h3>
<div><p>cloud data infrastucture,Data Processing,communication</p></div></div></div></div><p></p>
<br><div><h3 class="jobSectionHeader"><b>About UST</b></h3>
<div>UST is a global digital transformation solutions provider. For more than 20 years, UST has worked side by side with the world&rsquo;s best companies to make a real impact through transformation. Powered by technology, inspired by people and led by purpose, UST partners with their clients from design to operation. With deep domain expertise and a future-proof philosophy, UST embeds innovation and agility into their clients&rsquo; organizations. With over 30,000 employees in 30 countries, UST builds for boundless impact&mdash;touching billions of lives in the process.</div></div>

Lead I - Big Data Engineer AWS

MetLife

<div><div><div><div><div><div>Country
</div><div>India
</div></div><div><div>Working Schedule
</div><div>Full-Time
</div></div><div><div>Work Arrangement
</div><div>Hybrid
</div></div><div><div>Relocation Assistance Available
</div><div>No
</div></div><div><div>Posted Date
</div><div>15-Sep-2025
</div></div><div><div>Job ID
</div><div>12094
</div></div></div></div></div><div><div><div><h3 class="jobSectionHeader"><b>Description and Requirements
</b></h3></div></div><div><div><div><div><div><div><div><div><div><b>Position Summary</b></div></div></div></div><div><div><div><div><br>
MetLife established a Global capability center (MGCC) in India to scale and mature Data &amp; Analytics, technology capabilities in a cost-effective manner and make MetLife future ready. The center is integral to Global Technology and Operations with a with a focus to protect &amp; build MetLife IP, promote reusability and drive experimentation and innovation. The Data &amp; Analytics team in India mirrors the Global D&amp;A team with an objective to drive business value through trusted data, scaled capabilities, and actionable insights</div></div><div></div><div><div><b><br>
Role Value Proposition</b></div></div><div><div>
MetLife Global Capability Center (MGCC) is looking for a Senior Cloud data engineer who has the responsibility of building ETL/ELT, data warehousing and reusable components using Azure, Databricks and spark. He/She will collaborate with the business systems analyst, technical leads, project managers and business/operations teams in building data enablement solutions across different LOBs and use cases.</div></div></div></div><div><div><div><div><b><br>
Job Responsibilities</b></div></div></div></div><div><div><div><br>
<ul><li>
Collect, store, process and analyze large datasets to build and implement extract, transfer, load (ETL) processes</li><li>
Develop metadata and configuration based reusable frameworks to reduce the development effort</li><li>
Develop quality code with integral performance optimizations in place right at the development stage.</li><li>
Collaborate with global team in driving the delivery of projects and recommend development and performance improvements.</li><li>
Extensive experience of various databases types and knowledge to leverage the right one for the need.</li><li>
Strong understanding of data tools and ability to leverage them to understand the data and generate insights</li><li>
Hands on experience in building/designing at-scale Data Lake, Data warehouses, data stores for analytics consumption On prem and Cloud (real time as well as batch use cases)</li><li>
Ability to interact with business analysts and functional analysts in getting the requirements and implementing the ETL solutions.</li></ul></div></div></div><div></div><div><div><div><div><b><br>
Education, Technical Skills &amp; Other Critical Requirement</b></div></div></div></div><div><div><div><div><b><br>
Education</b></div></div></div><div><div><div><br>
Bachelor&rsquo;s degree in computer science, Engineering, or related discipline</div></div></div></div><div><div><div><div><b><br>
Experience</b></div></div><div><div><i>
(In Years)</i></div></div></div><div><div><div><br>
 8 to 10 years of working experience on Azure Cloud using Databricks or Synapse</div></div></div></div><div><div><div><div><b><br>
Technical Skills</b></div></div></div><div><div><ul><li><div><br>
Experience in transforming data using Python, Spark or Scala</div></li></ul></div><div><ul><li><div>
Technical depth in Cloud Architecture Framework, Lakehouse Architecture and One Lake solutions.</div></li></ul></div><div><ul><li><div>
Experience in implementing data ingestion and curation process using Azure with tools such as Azure Data Factory, Databricks Workflows, Azure Synapse, Cosmos DB, Spark (Scala/python), Data bricks.</div></li></ul></div><div><ul><li><div>
Experience in cloud optimized code on Azure using Databricks, Synapse dedicated SQL Pool and serverless Pools, Cosmos, SQL APIs loading and consumption optimizations.</div></li></ul></div><div><ul><li><div>
Scripting experience primarily on shell/bash/PowerShell would be desirable.</div></li></ul></div><div><ul><li><div>
Experience in writing SQL and performing data analysis skills for data anomaly detection and data quality assurance.</div></li></ul></div></div></div><div><div><div><div><b><br>
Other </b><b>Preferred Skills</b></div></div></div><div><div><ul><li><div><br>
Expertise in Python and experience writing Azure functions using Python/Node.js</div></li></ul></div><div><ul><li><div>
Experience using Event Hub for data integrations.</div></li></ul></div><div><ul><li><div>
Required working knowledge of Azure DevOps pipelines</div></li></ul></div><div><ul><li><div>
Self-starter and ability to adapt with changing business needs</div></li></ul></div></div></div></div><br>
<div></div>
</div></div></div></div></div><div><div><div><div><div><b>About MetLife</b><div></div><div><br>
Recognized on<i> Fortune magazine's list of the 2025 &quot;World's Most Admired Companies&quot; and Fortune World&rsquo;s 25 Best Workplaces&trade; for 2024, </i>MetLife<i>,</i> through its subsidiaries and affiliates, is one of the world&rsquo;s leading financial services companies; providing insurance, annuities, employee benefits and asset management to individual and institutional customers. With operations in more than 40 markets, we hold leading positions in the United States, Latin America, Asia, Europe, and the Middle East.</div><div><br>
Our purpose is simple - to help our colleagues, customers, communities, and the world at large create a more confident future. United by purpose and guided by empathy, we&rsquo;re inspired to transform the next century in financial services. At MetLife, it&rsquo;s #AllTogetherPossible. Join us!</div></div></div></div></div></div></div>

Senior Big Data Engineer - I

Accenture

<p><b>Project Role : </b>Data Engineer<br>
<b>Project Role Description : </b>Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems.
<br>
<b>Must have skills : </b>Microsoft Azure Databricks<br>
<b>Good to have skills : </b>NA<br>
Minimum <b>5</b> year(s) of experience is required<br>
<b>Educational Qualification : </b>15 years full time education<br>
<br>
Summary: As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your typical day will involve creating data pipelines, ensuring data quality, and implementing ETL processes to effectively migrate and deploy data across various systems. You will collaborate with cross-functional teams to understand their data needs and provide innovative solutions that enhance data accessibility and usability. Key Responsibilities: 1. Function as the Lead Data Architect for a small, simple project/proposal or as a team lead for medium/large sized project or proposal 2. Discuss specific Big data architecture and related issues with client architect/team (in area of expertise) 3. Analyze and assess the impact of the requirements on the data and its lifecycle 4. Lead Big data architecture and design medium-big Cloud based, Big Data and Analytical Solutions using Lambda architecture. 5. Breadth of experience in various client scenarios and situations 6. Experienced in Big Data Architecture-based sales and delivery 7. Thought leadership and innovation 8. Lead creation of new data assets &amp; offerings 9. Experience in handling OLTP and OLAP data workloads Technical Experience: 1. Experience working in Medallion architecture involving Delta lake house principles 2. Expert level in Designing and Architect solutions in Azure Data factory, Azure Databricks, Azure Datalake, Delta Lake implementation. 3. Expert level experience in Azure cloud technologies like PySpark, ADF, Databricks, Python, Scala and SQL. 4. Exp in one or more Real-time/batch ingestion including: Azure Delta live tables , Autoloader 5. Exp in handling medium to large Big Data implementations 6. Strong understanding of data strategy. Data Quality and Delta lake components 7. Candidate must have 7-10 years of IT experience and around 5 years of Big data experience (design + build) in Databricks 8. Architect for a medium sized client delivery project Professional Experience: 1. Should be able to drive the technology design meetings, propose technology design and architecture 2. Should have excellent client communication skills 3. Should have good analytical and problem-solving skills Educational Qualification: 1. Must have: BE/BTech/MCA 2. Good to have: ME/MTech</p>
<br>
15 years full time education

Data Engineer

BPK Tech

<div><div><h3 class="jobSectionHeader"><b>Job Information</b></h3>
<ul><div><p>Industry</p>
<div><b>IT Services</b><b>
</b></div></div><div><p>Date Opened</p>
<div><b>09/10/2025</b><b>
</b></div></div><div><p>Salary</p>
<div><b>Confidential</b><b>
</b></div></div><div><p>Job Type</p>
<div><b>Full time</b><b>
</b></div></div><div><p>Work Experience</p>
<div><b>8 + years</b><b>
</b></div></div><div><p>City</p>
<div><b>Hyd/ Pune/Open to Remote</b><b>
</b></div></div></ul></div><div><h3 class="jobSectionHeader"><b>Job Description</b></h3><div><div><br>
<br>
</div><div>Veltris is a Digital Product Engineering Services partner committed to driving technology-enabled transformation across enterprises, businesses, and industries. We specialize in delivering next-generation solutions for sectors including healthcare, technology, communications, manufacturing, and finance.</div><div></div><div><br>
With a focus on innovation and acceleration, Veltris empowers clients to build, modernize, and scale intelligent products that deliver connected, AI-powered experiences. Our experience-centric approach, agile methodologies, and exceptional talent enable us to streamline product development, maximize platform ROI, and drive meaningful business outcomes across both digital and physical ecosystems.</div><div></div><div><br>
In a strategic move to strengthen our healthcare offerings and expand industry capabilities, Veltris has acquired BPK Technologies. This acquisition enhances our domain expertise, broadens our go-to-market strategy, and positions us to deliver even greater value to enterprise and mid-market clients in healthcare and beyond.</div><div></div><div><b><br>
Position-Senior Big Data Engineer</b></div><div></div><div><br>
Must have Big Data analytics platform experience.<br>
</div><ul><li>Key stacks: Spark, Druid, Drill, ClickHouse.</li><li>8+ years experience in Python/Java, CI/CD, infrastructure &amp; cloud, Terraform, plus depth in:</li></ul><div>
 o Big Data pipelines: Spark, Kafka, Glue, EMR, Hudi, Schema Registry, Data Lineage.<br>
</div><div>
 o Graph DBs: Neo4j, Neptune, JanusGraph, Dgraph.</div><div></div><div><br>
Preferred Qualifications:<br>
</div><ul><li>Master&rsquo;s degree (M.Tech/MS) or Ph.D. in Computer Science, Information Technology, Data Science, Artificial Intelligence, Machine Learning, Software Engineering, or a related technical field.</li><li>Candidates with an equivalent combination of education and relevant industry experience will also be considered.</li></ul><div></div><div><b><br>
Disclaimer</b>:<br>
</div><div>
The information provided herein is for general informational purposes only and reflects the current strategic direction and service offerings of Veltris. While we strive for accuracy, Veltris makes no representations or warranties regarding the completeness, reliability, or suitability of the information for any specific purpose. Any statements related to business growth, acquisitions, or future plans, including the acquisition of BPK Technologies, are subject to change without notice and do not constitute a binding commitment. Veltris reserves the right to modify its strategies, services, or business relationships at its sole discretion. For the most up-to-date and detailed information, please contact Veltris directly</div></div></div></div><p></p>

Senior Software Engineer (Big Data)

Search 10000+ Big data engineer jobs near you this September. New roles added regularly across leading industries. Apply today.