hadoop engineer

Legacy Smart Employment Services

- Architect and manage robust big data pipelines using the Hadoop ecosystem (HDFS, Hive, Pig, HBase, Sqoop, Flume, etc.) - Develop and optimize solutions using Spark, MapReduce, and NoSQL databases like Cassandra and HBase. - Write clean, scalable code in Java, Python, Scala, or R — bonus points if you&#x27;re comfortable in Linux environments. - Own end-to-end data engineering deliverables and lead the charge on performance, scalability, and reliability. - Collaborate with cross-functional teams to solve complex data and system integration challenges.

Senior Data Engineer

Datamatics Global Services Ltd

<div><h2 class="jobSectionHeader">Cloudera Data Engineer</h2>
Location: Remote (UAE Time Zone) 
Experience: 5+ Years 
Availability: Immediate joiners or candidates who can start within 30 days preferred<h3 class="jobSectionHeader">
Role Overview</h3>
We are seeking an experienced Cloudera Data Engineer to design, build, and maintain data pipelines and processing frameworks across Cloudera&rsquo;s ecosystem. The ideal candidate will have strong expertise in Spark, Hive, Impala, NiFi, and Oozie, and proven experience optimizing large-scale data pipelines for performance and reliability.<h3 class="jobSectionHeader">
Key Responsibilities</h3><ul><li>
Design, build, and maintain ETL/ELT pipelines using Spark, Hive, Impala, NiFi, and Oozie.</li><li>
Optimize data pipelines for performance, reliability, and cost efficiency across batch and streaming workloads.</li><li>
Develop automation scripts and reusable frameworks for data ingestion, transformation, and cleansing.</li><li>
Collaborate with Cloudera Administrators to tune performance and manage cluster resources efficiently.</li><li>
Troubleshoot and resolve data pipeline failures, performance bottlenecks, and data quality issues.</li><li>
Monitor cluster utilization, job performance, and data throughput using Cloudera Manager.</li><li>
Work with DevOps/Infrastructure teams to define and maintain CI/CD pipelines, automation for deployments, and monitoring of data jobs.</li><li>
Participate in incident management, root cause analysis, and preventive maintenance.</li><li>
Ensure pipelines meet SLAs, data quality, and latency requirements for analytics workloads.</li></ul><h3 class="jobSectionHeader">
Requirements</h3><ul><li>
5+ years of experience as a Data Engineer in a Cloudera-based environment.</li><li>
Strong hands-on experience with Spark, Hive, Impala, NiFi, and Oozie.</li><li>
Solid understanding of ETL design, performance optimization, and data governance principles.</li><li>
Experience with Linux/Unix scripting and CI/CD integration.</li><li>
Excellent troubleshooting and communication skills.</li></ul>
IppsLo883Y</div>

Cloudera Data Engineer

Prodapt Solutions

<div>Overview:
<h3 class="jobSectionHeader">Overview:</h3>
We are seeking a talented and motivated GCP Data Engineer with experience in real-time streaming technologies, specifically Apache Kafka, and proficiency in Java. The ideal candidate will be responsible for designing, developing, and maintaining data pipelines and streaming applications in a cloud environment. 

Responsibilities:
<h3 class="jobSectionHeader">Key Responsibilities:</h3><ul><li>
Data Pipeline Development: Design and implement robust data pipelines using Google Cloud services (BigQuery, Dataflow, Pub/Sub) integrated with Kafka for real-time data processing.</li><li>
Streaming Applications: Develop and maintain real-time streaming applications using Kafka and Java, ensuring high availability and performance.</li><li>
Cloud Infrastructure: Utilize GCP technologies to build scalable and efficient data architectures, leveraging tools such as Cloud Storage, Cloud Functions, and Kubernetes.</li><li>
Monitoring and Optimization: Monitor data flow, performance metrics, and troubleshoot issues in production environments. Optimize data pipeline performance and reliability.</li><li>
Collaboration: Work closely with data scientists, analysts, and other engineers to understand data requirements and translate them into technical specifications.</li><li>
Documentation: Maintain comprehensive documentation of data pipelines, architecture, and processes to ensure knowledge sharing and compliance.</li></ul> 

Requirements:
<h3 class="jobSectionHeader">Required Skills:</h3><ul><li>
Cloud Skills: Proficiency in Google Cloud Platform (GCP), including services like BigQuery, Dataflow, and Pub/Sub.</li><li>
Streaming Technologies: Strong experience with Kafka for real-time data streaming and messaging.</li><li>
Programming Language: Proficient in Java, with a solid understanding of object-oriented programming principles and best practices.</li><li>
Data Management: Familiarity with data modeling, ETL processes, and database management.</li><li>
Development Tools: Experience with version control systems (e.g., Git), CI/CD tools, and Agile methodologies.</li><li>
Problem-Solving: Strong analytical and troubleshooting skills, with the ability to work independently and as part of a team.</li></ul><h3 class="jobSectionHeader"> 
Preferred Qualifications:</h3><ul><li>
Bachelor's degree in Computer Science, Information Technology, or a related field.</li><li>
Experience with other programming languages (Python, Scala) is a plus.</li><li>
Knowledge of additional GCP services or data visualization tools (e.g., Looker, Tableau) is advantageous.</li></ul></div>

GCP Data Engineer (Kafka + Java)

Cognizant Technology Solutions

<div>Skills- AWS +Pyspark
Experience: 4 to 9 years
Location: AIA-Pune
As an AWS Data Engineer, you will design, build, and manage robust data pipelines on the AWS cloud platform. You will leverage your expertise in AWS data services and programming languages to ingest, process, transform, and store data efficiently, enabling data-driven insights and decision-making.
Key Responsibilities :
<ul><li>Data Pipeline Development: Design, implement, and maintain scalable data pipelines using AWS services such as S3, Glue, Lambda, EMR, and Kinesis.</li><li>Data Processing and Transformation: Utilize PySpark, Spark, and SQL to perform complex data transformations and aggregations on large datasets within the AWS ecosystem.</li><li>Data Storage and Management: Design and implement data storage solutions using Amazon S3, RDS, DynamoDB, and Redshift, ensuring data quality, integrity, and accessibility.</li><li>Data Modeling and Warehousing: Develop and maintain data models to support analytics and reporting, leveraging Redshift as the data warehousing solution.</li><li>Infrastructure and Cloud Technologies: Provision and manage scalable data infrastructure using EC2, VPC, IAM, CloudFormation, and other relevant AWS services.</li><li>Performance Optimization and Monitoring: Continuously monitor and optimize data pipelines and systems using CloudWatch, identifying and resolving performance bottlenecks.</li><li>Collaboration and Knowledge Sharing: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and provide technical guidance.</li></ul>
Qualifications :
<ul><li>Proficiency in Python, SQL, PySpark, and Spark.</li><li>Strong expertise in AWS data services (S3, Glue, Lambda, EMR, Redshift, DynamoDB, RDS).</li><li>Experience with data warehousing, ETL processes, and data modeling.</li><li>Excellent problem-solving, analytical, and communication skills.</li></ul></div>

AWS + Pyspark

Nets-international Communication

Job Summary We are looking for a skilled Data Center Engineer to join NETS International&rsquo;s dynamic infrastructure team. In this role you will ensure the high availability, performance and scalability of our data-centre operations while supporting advanced analytics, cloud services and machine learning initiatives.Key Responsibilities:<ul><li>Design, implement and maintain data centre infrastructure to deliver reliable, high-performance operations.</li><li>Collaborate with cross-functional teams to develop and deploy machine learning models using TensorFlow, R and Python.</li><li>Manage ETL workflows and database architecture for efficient data storage and access.</li><li>Leverage big-data technologies such as Hadoop and Spark for large-scale data processing and analytics.</li><li>Conduct data mining and analysis using SQL, SAS and analytics tools to derive actionable insights.</li><li>Apply AI and NLP techniques to enhance operational efficiency and decision-making.</li><li>Oversee ML model training, deployment and monitoring for continuous improvement.</li><li>Work with linked-data/integration frameworks to unify disparate datasets for robust analytics.</li><li>Use programming/scripting languages (Java, C, VBA, Bash) for automation, operational support and custom solutions.</li><li>Support emerging quantum-engineering initiatives by applying advanced computational techniques where relevant.</li></ul>Required Skills &amp; Qualifications:<ul><li>Proficient in AWS cloud services and infrastructure management.</li><li>Strong understanding of machine learning frameworks (TensorFlow, Spark, etc.).</li><li>Experience with unsupervised learning and other ML techniques.</li><li>Hands-on experience in ETL processes and tools (e.g., Talend) for data integration.</li><li>Solid knowledge of database design, SQL and relational database management.</li><li>Familiarity with big-data ecosystems (Hadoop, Spark).</li><li>Proficiency in programming languages: Python, Java, C, R, Bash/Unix shell, VBA.</li><li>Experience with analytics/visualisation tools (e.g., Looker).</li><li>Excellent problem-solving skills with focus on ML model deployment and optimisation.</li><li>A drive to innovate and apply cutting-edge technologies in a fast-moving environment.</li></ul>Job Type: Full-timeAbility to commute/relocate:<ul><li>Muscat: Reliably commute or planning to relocate before starting work (Required)</li></ul>Application Question(s):<ul><li>what is your monthly current salary?</li><li>What is your monthly expected salary?</li><li>What is your notice period?</li></ul>Education:<ul><li>Bachelor's (Required)</li></ul>Experience:<ul><li>AI Data Center Engineer: 3 years (Required)</li></ul>

AI Data Center Engineer

Job Summary We are looking for a skilled Data Center Engineer to join NETS International&rsquo;s dynamic infrastructure team. In this role you will ensure the high availability, performance and scalability of our data-centre operations while supporting advanced analytics, cloud services and machine learning initiatives.Key Responsibilities:<ul><li>Design, implement and maintain data centre infrastructure to deliver reliable, high-performance operations.</li><li>Collaborate with cross-functional teams to develop and deploy machine learning models using TensorFlow, R and Python.</li><li>Manage ETL workflows and database architecture for efficient data storage and access.</li><li>Leverage big-data technologies such as Hadoop and Spark for large-scale data processing and analytics.</li><li>Conduct data mining and analysis using SQL, SAS and analytics tools to derive actionable insights.</li><li>Apply AI and NLP techniques to enhance operational efficiency and decision-making.</li><li>Oversee ML model training, deployment and monitoring for continuous improvement.</li><li>Work with linked-data/integration frameworks to unify disparate datasets for robust analytics.</li><li>Use programming/scripting languages (Java, C, VBA, Bash) for automation, operational support and custom solutions.</li><li>Support emerging quantum-engineering initiatives by applying advanced computational techniques where relevant.</li></ul>Required Skills &amp; Qualifications:<ul><li>Proficient in AWS cloud services and infrastructure management.</li><li>Strong understanding of machine learning frameworks (TensorFlow, Spark, etc.).</li><li>Experience with unsupervised learning and other ML techniques.</li><li>Hands-on experience in ETL processes and tools (e.g., Talend) for data integration.</li><li>Solid knowledge of database design, SQL and relational database management.</li><li>Familiarity with big-data ecosystems (Hadoop, Spark).</li><li>Proficiency in programming languages: Python, Java, C, R, Bash/Unix shell, VBA.</li><li>Experience with analytics/visualisation tools (e.g., Looker).</li><li>Excellent problem-solving skills with focus on ML model deployment and optimisation.</li><li>A drive to innovate and apply cutting-edge technologies in a fast-moving environment.</li></ul>Job Type: Full-timeAbility to commute/relocate:<ul><li>&#305;stanbul: Reliably commute or planning to relocate before starting work (Required)</li></ul>Application Question(s):<ul><li>What is your monthly current salary?</li><li>What is your monthly expected salary?</li><li>What is your notice period?</li></ul>Education:<ul><li>Bachelor's (Preferred)</li></ul>Experience:<ul><li>AI Data Centre Engineer: 3 years (Required)</li></ul>

Stripe

<div>
 Who we are
 
 
 
 About Stripe
 
 
 
 Stripe is a financial infrastructure platform for businesses. Millions of companies&mdash;from the world&rsquo;s largest enterprises to the most ambitious startups&mdash;use Stripe to accept payments, grow their revenue, and accelerate new business opportunities. Our mission is to increase the GDP of the internet, and we have a staggering amount of work ahead. That means you have an unprecedented opportunity to put the global economy within everyone&rsquo;s reach while doing the most important work of your career.
 
 
 
 About The Team
 
 
 
 The Batch Compute team at Stripe manages the infrastructure, tooling and systems behind running batch processing systems at Stripe, which are currently powered by Hadoop and Spark. Batch processing systems power several core asynchronous workflows at Stripe and operate at significant scale.
 
 
 What you&rsquo;ll do
 
 
 We're looking for a Software Engineer with experience designing, building and maintaining high-scale, distributed systems. You will work with a team that is in charge of the core infrastructure used by the product teams to build and operate batch processing jobs. You will have an opportunity to play a hands-on role in significantly rearchitecting our current infrastructure to be much more efficient and resilient. This re-architecture will introduce disaggregation of Hadoop storage and compute with open source solutions.
 
 
 
 Responsibilities
 
 
 
 <ul>
 <li>
 Scope and lead technical projects within the Batch Compute domain
 </li>
 <li>
 Build and maintain the infrastructure which powers the core of Stripe.
 </li>
 <li>
 Directly contribute to core systems and write code.
 </li>
 <li>
 Work closely with the open source community to identify opportunities for adopting new open source features as well contribute back to the OSS.
 </li>
 <li>
 Ensure operational excellence and enable a highly available, reliable and secure Batch Compute platform
 
 
 
 </li>
 </ul>
 Who you are
 
 
 We&rsquo;re looking for someone who meets the minimum requirements to be considered for the role. If you meet these requirements, you are encouraged to apply. The preferred qualifications are a bonus, not a requirement.
 
 
 
 Minimum Requirements
 
 
 
 <ul>
 <li>
 8+ years of professional experience writing high quality production level code or software programs.
 </li>
 <li>
 Have experience with distributed data systems such as Spark, Hadoop, Flink, Trino, Kafka ,etc
 </li>
 <li>
 Experience developing, maintaining and debugging distributed systems built with open source tools.
 </li>
 <li>
 Experience building infrastructure as a product centered around user needs.
 </li>
 <li>
 Experience optimizing the end to end performance of distributed systems.
 </li>
 <li>
 Experience with scaling distributed systems in a rapidly moving environment.
 
 
 
 </li>
 </ul>
 
 Preferred Qualifications
 
 
 
 <ul>
 <li>
 Experience as a user of batch processing systems (Hadoop, Spark)
 </li>
 <li>
 Track record of open source contributions to data processing or big data systems (Hadoop, Spark, Celeborn, Flink, etc)
 
 
 
 </li>
 </ul>
 In-office expectations
 
 
 Office-assigned Stripes in most of our locations are currently expected to spend at least 50% of the time in a given month in their local office or with users. This expectation may vary depending on role, team and location. For example, Stripes in our Bucharest, Romania site have an 80% in-office expectation, and those in Stripe Delivery Center roles in Mexico City, Mexico and Bengaluru, India work 100% from the office. Also, some teams have greater in-office attendance requirements, to appropriately support our users and workflows, which the hiring manager will discuss. This approach helps strike a balance between bringing people together for in-person collaboration and learning from each other, while supporting flexibility when possible.
 
 
 Pay and benefits
 
 
 Stripe does not yet include pay ranges in job postings in every country. Stripe strongly values pay transparency and is working toward pay transparency globally.
</div>

Software Engineer, Batch Compute

Search 8519+ Hadoop engineer jobs near you this November. New roles added regularly across leading industries. Apply today.

8519 hadoop engineer jobs NEAR_ME

Senior Data Engineer

Cloudera Data Engineer

GCP Data Engineer (Kafka + Java)

AWS + Pyspark

AI Data Center Engineer

AI Data Center Engineer

Software Engineer, Batch Compute