big data developer

CRUTZ LEELA ENTERPRISES

<p>Sr. Spark Big Data Developer</p><p>PFB the detailed JD:</p><p>&#65533;&#65533; Experience: 5 to 7+ years (Hands-on)</p><p>&#65533;&#65533; Notice Period: 0-30 Days</p><p>Must Have:</p><p>&#61623; Proficiency in at least one of the following programming languages:</p><p>Java/Scala/Python</p><p>&#61623; Good understanding of SQL</p><p>&#61623; Experience of development and deployment of at least one end-to-end data</p><p>storage/processing pipeline</p><p>&#61623; Strong Experience in Spark development with batch and streaming</p><p>&#61623; Intermediate level expertise in HDFS and Hive</p><p>&#61623; Experience with Pyspark and Data Engineering</p><p>&#61623; ETL implementation and migration to spark</p><p>&#61623; Experience of working with Hadoop cluster</p><p>&#61623; Python, PySpark, Data Bricks developer with knowledge of cloud</p><p>&#61623; Experience with Kafka and Spark streaming (Dstream and Structured Streaming)</p><p>&#61623; Experience with using Jupyter notebooks or any other developer tool</p><p>&#61623; Experience with Airflow or other workflow engines</p><p>&#61623; Good communication skills and logical skills</p><p>Good to Have Skills:</p><p>&#61623; Prior experience of writing Spark jobs using Java is highly appreciated</p><p>&#61623; Prior experience of working with Cloudera Data Platform (CDP)</p><p>&#61623; Hands-on experience with NoSQL databases like HBase, Cassandra,</p><p>Elasticsearch, etc.</p><p>&#61623; Experience of using maven and git</p><p>&#61623; Agile scrum methodologies</p><p>&#61623; Flink and Kudu streaming</p><p>&#61623; Automation of workflows CI/CD</p><p>&#61623; Nifi streaming and transformation</p><p>Job Types: Full-time, Permanent</p><p>Pay: &#8377;500,000.00 - &#8377;1,500,000.00 per year</p><p>Benefits:</p><ul><li>Provident Fund</li></ul><p>Work Location: In person</p>

Sr. Spark Big Data Developer

Recruitment Hub 365

<p><b>Job Title: Big Data Hadoop Developer</b><br/><b>Location: Hyderabad</b><br/><b>Work mode: 5 Days - (WFO)</b><br/><b>Experience Required: 4+ years</b><br/><b>Key Skills: Big Data, Hadoop, Spark, Hive</b></p><p><b>Job Overview</b><br/>We are looking for an experienced Big Data Hadoop Developer to design, build, and optimize large-scale data processing systems. The ideal candidate should have strong expertise in the Hadoop ecosystem, hands-on experience with Spark and Hive, and the ability to develop efficient ETL pipelines.</p><p><b>Key Responsibilities</b></p><ul><li>Design, develop, and maintain scalable big data systems and data pipelines.</li><li>Implement data processing frameworks using <b>Hadoop, Spark, and Hive</b>.</li><li>Develop and manage ETL workflows to ensure data accuracy, consistency, and availability.</li><li>Collaborate with data architects, analysts, and business teams to translate requirements into robust big data solutions.</li><li>Perform data validation, quality checks, and issue resolution for data pipelines.</li><li>Optimize data storage, query performance, and cluster utilization in the Hadoop ecosystem.</li><li>Ensure compliance with security, governance, and data management standards.</li></ul><p><b>Required Skills &amp; Qualifications</b></p><ul><li>4+ years of hands-on experience in <b>Big Data technologies</b>.</li><li>Strong understanding of the <b>Hadoop ecosystem</b> (HDFS, MapReduce, YARN).</li><li>Proficiency in <b>Apache Spark</b> (batch and streaming) and <b>Hive</b>.</li><li>Experience in building and maintaining data pipelines and ETL processes.</li><li>Strong knowledge of data optimization, partitioning, and performance tuning.</li><li>Familiarity with NoSQL databases such as <b>HBase, Cassandra, or MongoDB</b> is an advantage.</li><li>Experience with programming/scripting languages: <b>Java, Scala, Python, or Shell</b>.</li><li>Strong analytical and problem-solving skills.</li></ul><p>Job Type: Full-time</p><p>Pay: &#8377;1,500,000.00 - &#8377;2,400,000.00 per year</p><p>Application Question(s):</p><ul><li>How many years of experience you have with Big Data, Hadoop, Spark, Hive?</li><li>Mention your Notice Period, Current CTC and ECTC.</li><li>How many years of experience you have with Big Data / Hadoop Developer?</li></ul><p>Work Location: In person</p>

Big Data/Hadoop Developer

KMCCorp India

<div><div><h3 class="jobSectionHeader"><b>Job Information</b></h3>
<ul><div><p>Date Opened</p>
<div><b>11/19/2025</b><b>
</b></div></div><div><p>Industry</p>
<div><b>Information Technology</b><b>
</b></div></div><div><p>Job Type</p>
<div><b>Full time</b><b>
</b></div></div><div><p>Work Experience</p>
<div><b>5+ years</b><b>
</b></div></div><div><p>City</p>
<div><b>Hyderabad</b><b>
</b></div></div><div><p>State/Province</p>
<div><b>Telangana</b><b>
</b></div></div><div><p>Country</p>
<div><b>India</b><b>
</b></div></div><div><p>Zip/Postal Code</p>
<div><b>500003</b><b>
</b></div></div></ul></div><div><h3 class="jobSectionHeader"><b>Job Description</b></h3>
<div><div>Job Description<br>
<br>
The successful candidate will work on batch and real-time data pipelines, leveraging technologies such as Scala, Java, and Apache Spark. They will manage distributed systems and cloud resources, ensure security compliance using HashiCorp Vault, orchestrate workflows with Apache Airflow, and investigate production issues by analyzing logs and monitoring metrics.<br>
<br>
Responsibilities<br>
<br>
<ul><li>Develop and maintain batch and streaming data pipelines using Scala and Java.</li><li>Write and execute shell scripts and Yarn commands for Spark job management.</li><li>Manage and optimize big data environments using Apache Spark, EMR, Hadoop, and YARN.</li><li>Utilize cloud storage and container platforms such as Amazon S3 and EKS.</li><li>Implement and manage security features, including HashiCorp Vault and tokenization/encryption protocols.</li><li>Schedule and orchestrate batch workflows using Apache Airflow.</li><li>Conduct root cause analysis and handle production incidents.</li><li>Analyze application, Spark executor, and Dynatrace logs.</li><li>Monitor jobs and EMR clusters using Dynatrace metrics.</li><li>Validate data and set up production alerts.</li></ul></div><h3 class="jobSectionHeader"><b><br>
Requirements</b></h3><div>
Experience with batch and streaming processing.<br>
<ul><li>Proficiency in distributed systems management.</li><li>Knowledge of cloud platforms (AWS S3, EKS).</li><li>Security expertise (Vault, tokenization, encryption).</li><li>Workflow orchestration (Airflow).</li><li>Strong analytical and troubleshooting skills for production investigation</li></ul></div></div></div></div><p></p>

Big Data/ Scala Developer

HCLTech

<div>
 <p>
  <strong>
   <u>
    Job Title: Python Developer
   </u>
  </strong>
 </p>
 <p>
  <br/>
 </p>
 <p>
  <strong>
   Experience: 5+Years
  </strong>
 </p>
 <p>
  <strong>
   Location: Vijayawada
  </strong>
 </p>
 <p>
  <strong>
   Mode of Interview : 2 round of interview
  </strong>
 </p>
 <p>
  <strong>
   TP1 : F2F
  </strong>
 </p>
 <p>
  <br/>
 </p>
 <p>
  <strong>
   Detailed Responsibilities
  </strong>
 </p>
 <p>
  <strong>
   Skill Requirements:
  </strong>
 </p>
 <p>
  <br/>
 </p>
 <ul>
  <li>
   Proven experience in Python development, including object-oriented programming and scripting.
  </li>
  <li>
   Strong hands-on expertise in PySpark for large-scale data processing.
  </li>
  <li>
   Proficiency in SQL for data extraction, manipulation, and analysis.
  </li>
  <li>
   Experience with distributed computing frameworks and big data ecosystems.
  </li>
  <li>
   Solid understanding of software development life cycle (SDLC) and agile methodologies.
  </li>
  <li>
   Familiarity with version control systems such as Git.
  </li>
  <li>
   Excellent problem-solving skills and attention to detail.
  </li>
  <li>
   Effective communication and teamwork abilities.
  </li>
 </ul>
 <p>
 </p>
</div>


Python Developer

Tata Consultancy Services (TCS)

<div></div><div><div><div><div><p>Role: Big Data/Spark/Scala</p>
<p>
Exp:5-10 years</p>
<p>
Location: Bangalore, Bhubaneswar, Pune</p>
<p>
Job Responsibilities:</p>
<p>
Design, build, and maintain scalable big data pipelines for processing and analyzing large datasets using Spark, Airflow, Bodo, Flume, Flink, etc.</p>
<ul><li>Utilize technologies like Hadoop, Spark, Kafka, and NoSQL databases.</li></ul>
<ul><li>Develop data ingestion, transformation, and aggregation strategies.</li></ul>
<ul><li>Design and implement data warehouses and data marts using Presto, Snowflake, or similar technologies.</li></ul>
<ul><li>Create efficient data models and schemas for optimal query performance.</li></ul>
<ul><li>Great to have an understanding of graph query (GQL), Gremlin, Cypher, etc.</li></ul>
<ul><li>Know to build business system-engineering architectures using graphs, how to scale the workload on distributed systems.</li></ul>
<ul><li>Hands on experience applying graphs for business impact, including experience on data management, infrastructure, budgeting, trade-offs, project workflow management, and business processes engineering.</li></ul>
<ul><li>Strong programming skills: Python/Scala/Rust for Supercomputing on TB+ data.</li></ul>
<ul><li>Experience with creating quantitative analytics and business operation dashboards using Tableau, SuperSet or other visualisation tools.</li></ul>
<ul><li>Experience with data mining, relational and noSQL databases, and data warehouse for data automation.</li></ul>
<ul><li>Experience working with large language model and natural language processing is a plus.</li></ul>
<ul><li>Excellent problem solving, critical thinking backed by expert understanding of probability, statistics, algorithms and mathematics.</li></ul>
<ul><li>Be able to strategize and innovate business solutions for high impact projects.</li></ul>
<ul><li>Be able to evaluate and transform latest, untested academic methods into production workflows.</li></ul>
<ul><li>Self-driven, highly motivated and ability to work both independently and within a team.</li></ul>
<p>
- Operate optimally in fast pace development environment with dynamic changes, tight deadlines and limited resources</p><br>
<p></p>
</div></div></div><div><div><div><b>Location</b></div>
<div>Hyderabad</div>
</div><div><div><b>Job Function</b></div>
<div>TECHNOLOGY</div>
</div><div><div><b>Role</b></div>
<div>Developer</div>
</div><div><div><b>Job Id</b></div>
<div>377325</div>
</div><div><div><b>Desired Skills</b></div>
<div>Big Data | Spark</div>
</div></div><p>Desired Candidate Profile
</p><div><div><b>Qualifications</b> : BACHELOR OF ENGINEERING</div></div></div><div></div>

Developer

MASTER-WORKS

<div><h3 class="jobSectionHeader"><b>Enterprise Data Warehouse ETL Developer (Ab Initio Specialist)</b></h3><h3 class="jobSectionHeader"><b>
Position Overview</b></h3><p>
We are seeking an experienced ETL Developer with specialized expertise in Ab Initio for developing complex data integration solutions for Teradata EDW. The candidate must have strong telecommunications domain knowledge and ability to translate technical specifications into high-performance ETL solutions.</p><p><b>
Experience Required:</b> Minimum 5+ years in ETL development with Ab Initio</p><p>
</p><h3 class="jobSectionHeader"><b>Core Responsibilities</b></h3><p><b>
Ab Initio Development</b></p><ul>
<li>
Design and develop complex ETL workflows using <b>Ab Initio GDE</b> for Teradata EDW</li>
<li>Build scalable data pipelines using Ab Initio components (Reformat, Join, Rollup, Dedup, Partition, Sort, Lookup)</li>
<li>Develop reusable components, sub-graphs, and implement error handling, checkpoints, and recovery mechanisms</li>
<li>Optimize performance through parallelization, partitioning, and efficient component usage</li>
<li>Work with Ab Initio EME, Control Center, Conduct&gt;It, and Co&gt;Operating System</li>
</ul><p><b>
Data Integration &amp; Loading</b></p><ul>
<li>
Integrate data from Oracle, flat files, XML, JSON into Teradata using Ab Initio</li>
<li>Implement Ab Initio to Teradata connectivity using BTEQ, FastLoad, MultiLoad, TPump</li>
<li>Develop incremental/full load strategies, CDC logic, and SCD Types 1, 2, 3 implementations</li>
<li>Create data quality checks, validation rules, and reconciliation processes</li>
<li>Work with <b>Cloudera Data Platform (CDP)</b> for big data integration (Plus)</li>
</ul><p><b>
Documentation &amp; Collaboration</b></p><ul>
<li>
Translate LLD and mapping documents into Ab Initio graphs</li>
<li>Create technical documentation: design docs, graph specifications, deployment guides</li>
<li>Collaborate with data modelers, architects, QA teams, and Teradata DBAs</li>
<li>Develop Ab Initio scripts (.pset, .ksh, .plan files) and Unix shell scripts for automation</li>
</ul><p><b>
Production Support</b></p><ul>
<li>
Troubleshoot Ab Initio job failures, analyze logs, and implement performance tuning</li>
<li>Manage version control (Git/SVN) and coordinate deployments</li>
<li>Monitor jobs through Control Center and provide production support</li>
</ul><p>
</p><h3 class="jobSectionHeader"><b>Required Skills</b></h3><p><b>
Ab Initio (Primary - Required)</b></p><ul>
<li>
5+ years hands-on experience with Ab Initio GDE, Co&gt;Operating System, Conduct&gt;It, Express&gt;It</li>
<li>Advanced expertise in Ab Initio components, graph development, and performance tuning</li>
<li>Ab Initio DML, parameter management, EME, and deployment procedures</li>
<li>Unix/Linux shell scripting for Ab Initio automation</li>
</ul><p><b>
Teradata &amp; Database (Required)</b></p><ul>
<li>
Strong Teradata SQL, utilities (BTEQ, FastLoad, MultiLoad, TPump, TPT)</li>
<li>Understanding of Teradata architecture (AMP, PE, partitioning, indexing: PI, SI, NUSI)</li>
<li>Oracle SQL and PL/SQL for source data extraction</li>
</ul><p><b>
Big Data (Plus)</b></p><ul><li>
Cloudera Data Platform (CDP) experience; Hadoop ecosystem (Hive, HDFS, Sqoop)</li></ul><p><b>
Telecommunications Domain (Required)</b></p><ul>
<li>
Strong understanding of telecom processes, data flows, and industry standards</li>
<li><b>OSS</b>: Network inventory, provisioning, fault/performance management</li>
<li><b>BSS</b>: Billing, customer management, order management, revenue assurance</li>
<li>Telecom KPIs: ARPU, churn rate, network utilization</li>
</ul><p><b>
Professional Skills</b></p><ul>
<li>
Full SDLC (Agile/Scrum, Waterfall); strong analytical and debugging skills</li>
<li>Good communication for technical and business collaboration</li>
</ul><p>
</p><h3 class="jobSectionHeader"><b>Preferred Qualifications</b></h3><ul>
<li>Bachelor's degree in Computer Science or related field</li>
<li>Ab Initio certification</li>
<li>Experience with Ab Initio Express&gt;It for data quality</li>
</ul><p>
</p><h3 class="jobSectionHeader"><b>Key Deliverables</b></h3><ul>
<li>Functional Ab Initio ETL graphs with parameter files and scripts</li>
<li>Technical documentation: design docs, graph specs, deployment guides</li>
<li>Unit tested and optimized code ready for production</li>
<li>Data reconciliation reports and performance tuning recommendations</li></ul></div>

EDW Developer

Apple

<div>Apple is where individual imaginations gather together, committing to the values that lead to great work. Every new product we build, service we create, or Apple Store experience we deliver is the result of us making each other&rsquo;s ideas stronger. That happens because every one of us shares a belief that we can make something wonderful and share it with the world, changing lives for the better. It&rsquo;s the diversity of our people and their thinking that inspires the innovation that runs through everything we do. When we bring everybody in, we can do the best work of our lives. Here, you&rsquo;ll do more than join something - you&rsquo;ll add something.<br>
<br>
As a BigData Software Engineer, you are an integral part of a small data-centric team driving large-scale data infrastructure and processes development, implementation, and improvement. Our organization thrives on collaborative partnerships. Join and play a key role in developing and driving the adoption of Data Mesh and data-centric micro-services. Apple's Manufacturing Systems and Infrastructure (MSI) team is responsible for capturing, consolidating and tracking all manufacturing data for Apple&rsquo;s products and modules worldwide.
<br>
<br>
Our tools enable teams to confidently use data to shape the next generation of product manufacturing at Apple. We seek a practitioner with experience building large-scale data platforms, analytic tools, and solutions. If you are passionate about making data easily accessible, trusted, and available across the entire business at scale, we'd love to hear from you.<br>
<br>
<b>Description</b><br>
As a BigData Engineer, you will lead the development and implementation of our cloud-based data infrastructure. You will work closely with cross-functional teams to understand data requirements, design scalable solutions, and ensure the integrity and availability of our data. The ideal candidate will have a deep understanding of cloud technologies, data engineering best practices, and a proven track record of successfully delivering complex data projects.<br>
<br>
<ul><li>Lead the design and implementation of cloud-based data architectures.</li></ul><br>
<ul><li>Collaborate with data scientists, analysts, and business stakeholders to understand data requirements.</li></ul><br>
<ul><li>Implement data security and privacy best practices.</li></ul><br>
<ul><li>Optimize data storage, retrieval, and processing for performance and cost efficiency.</li></ul><br>
<ul><li>Lead efforts to ensure data quality, consistency, and accuracy.</li></ul><br>
<ul><li>Mentor and provide technical guidance to junior data engineers.</li></ul><br>
<ul><li>Stay current with industry trends and emerging technologies in cloud data engineering.</li></ul><br>
<b>
Preferred Qualifications</b><br>
Certification in cloud platforms (AWS Certified Big Data - Specialty, Microsoft Certified: Azure Data Engineer, etc.)<br>
Experience building data streaming solutions using Apache Spark, Storm, Link, Flume<br>
Experience using ETL tools and technologies like Reltio, Stibo, Informatics, Apache Atlas<br>
Knowledge of machine learning and data science concepts<br>
Contributions to the open source community<br>
Familiarity with data governance and compliance<br>
Strong problem-solving and analytical skills<br>
Excellent communication and leadership skills<br>
<br>
<b>Minimum Qualifications</b><br>
Bachelor&rsquo;s or Master's degree in Computer Science or a related field<br>
Proven experience as a Big Data developer, with a focus on cloud platforms<br>
Strong proficiency in cloud services such as AWS, Azure, or Google Cloud<br>
Expertise in data modelling and data integration<br>
Experience with big data technologies (Hadoop, Spark, etc.)<br>
Proficient in programming languages such as Python, Java, Scala, GoLang<br>
Knowledge of data warehouse solutions (Redshift, BigQuery, Snowflake, Druid)<br>
Experience with streaming data technologies (Kafka, Kinesis)&quot;,&quot;internalDetails&quot;:null,&quot;eeoContent&quot;:null</div>

Big Data Software Engineer - Manufacturing Systems & Infrastructure

Search 10000+ Big data developer jobs near you this November. New roles added regularly across leading industries. Apply today.