spark developer

CRUTZ LEELA ENTERPRISES

<p>Sr. Spark Big Data Developer</p><p>PFB the detailed JD:</p><p>&#65533;&#65533; Experience: 5 to 7+ years (Hands-on)</p><p>&#65533;&#65533; Notice Period: 0-30 Days</p><p>Must Have:</p><p>&#61623; Proficiency in at least one of the following programming languages:</p><p>Java/Scala/Python</p><p>&#61623; Good understanding of SQL</p><p>&#61623; Experience of development and deployment of at least one end-to-end data</p><p>storage/processing pipeline</p><p>&#61623; Strong Experience in Spark development with batch and streaming</p><p>&#61623; Intermediate level expertise in HDFS and Hive</p><p>&#61623; Experience with Pyspark and Data Engineering</p><p>&#61623; ETL implementation and migration to spark</p><p>&#61623; Experience of working with Hadoop cluster</p><p>&#61623; Python, PySpark, Data Bricks developer with knowledge of cloud</p><p>&#61623; Experience with Kafka and Spark streaming (Dstream and Structured Streaming)</p><p>&#61623; Experience with using Jupyter notebooks or any other developer tool</p><p>&#61623; Experience with Airflow or other workflow engines</p><p>&#61623; Good communication skills and logical skills</p><p>Good to Have Skills:</p><p>&#61623; Prior experience of writing Spark jobs using Java is highly appreciated</p><p>&#61623; Prior experience of working with Cloudera Data Platform (CDP)</p><p>&#61623; Hands-on experience with NoSQL databases like HBase, Cassandra,</p><p>Elasticsearch, etc.</p><p>&#61623; Experience of using maven and git</p><p>&#61623; Agile scrum methodologies</p><p>&#61623; Flink and Kudu streaming</p><p>&#61623; Automation of workflows CI/CD</p><p>&#61623; Nifi streaming and transformation</p><p>Job Types: Full-time, Permanent</p><p>Pay: &#8377;500,000.00 - &#8377;1,500,000.00 per year</p><p>Benefits:</p><ul><li>Provident Fund</li></ul><p>Work Location: In person</p>

Sr. Spark Big Data Developer

Recruitment Hub 365

<p><b>Job Title: Big Data Hadoop Developer</b><br/><b>Location: Hyderabad</b><br/><b>Work mode: 5 Days - (WFO)</b><br/><b>Experience Required: 4+ years</b><br/><b>Key Skills: Big Data, Hadoop, Spark, Hive</b></p><p><b>Job Overview</b><br/>We are looking for an experienced Big Data Hadoop Developer to design, build, and optimize large-scale data processing systems. The ideal candidate should have strong expertise in the Hadoop ecosystem, hands-on experience with Spark and Hive, and the ability to develop efficient ETL pipelines.</p><p><b>Key Responsibilities</b></p><ul><li>Design, develop, and maintain scalable big data systems and data pipelines.</li><li>Implement data processing frameworks using <b>Hadoop, Spark, and Hive</b>.</li><li>Develop and manage ETL workflows to ensure data accuracy, consistency, and availability.</li><li>Collaborate with data architects, analysts, and business teams to translate requirements into robust big data solutions.</li><li>Perform data validation, quality checks, and issue resolution for data pipelines.</li><li>Optimize data storage, query performance, and cluster utilization in the Hadoop ecosystem.</li><li>Ensure compliance with security, governance, and data management standards.</li></ul><p><b>Required Skills &amp; Qualifications</b></p><ul><li>4+ years of hands-on experience in <b>Big Data technologies</b>.</li><li>Strong understanding of the <b>Hadoop ecosystem</b> (HDFS, MapReduce, YARN).</li><li>Proficiency in <b>Apache Spark</b> (batch and streaming) and <b>Hive</b>.</li><li>Experience in building and maintaining data pipelines and ETL processes.</li><li>Strong knowledge of data optimization, partitioning, and performance tuning.</li><li>Familiarity with NoSQL databases such as <b>HBase, Cassandra, or MongoDB</b> is an advantage.</li><li>Experience with programming/scripting languages: <b>Java, Scala, Python, or Shell</b>.</li><li>Strong analytical and problem-solving skills.</li></ul><p>Job Type: Full-time</p><p>Pay: &#8377;1,500,000.00 - &#8377;2,400,000.00 per year</p><p>Application Question(s):</p><ul><li>How many years of experience you have with Big Data, Hadoop, Spark, Hive?</li><li>Mention your Notice Period, Current CTC and ECTC.</li><li>How many years of experience you have with Big Data / Hadoop Developer?</li></ul><p>Work Location: In person</p>

Big Data/Hadoop Developer

Tata Consultancy Services (TCS)

<div></div><div><div><div><div><p>Role: Big Data/Spark/Scala</p>
<p>
Exp:5-10 years</p>
<p>
Location: Bangalore, Bhubaneswar, Pune</p>
<p>
Job Responsibilities:</p>
<p>
Design, build, and maintain scalable big data pipelines for processing and analyzing large datasets using Spark, Airflow, Bodo, Flume, Flink, etc.</p>
<ul><li>Utilize technologies like Hadoop, Spark, Kafka, and NoSQL databases.</li></ul>
<ul><li>Develop data ingestion, transformation, and aggregation strategies.</li></ul>
<ul><li>Design and implement data warehouses and data marts using Presto, Snowflake, or similar technologies.</li></ul>
<ul><li>Create efficient data models and schemas for optimal query performance.</li></ul>
<ul><li>Great to have an understanding of graph query (GQL), Gremlin, Cypher, etc.</li></ul>
<ul><li>Know to build business system-engineering architectures using graphs, how to scale the workload on distributed systems.</li></ul>
<ul><li>Hands on experience applying graphs for business impact, including experience on data management, infrastructure, budgeting, trade-offs, project workflow management, and business processes engineering.</li></ul>
<ul><li>Strong programming skills: Python/Scala/Rust for Supercomputing on TB+ data.</li></ul>
<ul><li>Experience with creating quantitative analytics and business operation dashboards using Tableau, SuperSet or other visualisation tools.</li></ul>
<ul><li>Experience with data mining, relational and noSQL databases, and data warehouse for data automation.</li></ul>
<ul><li>Experience working with large language model and natural language processing is a plus.</li></ul>
<ul><li>Excellent problem solving, critical thinking backed by expert understanding of probability, statistics, algorithms and mathematics.</li></ul>
<ul><li>Be able to strategize and innovate business solutions for high impact projects.</li></ul>
<ul><li>Be able to evaluate and transform latest, untested academic methods into production workflows.</li></ul>
<ul><li>Self-driven, highly motivated and ability to work both independently and within a team.</li></ul>
<p>
- Operate optimally in fast pace development environment with dynamic changes, tight deadlines and limited resources</p><br>
<p></p>
</div></div></div><div><div><div><b>Location</b></div>
<div>Hyderabad</div>
</div><div><div><b>Job Function</b></div>
<div>TECHNOLOGY</div>
</div><div><div><b>Role</b></div>
<div>Developer</div>
</div><div><div><b>Job Id</b></div>
<div>377325</div>
</div><div><div><b>Desired Skills</b></div>
<div>Big Data | Spark</div>
</div></div><p>Desired Candidate Profile
</p><div><div><b>Qualifications</b> : BACHELOR OF ENGINEERING</div></div></div><div></div>

Developer

Virtusa

<div>We are seeking an experienced Databricks Developer with 5 to 7 years of expertise in big data engineering and analytics platforms. The ideal candidate will have deep experience in Apache Spark Databricks PySpark and cloud platforms such as Azure or AWS and a strong background in data pipeline development performance optimization and data governance.
<br>
<br>
Key Responsibilities
<br>
Design develop and maintain scalable data pipelines using Apache Spark on Databricks
<br>
Write efficient and production ready PySpark or Scala code for data transformation and ETL processes
<br>
Integrate data from various structured and unstructured sources into a unified platform
<br>
Implement Delta Lake and manage data versioning updates and schema evolution
<br>
Optimize data processing workflows for performance scalability and cost efficiency
<br>
Collaborate with data scientists analysts and business stakeholders to deliver high quality datasets
<br>
Implement data quality checks validation routines and logging mechanisms
<br>
Monitor and debug production jobs using Databricks jobs notebooks and clusters
<br>
Ensure security privacy and compliance standards are met throughout the data lifecycle
<br>
Provide guidance and mentorship to junior team members
<br>
<br>
Required Skills and Qualifications
<br>
5 to 7 years of experience in Big Data development
<br>
Hands on experience with Databricks including Workflows Notebooks Delta Live Tables Unity Catalog
<br>
Strong programming skills in PySpark and or Scala
<br>
Solid understanding of Delta Lake architecture
<br>
Proficient in SQL for data analysis and transformation
<br>
Experience with cloud platforms such as Azure Azure Data Lake Data Factory Synapse or AWS S3 Glue Redshift
<br>
Familiarity with CICD for Databricks deployments using GitHub Actions Azure DevOps
<br>
Knowledge of data governance cataloguing and security best practices
<br>
Experience working in an Agile Scrum environment
<br>
<br>
Preferred Skills
<br>
Experience with Databricks Unity Catalog and Delta Live Tables
<br>
Exposure to machine learning workflows in Databricks
<br>
Experience with Apache Airflow Kafka or other orchestration messaging tools
<br>
Certifications Databricks Certified Data Engineer Associate Professional Azure AWS certification<div><p><b>
About Virtusa</b></p>
<p>Teamwork, quality of life, professional and personal development: values that Virtusa is proud to embody. When you join us, you join a team of 27,000 people globally that cares about your growth &mdash; one that seeks to provide you with exciting projects, opportunities and work with state of the art technologies throughout your career with us.</p>
<p>Great minds, great potential: it all comes together at Virtusa. We value collaboration and the team environment of our company, and seek to provide great minds with a dynamic place to nurture new ideas and foster excellence.</p>
<p>Virtusa was founded on principles of equal opportunity for all, and so does not discriminate on the basis of race, religion, color, sex, gender identity, sexual orientation, age, non-disqualifying physical or mental disability, national origin, veteran status or any other basis covered by appropriate law. All employment is decided on the basis of qualifications, merit, and business need.</p></div></div><p></p>

Databrick

DataBrick

Cognizant Technology Solutions

<div><p><b>Skills- Databricks+ Pyspark</b></p><p><b>
Experience: 4 to 13 years</b></p><p></p><p><b>
We are looking for a highly skilled Data Engineer with expertise in PySpark and Databricks to design, build, and optimize scalable data pipelines for processing massive datasets.</b></p><p><b>
Key Responsibilities:</b></p><ul><li>
Build &amp; Optimize Pipelines: Develop high-throughput ETL workflows using PySpark on Databricks.</li><li>
Data Architecture &amp; Engineering: Work on distributed computing solutions, optimize Spark jobs, and build efficient data models.</li><li>
Performance &amp; Cost Optimization: Fine-tune Spark configurations, optimize Databricks clusters, and reduce compute/storage costs.</li><li>
Collaboration: Work closely with Data Scientists, Analysts, and DevOps teams to ensure data reliability.</li><li>
ETL &amp; Data Warehousing: Implement scalable ETL processes for structured &amp; unstructured data.</li><li>
Monitoring &amp; Automation: Implement logging, monitoring, and alerting mechanisms for data pipeline health and fault tolerance.</li></ul></div><p></p>

Databricks + Pyspark

KMCCorp India

<div><div><h3 class="jobSectionHeader"><b>Job Information</b></h3>
<ul><div><p>Date Opened</p>
<div><b>11/19/2025</b><b>
</b></div></div><div><p>Industry</p>
<div><b>Information Technology</b><b>
</b></div></div><div><p>Job Type</p>
<div><b>Full time</b><b>
</b></div></div><div><p>Work Experience</p>
<div><b>5+ years</b><b>
</b></div></div><div><p>City</p>
<div><b>Hyderabad</b><b>
</b></div></div><div><p>State/Province</p>
<div><b>Telangana</b><b>
</b></div></div><div><p>Country</p>
<div><b>India</b><b>
</b></div></div><div><p>Zip/Postal Code</p>
<div><b>500003</b><b>
</b></div></div></ul></div><div><h3 class="jobSectionHeader"><b>Job Description</b></h3>
<div><div>Job Description<br>
<br>
The successful candidate will work on batch and real-time data pipelines, leveraging technologies such as Scala, Java, and Apache Spark. They will manage distributed systems and cloud resources, ensure security compliance using HashiCorp Vault, orchestrate workflows with Apache Airflow, and investigate production issues by analyzing logs and monitoring metrics.<br>
<br>
Responsibilities<br>
<br>
<ul><li>Develop and maintain batch and streaming data pipelines using Scala and Java.</li><li>Write and execute shell scripts and Yarn commands for Spark job management.</li><li>Manage and optimize big data environments using Apache Spark, EMR, Hadoop, and YARN.</li><li>Utilize cloud storage and container platforms such as Amazon S3 and EKS.</li><li>Implement and manage security features, including HashiCorp Vault and tokenization/encryption protocols.</li><li>Schedule and orchestrate batch workflows using Apache Airflow.</li><li>Conduct root cause analysis and handle production incidents.</li><li>Analyze application, Spark executor, and Dynatrace logs.</li><li>Monitor jobs and EMR clusters using Dynatrace metrics.</li><li>Validate data and set up production alerts.</li></ul></div><h3 class="jobSectionHeader"><b><br>
Requirements</b></h3><div>
Experience with batch and streaming processing.<br>
<ul><li>Proficiency in distributed systems management.</li><li>Knowledge of cloud platforms (AWS S3, EKS).</li><li>Security expertise (Vault, tokenization, encryption).</li><li>Workflow orchestration (Airflow).</li><li>Strong analytical and troubleshooting skills for production investigation</li></ul></div></div></div></div><p></p>

Big Data/ Scala Developer

Search 3935+ Spark developer jobs near you this November. New roles added regularly across leading industries. Apply today.