spark developer

ixceed

<p><b>Role: Data Lead (Scala+Spark)</b></p><p><b>Location: Chennai</b></p><p><b>Mode: Hybrid</b></p><p><b>Type: Permanent</b></p><p><b>Job Description:</b></p><p>Responsibilities:</p><ul><li>Design, develop, and maintain robust and scalable data pipelines using Apache Spark and Scala on the Databricks platform.</li><li>Implement ETL (Extract, Transform, Load) processes for various data sources, ensuring data quality, integrity, and efficiency.</li><li>Optimize Spark applications for performance and cost-efficiency within the Databricks environment.</li><li>Work with Delta Lake for building reliable data lakes and data warehouses, ensuring ACID transactions and data versioning.</li><li>Collaborate with data scientists, analysts, and other engineering teams to understand data requirements and deliver solutions.</li><li>Implement data governance and security best practices within Databricks.</li><li>Troubleshoot and resolve data-related issues, ensuring data availability and reliability.</li><li>Stay updated with the latest advancements in Spark, Scala, Databricks, and related big data technologies.</li></ul><p>Required Skills and Experience:</p><ul><li>Proven experience as a Data Engineer with a strong focus on big data technologies.</li><li>Expertise in the Scala programming language for data processing and Spark application development.</li><li>In-depth knowledge and hands-on experience with Apache Spark, including Spark SQL, Spark Streaming, and Spark Core.</li><li>Proficiency in using Databricks platform features, including notebooks, jobs, workflows, and Unity Catalog.</li><li>Experience with Delta Lake and its capabilities for building data lakes.</li><li>Strong understanding of data warehousing concepts, data modeling, and relational databases.</li><li>Familiarity with cloud platforms (e.g., AWS, Azure, GCP) and their data services.</li><li>Experience with version control systems like Git.</li><li>Excellent problem-solving and analytical skills.</li><li>Ability to work independently and as part of a team.</li></ul><p>Preferred Qualifications (Optional):</p><ul><li>Experience with other big data technologies like Kafka, Flink, or Hadoop ecosystem components.</li><li>Knowledge of data visualization tools.</li><li>Understanding of DevOps principles and CI/CD pipelines for data engineering.</li><li>Relevant certifications in Spark or Databrick</li></ul><p>Job Type: Permanent</p><p>Pay: &#8377;3,500,000.00 - &#8377;4,500,000.00 per year</p><p>Application Question(s):</p><ul><li>Immediate Joiner</li></ul><p>Experience:</p><ul><li>Design and build ETL: 3 years (Required)</li><li>Scala: 4 years (Required)</li><li>Spark: 4 years (Required)</li><li>Databricks: 3 years (Required)</li><li>Team Lead: 2 years (Required)</li><li>Big data architecture: 2 years (Required)</li><li>Data Engineering: 3 years (Required)</li></ul><p>Work Location: In person</p>

Data Lead (Scala+Spark)

InfoCepts

<div><div><div><div><div><p>Title: Data Engineer - AWS</p>
<p>Purpose of the Position:</p>
<p>You will be a critical member of the InfoCepts Cloud Data Architect Team. This position requires a sound knowledge on Big Data using Cloud technologies, EMR, Athena, Spark-Scala, PySpark, S3, AWS Lambda etc. Strong foundation on database concepts and SQL is also required.</p><br>
<p></p>
<p><b>Key Result Areas and Activities:</b></p><br>
<p></p>
<ul><li><p>Study existing technology landscape and understand current data integration framework and do impact assessment for the requirements.</p>
</li></ul><ul><li><p>Develop spark jobs using Scala for new project requirements.</p>
</li></ul><ul><li><p>Enhance existing spark jobs for any ongoing product enhancement.</p>
</li></ul><ul><li><p>Performance tuning of spark jobs, stress testing etc.</p>
</li></ul><ul><li><p>Create new data pipelines for developed/enhanced spark jobs using AWS Lambda or Apache Airflow</p>
</li></ul><ul><li><p>Responsible for database design process, logical design, physical design, star schema, snowflake schema etc.</p>
</li></ul><ul><li><p>Analyze data processing, integration, modelling and reporting requirements &amp; Define data loading strategies considering volume, data types, frequency and analytics specifications.</p>
</li></ul><ul><li><p>Ensure optimal balance between cost and performance.</p>
</li></ul><ul><li><p>Project documentation, Adheres to Quality guidelines &amp; Schedules.</p>
</li></ul><ul><li><p>Works hand in hand with PM for successful delivery of project and provide Estimation, scoping, scheduling assistance.</p>
</li></ul><ul><li><p>Manage build phase and quality assure code to ensure fulfilling requirements and adhering to Cloud architecture. Resolve difficult design and develop issues.</p><br>
</li></ul></div></div></div></div><p></p><div><div><div><div><div><p><b>Work and Technical Experience:</b></p>
<ul><li><p>5 years of IT experience, with at least 3 years on cloud-related projects.</p>
</li></ul><ul><li><p>Strong SQL skills and database fundamentals</p>
</li></ul><ul><li><p>Hands-on experience with AWS services: S3, EC2, EMR, Athena, Glue, Lambda</p>
</li></ul><ul><li><p>Big Data technologies: Hadoop, Hive, Spark, YARN</p>
</li></ul><ul><li><p>Programming languages: Scala, Python, Shell Scripts</p>
</li></ul><ul><li><p>Operating systems: Linux (any flavor), Windows</p>
</li></ul><ul><li><p>Experience in developing ETL workflows with complex transformations</p>
</li></ul><ul><li><p>Understanding of AWS networking: VPC, Subnets, Security Groups, Route Tables</p>
</li></ul><ul><li><p>Quick learner with adaptability to new AWS and Big Data technologies</p><br>
</li></ul><p></p><p><b>Qualifications:</b></p>
<ul><li><p>Bachelor&rsquo;s degree in computer science, Engineering, or related field</p>
</li></ul><ul><li><p>Technical certifications or evidence of continued learning</p>
</li></ul><ul><li><p>5 years of IT experience, with at least 3 years on cloud-related projects</p><br>
</li></ul><p></p><p><b>Qualities:</b></p>
<ul><li><p>Hold strong technical knowledge and experience</p>
</li></ul><ul><li><p>Should have the capability to deep dive and research in various technical related fields</p>
</li></ul><ul><li><p>Self-motivated and focused on delivering outcomes for a fast-growing team and firm</p>
</li></ul><ul><li><p>Able to communicate persuasively through speaking, writing, and client presentations</p>
</li></ul><ul><li><p>Able to consult, write, and present persuasively</p>
</li></ul><ul><li><p>Able to work in a self-organized and cross-functional team</p>
</li></ul><ul><li><p>Able to iterate based on new information, peer reviews, and feedback</p>
</li></ul><ul><li><p>Prior experience of working in a large media company would be added advantage</p>
</li></ul></div></div></div></div></div><div></div><div><div><div><div><div><b>Years Of Exp
</b></div></div><div><p>5 to 7 years</p></div></div></div></div><div></div><div><div><div><div><div><b>
Location
</b></div></div><div><p>India</p></div></div></div></div></div>

Data Engineer- AWS

AFFINITY GLOBAL ADVERTISING

<div><div><div><b>Responsibilities</b></div><div></div><ul><li><div><br>
Build Data applications with high accuracy and performance across traditional and distributed computing platforms.</div></li><li><div>
Design, build, and maintain high performance, reusable, and reliable code quality and features being delivered efficiently and on-time. Document everything.</div></li><li><div>
Develop database processes, gather, and process raw data at scale (including writing scripts, web scraping, calling APIs, write SQL queries in MySQL, handle data cloud etc.).</div></li><li><div>
Administer data processing workflows associated with tools like MySQL, Oozie, Zookeeper, Sqoop, Hive, Impala for data processing across the distributed platform.</div></li><li><div>
Work closely with our engineering team to integrate your amazing innovations and algorithms into our production systems.</div></li><li><div>
Support business decisions with ad hoc analysis as needed and troubleshoot production issues and identify practical solutions.</div></li><li><div>
Routine check-up, back-up and monitoring of the entire MySQL and Hadoop ecosystem.</div></li><li><div>
Take end-to-end responsibility of the Traditional Databases (MySQL), Big Data ETL, Analysis and processing life cycle in the organization and manage deployments of bigdata clusters across private and public cloud platforms.</div></li></ul><div></div><div><b><br>
Required Skills</b></div><ul><li><div>
4+ years of experience with SQL (MySQL) a must.<br>
2+ years of Hands-on experience working with Cloudera Hadoop Distribution platform and Apache Spark.</div></li><li><div>
Strong understanding of full dev life cycle, for backend database applications across RDBMS and distributed cloud platforms.</div></li><li><div>
Experience as a Database developer writing SQL queries, DDL/DML statements, managing databases, writing stored procedures, triggers and functions and knowledge of DB internals.</div></li><li><div>
Knowledge of database administration, performance tuning, replication, backup, and data restoration.</div></li><li><div>
Comprehensive knowledge of Hadoop Architecture and HDFS, to design, develop, document and architect Hadoop applications. Working knowledge of SQL, NoSQL, data warehousing &amp; DBA along with Map-Reduce, Hive, Impala, Kafka, HBase, Pig, and Java.</div></li><li><div>
Experience processing large amounts of structured and unstructured data, extracting, and transforming data from remote data stores, such as relational databases or distributed file systems.</div></li><li><div>
Working expertise with Apache Spark, Spark streaming, Jupyter Notebook, Python or Scala programming.</div></li><li><div>
Excellent communication skills, ability to tailor technical information for different audiences. Excellent teamwork skills, ability to self-start, share insights, ask questions, and report progress.</div></li><li><div>
Working knowledge of the general database architectures, trends, and emerging technologies. Familiarity with caching, partitioning, storage engines, query performance tuning, indexes, and distributed computing frameworks.</div></li><li><div>
Working knowledge &amp; understanding of data analytics or BI tools - like looker studio, Power BI, or any other BI tool is a must.</div></li></ul><div></div><div><b><br>
Additional Desired Skills</b></div><ul><li><div>
Adequate exposure to Linux environment and file system.</div></li><li><div>
Exposure to handling large datasets (billions of records) is a plus.</div></li><li><div>
Excellent communication skills, ability to tailor technical information for different audiences.</div></li><li><div>
Excellent teamwork skills, ability to self-start, share insights, ask questions, and report progress.</div></li><li><div>
Strong enthusiasm in solving messy problems with good solution design skills.</div></li><li><div>
Adequate experience with data analysis or software development in a Unix-based system.</div></li><li><div>
Ability to grasp emerging technical standards and apply in right direction</div></li></ul><div></div><div><div><b><br>
About Affinity:</b></div><div>
Affinity is an ad-tech company which creates user engagement products (branding and performance) for digital media. It is in the business of creating sustainable and scalable advertising/media products with special attention to user experience. Established in 2006, Affinity is a 400+ employee company that operates 7 business units, namely - mCanvas, Siteplug, VEVE, AdopsOne, Yield Solutions , Nucleus-Links and Affinity Germany . For more information, visit www.affinity.com.</div></div></div></div><div></div>

Sr. Software Engineer, Database (VEVE)

Cognizant Technology Solutions

<div><p><b>Skill:</b> Databricks</p><p><b>
Experience:</b> 6 to 9 years</p><p><b>
Location:</b> AIA Kolkata</p><p></p><p><b>
Job description</b></p><p><b>
Responsibilities:</b></p><ul><li>
Designing, building, and deploying data pipelines and workflows using Databricks.</li><li>
Developing and optimizing Apache Spark applications for data processing and analytics.</li><li>
Integrating Databricks with various data sources and platforms.</li><li>
Implementing data quality checks and monitoring processes.</li><li>
Managing and configuring the Databricks environment, including clusters and notebooks.</li><li>
Troubleshooting and resolving issues related to Databricks performance and functionality.</li><li>
Collaborating with data scientists and analysts to implement machine learning models.</li><li>
Staying up-to-date with the latest Databricks features and best practices.</li></ul><p><b>
Qualifications:</b></p><ul><li>
Strong proficiency in Python or Scala programming languages.</li><li>
Experience with Apache Spark and big data processing frameworks.</li><li>
Knowledge of cloud platforms like AWS, Azure, or GCP.</li><li>
Familiarity with data warehousing and ETL processes.</li><li>
Understanding of machine learning concepts and algorithms.</li><li>
Strong problem-solving and analytical skills.</li></ul><p><b>
Additional Skills (depending on the specific role):</b></p><ul><li>
Experience with Databricks Delta Lake or MLflow.</li><li>
Knowledge of infrastructure as code (IaC) tools like Terraform.</li><li>
Familiarity with CI/CD pipelines for data and machine learning workflows.</li><li>
Experience with containerization technologies like Docker.</li></ul></div><p></p>

Databricks

<div><p><b>Skill:</b> Databricks</p><p><b>
Experience:</b> 6 to 13 years</p><p><b>
Location:</b> AIA Group-Bhubaneswar</p><p></p><p><b>
Job description</b></p><p><b>
Responsibilities:</b></p><ul><li>
Designing, building, and deploying data pipelines and workflows using Databricks.</li><li>
Developing and optimizing Apache Spark applications for data processing and analytics.</li><li>
Integrating Databricks with various data sources and platforms.</li><li>
Implementing data quality checks and monitoring processes.</li><li>
Managing and configuring the Databricks environment, including clusters and notebooks.</li><li>
Troubleshooting and resolving issues related to Databricks performance and functionality.</li><li>
Collaborating with data scientists and analysts to implement machine learning models.</li><li>
Staying up-to-date with the latest Databricks features and best practices.</li></ul><p><b>
Qualifications:</b></p><ul><li>
Strong proficiency in Python or Scala programming languages.</li><li>
Experience with Apache Spark and big data processing frameworks.</li><li>
Knowledge of cloud platforms like AWS, Azure, or GCP.</li><li>
Familiarity with data warehousing and ETL processes.</li><li>
Understanding of machine learning concepts and algorithms.</li><li>
Strong problem-solving and analytical skills.</li></ul><p><b>
Additional Skills (depending on the specific role):</b></p><ul><li>
Experience with Databricks Delta Lake or MLflow.</li><li>
Knowledge of infrastructure as code (IaC) tools like Terraform.</li><li>
Familiarity with CI/CD pipelines for data and machine learning workflows.</li><li>
Experience with containerization technologies like Docker.</li></ul></div>

Azure Databricks, Data Factory and SQL

Walmart Global Tech India

<div>
 Position Summary...
 <br/>
 <br/>
 What you'll do...
 <br/>
 <br/>
 <strong>
  Job Description
  <br/>
  <br/>
 </strong>
 <strong>
  About Team:
  <br/>
  <br/>
 </strong>
 As part of the Replenishment &amp; Forecasting team at Walmart, you&rsquo;ll have the opportunity to work with some of the smartest engineers/architects/data scientist and will have the opportunity to solve one of the most critical problems for Walmart
 <br/>
 <br/>
 We are looking for passionate Data Engineers to join our team &amp; help us in building our next generation Replenishment &amp; Forecasting Platform for Walmart. We drive $200B+ inventory replenishments annually. Improved forecast will help drive down the inventory days of supply and reduce waste for perishable grocery &ndash; savings measured in billions of dollars &ndash; while improving our in stocks. Effectively predicting demand involves learning from past sales and modelling the effects price, promotions, store placements, local events, weather etc. Replenishment ensures we have the effective distribution of the demand to the final destinations (Stores, DCs, etc.)
 <br/>
 <br/>
 Doing this a Walmart scale &ndash; over 500 M store item combinations - makes it one of the largest and most challenging problems of its kind in the world while also being the most impactful. Talk to us if you are interested in taking up the challenge.
 <br/>
 <br/>
 <strong>
  What you'll do:
  <br/>
  <br/>
 </strong>
 <ul>
  <li>
   Write efficient and quality code that can work @ Walmart scale
  </li>
  <li>
   Learn various aspects around data elements and their relevance to business.
  </li>
  <li>
   End-to-end ownership of your business stories including development, deployment to QA and production.
  </li>
  <li>
   Be responsible to on-call support on a need basis.
  </li>
  <li>
   Adopt Walmart&rsquo;s quality standards and develop and recommend process standards and best practices across the retail industry
  </li>
  <li>
   Research, learn &amp; adapt new technologies to solve problems &amp; improve existing solutions
  </li>
  <li>
   Participate in events to build innovative solutions
  </li>
  <li>
   Adhere to company policies, procedures, mission, values, and standards of ethics and integrity.
   <br/>
   <br/>
  </li>
 </ul>
 <strong>
  What you'll bring:
  <br/>
  <br/>
 </strong>
 <ul>
  <li>
   B.E/B.Tech from a reputed institution.
  </li>
  <li>
   Minimum of 3 years of solid Java experience.
  </li>
  <li>
   Good understanding of Spring Framework.
  </li>
  <li>
   Good understanding in Big Data Ecosystem - Hadoop and Spark.
  </li>
  <li>
   Must have project experience with Spark.
  </li>
  <li>
   Ability to independently troubleshoot Spark jobs.
  </li>
  <li>
   Good understanding of distributed systems.
  </li>
  <li>
   Fast learner and quickly adapt to new technologies.
  </li>
  <li>
   Prefer individuals with high ownership and commitment.
   <br/>
   <br/>
  </li>
 </ul>
 <strong>
  About Walmart Global Tech
  <br/>
  <br/>
 </strong>
 Imagine working in an environment where one line of code can make life easier for hundreds of millions of people. That&rsquo;s what we do at Walmart Global Tech. We&rsquo;re a team of software engineers, data scientists, cybersecurity expert's and service professionals within the world&rsquo;s leading retailer who make an epic impact and are at the forefront of the next retail disruption. People are why we innovate, and people power our innovations. We are people-led and tech-empowered.
 <br/>
 <br/>
 We train our team in the skillsets of the future and bring in experts like you to help us grow. We have roles for those chasing their first opportunity as well as those looking for the opportunity that will define their career. Here, you can kickstart a great career in tech, gain new skills and experience for virtually every industry, or leverage your expertise to innovate at scale, impact millions and reimagine the future of retail.
 <br/>
 <br/>
 <strong>
  Flexible, hybrid work
  <br/>
  <br/>
 </strong>
 Walmart&rsquo;s culture sets us apart, and we know being together helps us innovate, learn and grow great careers. This role is based in our [Bangalore/Chennai] office for daily work, with the flexibility for associates to manage their personal lives.
 <br/>
 <br/>
 <strong>
  Benefits
  <br/>
  <br/>
 </strong>
 Beyond our great compensation package, you can receive incentive awards for your performance. Other great perks include a host of best-in-class benefits maternity and parental leave, pto, health benefits, and much more.
 <br/>
 <br/>
 <strong>
  Belonging
  <br/>
  <br/>
 </strong>
 We aim to create a culture where every associate feels valued for who they are, rooted in respect for the individual. Our goal is to foster a sense of belonging, to create opportunities for all our associates, customers and suppliers, and to be a Walmart for everyone.
 <br/>
 <br/>
 At Walmart, our vision is "everyone included." by fostering a workplace culture where everyone is&mdash;and feels&mdash;included, everyone wins. Our associates and customers reflect the makeup of all 19 countries where we operate. By making Walmart a welcoming place where all people feel like they belong, we&rsquo;re able to engage associates, strengthen our business, improve our ability to serve customers, and support the communities where we operate.
 <br/>
 <br/>
 <strong>
  Equal opportunity employer
  <br/>
  <br/>
 </strong>
 Walmart, inc., is an equal opportunities employer &ndash; by choice. We believe we are best equipped to help our associates, customers and the communities we serve live better when we really know them. That means understanding, respecting and valuing unique styles, experiences, identities, ideas and opinions &ndash; while being inclusive of all people.
 <br/>
 <br/>
 <strong>
  Minimum Qualifications...
  <br/>
  <br/>
 </strong>
 <em>
  <em>
   Outlined below are the required minimum qualifications for this position. If none are listed, there are no minimum qualifications.
   <br/>
   <br/>
  </em>
 </em>
 Option 1: Bachelor's degree in computer science, computer engineering, computer information systems, software engineering, or related area and 2 years&rsquo; experience in software engineering or related area.
 <br/>
 <br/>
 Option 2: 4 years&rsquo; experience in software engineering or related area.
 <br/>
 <br/>
 <strong>
  Preferred Qualifications...
  <br/>
  <br/>
 </strong>
 <em>
  <em>
   Outlined below are the optional preferred qualifications for this position. If none are listed, there are no preferred qualifications.
   <br/>
   <br/>
  </em>
 </em>
 Master&rsquo;s degree in Computer Science, Information Technology, Engineering, Information Systems, Cybersecurity, or related area, We value candidates with a background in creating inclusive digital experiences, demonstrating knowledge in implementing Web Content Accessibility Guidelines (WCAG) 2.2 AA standards, assistive technologies, and integrating digital accessibility seamlessly. The ideal candidate would have knowledge of accessibility best practices and join us as we continue to create accessible products and services following Walmart&rsquo;s accessibility standards and guidelines for supporting an inclusive culture.
 <br/>
 <br/>
 <strong>
  Primary Location...
  <br/>
  <br/>
 </strong>
 BLOCK- 1, PRESTIGE TECH PACIFIC PARK, SY NO. 38/1, OUTER RING ROAD KADUBEESANAHALLI, , India R-2305413
</div>


Software Engineer III

Srijan Technologies Pvt Ltd

<div><div><b>Material</b> is a global strategy, insights, design, and technology partner to companies striving for true customer centricity and ongoing relevance in a digital-first, customer-led world. By leveraging proprietary, science-based tools that enable human understanding, we inform and create customer-centric business models and experiences + deploy measurement systems &ndash; to build transformational relationships between businesses and the people they serve.</div>
<div><div>https://www.materialplus.io/</div></div><br>
<p></p>
<p><b>
Senior Developer/ Lead - Data Science</b></p>
<p>
We are looking for <b>Senior Developer/ Lead - Data Scientist</b> with strong Generative AI experience, skilled in Python, TensorFlow/PyTorch/Scikit&#8209;learn, Azure OpenAI GPT, and multi-agent frameworks like LangChain or AutoGen, with strong data preprocessing, feature engineering, and model evaluation expertise.</p>
<p><b>
Bonus</b>: familiarity with Big Data tools (Spark, Hadoop, Databricks, SQL/NoSQL) and ReactJS</p><br>
<p></p>
<p><b>
Experience: 5+ year</b></p>
<p><b>
Preferred Location: -</b> Gurgaon/ Bangalore</p>
<p><b>
Job Description: -</b></p>
<p><b>
Generative AI: </b>GenAI models (Eg.: Azure OpenAI GPT), Multi-Agent System Architecture:</p>
<ul>
<li>Proficiency in <b>Python</b> and AI/ML libraries (e.g., TensorFlow, PyTorch, Scikit-learn).</li>
<li>Experience with <b>LangChain</b>, <b>AutoGen</b>, or similar frameworks for multi-agent systems.</li>
<li>Strong knowledge of <b>Data Science</b> techniques, including data preprocessing, feature engineering, and model evaluation.</li></ul><br>
<p></p>
<p><b>
Nice to have:</b></p>
<ul>
<li>Familiarity with <b>Big Data</b> tools (e.g., Spark, Hadoop) / Databricks and databases (e.g., SQL, NoSQL).</li>
<li>Expertise in <b>ReactJS</b> for building responsive and interactive user interfaces.</li></ul><br>
<p></p>
<p><b>
What We Offer</b></p>
<ul>
<li>Professional Development and Mentorship.</li>
<li>Hybrid work mode with remote friendly workplace. (6 times in a row Great Place To Work Certified).</li>
<li>Health and Family Insurance.</li>
<li>40+ Leaves per year along with maternity &amp; paternity leaves.</li>
<li>Wellness, meditation and Counselling sessions</li>
</ul>
<p>If you are looking forward to it, kindly apply or share your resume with me at Vineet.kumar@materialplus.io</p></div>

Senior Developer/ Technical Lead - Data Scientist (Agentic AI)

Search 3718+ Spark developer jobs near you this October. New roles added regularly across leading industries. Apply today.