pyspark developer

CRUTZ LEELA ENTERPRISES

<p>Sr. Spark Big Data Developer</p><p>PFB the detailed JD:</p><p>&#65533;&#65533; Experience: 5 to 7+ years (Hands-on)</p><p>&#65533;&#65533; Notice Period: 0-30 Days</p><p>Must Have:</p><p>&#61623; Proficiency in at least one of the following programming languages:</p><p>Java/Scala/Python</p><p>&#61623; Good understanding of SQL</p><p>&#61623; Experience of development and deployment of at least one end-to-end data</p><p>storage/processing pipeline</p><p>&#61623; Strong Experience in Spark development with batch and streaming</p><p>&#61623; Intermediate level expertise in HDFS and Hive</p><p>&#61623; Experience with Pyspark and Data Engineering</p><p>&#61623; ETL implementation and migration to spark</p><p>&#61623; Experience of working with Hadoop cluster</p><p>&#61623; Python, PySpark, Data Bricks developer with knowledge of cloud</p><p>&#61623; Experience with Kafka and Spark streaming (Dstream and Structured Streaming)</p><p>&#61623; Experience with using Jupyter notebooks or any other developer tool</p><p>&#61623; Experience with Airflow or other workflow engines</p><p>&#61623; Good communication skills and logical skills</p><p>Good to Have Skills:</p><p>&#61623; Prior experience of writing Spark jobs using Java is highly appreciated</p><p>&#61623; Prior experience of working with Cloudera Data Platform (CDP)</p><p>&#61623; Hands-on experience with NoSQL databases like HBase, Cassandra,</p><p>Elasticsearch, etc.</p><p>&#61623; Experience of using maven and git</p><p>&#61623; Agile scrum methodologies</p><p>&#61623; Flink and Kudu streaming</p><p>&#61623; Automation of workflows CI/CD</p><p>&#61623; Nifi streaming and transformation</p><p>Job Types: Full-time, Permanent</p><p>Pay: &#8377;500,000.00 - &#8377;1,500,000.00 per year</p><p>Benefits:</p><ul><li>Provident Fund</li></ul><p>Work Location: In person</p>

Sr. Spark Big Data Developer

Virtusa

<div>We are seeking an experienced Databricks Developer with 5 to 7 years of expertise in big data engineering and analytics platforms. The ideal candidate will have deep experience in Apache Spark Databricks PySpark and cloud platforms such as Azure or AWS and a strong background in data pipeline development performance optimization and data governance.
<br>
<br>
Key Responsibilities
<br>
Design develop and maintain scalable data pipelines using Apache Spark on Databricks
<br>
Write efficient and production ready PySpark or Scala code for data transformation and ETL processes
<br>
Integrate data from various structured and unstructured sources into a unified platform
<br>
Implement Delta Lake and manage data versioning updates and schema evolution
<br>
Optimize data processing workflows for performance scalability and cost efficiency
<br>
Collaborate with data scientists analysts and business stakeholders to deliver high quality datasets
<br>
Implement data quality checks validation routines and logging mechanisms
<br>
Monitor and debug production jobs using Databricks jobs notebooks and clusters
<br>
Ensure security privacy and compliance standards are met throughout the data lifecycle
<br>
Provide guidance and mentorship to junior team members
<br>
<br>
Required Skills and Qualifications
<br>
5 to 7 years of experience in Big Data development
<br>
Hands on experience with Databricks including Workflows Notebooks Delta Live Tables Unity Catalog
<br>
Strong programming skills in PySpark and or Scala
<br>
Solid understanding of Delta Lake architecture
<br>
Proficient in SQL for data analysis and transformation
<br>
Experience with cloud platforms such as Azure Azure Data Lake Data Factory Synapse or AWS S3 Glue Redshift
<br>
Familiarity with CICD for Databricks deployments using GitHub Actions Azure DevOps
<br>
Knowledge of data governance cataloguing and security best practices
<br>
Experience working in an Agile Scrum environment
<br>
<br>
Preferred Skills
<br>
Experience with Databricks Unity Catalog and Delta Live Tables
<br>
Exposure to machine learning workflows in Databricks
<br>
Experience with Apache Airflow Kafka or other orchestration messaging tools
<br>
Certifications Databricks Certified Data Engineer Associate Professional Azure AWS certification<div><p><b>
About Virtusa</b></p>
<p>Teamwork, quality of life, professional and personal development: values that Virtusa is proud to embody. When you join us, you join a team of 27,000 people globally that cares about your growth &mdash; one that seeks to provide you with exciting projects, opportunities and work with state of the art technologies throughout your career with us.</p>
<p>Great minds, great potential: it all comes together at Virtusa. We value collaboration and the team environment of our company, and seek to provide great minds with a dynamic place to nurture new ideas and foster excellence.</p>
<p>Virtusa was founded on principles of equal opportunity for all, and so does not discriminate on the basis of race, religion, color, sex, gender identity, sexual orientation, age, non-disqualifying physical or mental disability, national origin, veteran status or any other basis covered by appropriate law. All employment is decided on the basis of qualifications, merit, and business need.</p></div></div><p></p>

Databrick

DataBrick

Tata Consultancy Services (TCS)

<div></div><div><div><div><div><div>PYSPARK, PYTHON, SQL</div><br>
<div><br>
</div><div>Must Have Technical/Functional Skills
</div><div>Primary Skill: PYSPARK , PYTHON SQL
</div><div>Secondary: UI &ndash; ORACLE &lt; UNIX
</div><div>Experience: Minimum 7 years</div><br>
<div><br>
</div><div>Roles &amp; Responsibilities
</div><div>Resource to design, develop, maintain and test software, collaborating with the development team and business partners to ensure successful delivery and implementation of application solutions in an agile environment. Should be hands on and able to successfully deliver on time without defects. Ability to analyze, design and support implementation of Oracle PL/SQL, Views, create/modify database tables based on business requirements and specifications</div><br>
<div><br>
</div><div>Responsibilities
</div><ul><li>Responsible for developing code in PYSPARK , PYTHON
</li></ul><ul><li>Responsible to develop/modify PL/SQL, SQL Views in Oracle
</li></ul><ul><li>Responsible to create a new tables/modify existing table structure.
</li></ul><ul><li>Performance tuning and good understanding of explain plan
</li></ul><ul><li>Normalization, renormalization and star schema
</li></ul><ul><li>Develops stored procedures, database triggers and SQL queries
</li></ul><ul><li>Must be able to work independently with minimum supervision.
</li></ul><ul><li>Responsible to develop test cases and perform unit testing.
</li></ul><ul><li>Must be able to interact and co-ordinate with distributed team.
</li></ul><ul><li>Must be good in communication skills.
</li></ul><ul><li>Responsible to provide regular task status update to the management
</li></ul><ul><li>Understand and be able to visualize and manage upstream/downstream dependencies.
</li></ul><ul><li>Work in Agile environments</li></ul><br>
<div><br>
</div><div>Salary Range- $95,000-$115,000 a year
</div>
<div><p>#LI-NR3</p>
<p><b>TCS Employee Benefits Summary:</b></p><br>
<p></p>
<ul><li>Discretionary Annual Incentive.</li>
<li>Comprehensive Medical Coverage: Medical &amp; Health, Dental &amp; Vision, Disability Planning &amp; I nsurance, Pet Insurance Plans.</li>
<li>Family Support: Maternal &amp; Parental Leaves.</li>
<li>Insurance Options: Auto &amp; Home Insurance, Identity Theft Protection.</li>
<li>Convenience &amp; Professional Growth: Commuter Benefits &amp; Certification &amp; Training Reimbursement.</li>
<li>Time Off: Vacation, Time Off, Sick Leave &amp; Holidays.</li>
<li>Legal &amp; Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.</li></ul><br>
</div>
</div></div></div><div></div><br>
<div><div><div><b>Location</b></div>
<div>Jersey City, NJ</div>
</div><div><div><b>Job Function</b></div>
<div>TECHNOLOGY</div>
</div><div><div><b>Role</b></div>
<div>Engineer</div>
</div><div><div><b>Job Id</b></div>
<div>380537</div>
</div><div><div><b>Desired Skills</b></div>
<div>Python | SQL</div>
</div><div><div><b>Salary Range</b></div>
<div>$95,000-$115,000 a year</div>
</div></div><p>Desired Candidate Profile
</p><div><div><b>Qualifications</b> : BACHELOR OF COMPUTER SCIENCE</div></div></div><div></div>

Engineer

Cognizant Technology Solutions

<div><p><b>Skills- Pyspark</b></p><p><b>
Experience: 6 to 9 years</b></p><p><b>
Location: AIA-Pune</b></p><p><b>
</b></p><p><b>Responsibilities</b></p><ul><li>
Develop and maintain scalable data pipelines using Python and PySpark.</li><li>
Collaborate with data engineers and data scientists to understand and fulfil data processing needs.</li><li>
Optimize and troubleshoot existing PySpark applications for performance improvements.</li><li>
Write clean, efficient, and well-documented code following best practices.</li><li>
Participate in design and code reviews.</li><li>
Develop and implement ETL processes to extract, transform, and load data.</li><li>
Ensure data integrity and quality throughout the data lifecycle.</li><li>
Stay current with the latest industry trends and technologies in big data and cloud computing.</li></ul></div><p></p>

Pyspark

NatWest Group

<p></p><div><p>Join us as a Software Engineer, PySpark</p><p></p><ul><li>
This is an opportunity for a technically minded individual to join us as a Software Engineer</li><li>
You&rsquo;ll be designing, producing, testing and implementing working software, working across the lifecycle of the system</li><li>
Hone your existing software engineering skills and advance your career in this critical role</li><li>
We're offering this role at vice president level</li></ul><p></p><h2 class="jobSectionHeader"><b>
What you'll do</b></h2><p></p><p>
Working in a permanent feature team, you&rsquo;ll be developing knowledge of aspects of the associated platform across the disciplines of business, applications, data and infrastructure. You&rsquo;ll also be liaising with principal engineers, architects in the domain and other key stakeholders to understand how the platform works and how it supports business objectives.</p><p></p><p>
You&rsquo;ll also be:</p><p></p><ul><li>
Applying Agile methods to the development of software on the backlog</li><li>
Producing resilient and long-lived software and acting flexibly to cope with future needs</li><li>
Delivering intentional architecture and formulating emergent design through innovative ideas, experimentation and prototyping</li><li>
Designing and developing software with a focus on the automation of build, test and deployment activities, using executable patterns</li></ul><p></p><h2 class="jobSectionHeader"><b>
The skills you'll need</b></h2><p></p><p>
We&rsquo;re looking for someone with strong full stack experience in software design and implementation, including being able to exploit programming languages to solve complex problems. You&rsquo;ll also need at least twelve years of experience with Python or PySpark with AWS and Airflow.</p><p></p><p>
You'll need experience in writing advance Spark SQL or ANSI SQL queries with experience Spark SQL query optimistaion and performance tuning. You'll also need experience with continuous integration, DevOps, Gitlab and Artifactory.</p><p></p><p>
Additionally, you&rsquo;ll demonstrate:</p><p></p><ul><li>
Experience of leading the implementation of programming best practice, especially around scalability, automation, virtualisation, optimisation, availability and performance</li><li>
Sound collaboration skills with the ability to work with business teams to produce pragmatic solutions that work for the business</li><li>
Experience of information security policies and practices within the financial sector</li><li>
Strong stakeholder management skills and communication skills with the ability to communicate complex technical concepts in a simple way</li></ul></div>

Software Engineer, PySpark, VP

Hudson agile ventures private limited

<p>Net Developer</p><p>Location: Bellevue, WA &ndash; Remote [ only washington local us citizen and GC , H4 EAD</p><p>W2 ONLY ]</p><p>Experience : 10 + to 15 +</p><p>Contract : 35/hr to 45/hr</p><p>[Duration: long term</p><p>Job Description: -<br/>We are seeking a highly skilled .NET Senior Lead / Architect to join our team and lead the design and development of a scalable platform using .NET Framework 4.x or .NET 5 and above. The successful candidate will work closely with stakeholders, developers, and DevOps engineers to ensure the microservices platform meets the business requirements and technical standards.<br/>Responsibilities:<br/>Develop API solutions by building REST Based Microservices using ASP.NET Core , SYNAPSE , PYSPARK | Open-source web framework for .NET and C# for eCommerce systems.</p><p>Job Type: Contract</p><p>Pay: $35.00 - $45.00 per hour</p><p>Experience:</p><ul><li>.NET Developer: 10 years (Preferred)</li><li>SYNAPSE: 10 years (Preferred)</li><li>PYSPARK: 10 years (Preferred)</li><li>DevOps: 10 years (Preferred)</li><li>API: 10 years (Preferred)</li><li>Rest: 10 years (Preferred)</li><li>Net Architect: 10 years (Preferred)</li><li>Microservices: 10 years (Preferred)</li><li>.NET Core: 10 years (Preferred)</li><li>C#: 10 years (Preferred)</li></ul><p>Work Location: Remote</p>

Sr .Net Developer

Search 3423+ Pyspark developer jobs near you this November. New roles added regularly across leading industries. Apply today.