data pipeline engineer

Envision integrated services private limited

Experience:3 YearsJob Summary: We are seeking a skilled Data Mining Analyst to provide support for the ScalierCoreX team. ScalerCoreX is a product designed to collect all manufacturing data across our global plants and centralize it into a single source of truth within the Google Cloud Platform (GCP). In this role you will be tasked in designing automated, efficient, and fast processes to accelerate data retrieval from GCP for customer access. Key Responsibilities: &#61623; Aid Interim Data Delivery (ICA): Design and implement rapid, efficient data retrieval processes from GCP for immediate customer needs. &#61623; Engineer Data Pipelines: Design, build, and optimize robust GCP data pipelines to ensure clean, consistent data. &#61623; Ensure Data Quality &amp; Integrity: Implement rigorous checks and validations to maintain high data quality and prevent &quot;dirty data,&quot; applying deep CDM knowledge in pipeline design for ScalerCoreX. &#61623; Collaborate &amp; Deliver Solutions: Partner with stakeholders to deliver high performance data solutions that support real-time insights for the team. &#61623; Monitor &amp; Enhance Reliability: Continuously monitor, troubleshoot, and improve data flow reliability. Required Skills and Qualifications: &#61623; Extensive GCP data engineering expertise. &#61623; Strong SQL proficiency for data pipelines. &#61623; Proven experience with data pipeline orchestration. &#61623; Solid knowledge of the Common Data Model (CDM). &#61623; Exceptional problem-solving and collaboration skillsJob Types: Full-time, PermanentPay: &#8377;13,775.97 - &#8377;25,000.00 per monthBenefits:<ul><li>Health insurance</li><li>Provident Fund</li></ul>Work Location: In person

Data Pipeline Engineer

S&P Global

About the Role: 
Grade Level (for internal use): 10
 S&P Mobility
 
The Role: Senior Software Engineer (UI Backend and Data Pipeline Engineer)
 
The Team: 
At S&P Mobility, we’re investing in technology to develop new products that help our customers drive their growth and transformation agenda. These include new data integration, advanced analytics, and modern applications that address new customer needs and are highly visible and strategic within the organization. Do you love building products on platforms at scale while leveraging cutting-edge technology? Do you want to deliver innovative solutions to complex problems? If so, be part of our mighty team of engineers and play a key role in driving our business strategies.
 
The Impact: 
We stand at the crossroads of innovation through Data Products to bring a competitive advantage to our business through the delivery of automotive forecasting solutions. Your work will contribute to the growth and success of our organization and provide valuable insights to our clients.
 
What’s in it for you: 
We are looking for an innovative and mission-driven software/data engineer to make a significant impact by designing and developing AWS cloud-native solutions that enable analysts to forecast long and short-term trends in the automotive industry. This role requires cutting-edge data and cloud-native technical expertise as well as the ability to work independently in a fast-paced, collaborative, and dynamic work environment.
 
Responsibilities: 
Design, develop, and maintain scalable data pipelines, including complex algorithms.
 Build and maintain UI backend services using Python, C#, or similar, ensuring responsiveness and high performance.
 Ensure data quality and integrity through robust validation processes.
 Strong understanding of data integration and data modeling concepts.
 Lead data integration projects and mentor junior engineers.
 Collaborate with cross-functional teams to gather data requirements.
 Collaborate with data scientists and analysts to optimize data flow and storage for advanced analytics.
 Take ownership of the modules you work on, deliver on time and with quality, and ensure software development best practices.
 Utilize Redis for caching and data storage solutions to enhance application performance.
 
What We’re Looking For: 
Bachelor's degree in Computer Science or a related field.
 Strong analytical and problem-solving skills.
 7+ years of experience in Data Engineering/Advanced Analytics.
 Proficiency in Python and experience with Flask for backend development.
 Strong knowledge of object-oriented programming.
 AWS proficiency is a big plus: ECR, Containerization, Batch, Step Functions, Redis, DynamoDB, Lambda.
 Experience with SQL and data warehousing.
 Experience with data manipulation libraries and analytical frameworks such as Pandas, Polars, and NumPy.
 Experience with GitLab CI/CD or similar tools for continuous integration and deployment.
 
Return to Work: 
Have you taken time out for caring responsibilities and are now looking to return to work? As part of our Return-to-Work initiative, we are encouraging enthusiastic and talented returners to apply and will actively support your return to the workplace.
 
About S&P Global Mobility
 S&P Global has recently announced the intent to separate our Mobility Segment into a standalone public company.
 
About S&P Global Mobility
 At S&P Global Mobility, we provide invaluable insights derived from unmatched automotive data, enabling our customers to anticipate change and make decisions with conviction. Our expertise helps them to optimize their businesses, reach the right consumers, and shape the future of mobility. We open the door to automotive innovation, revealing the buying patterns of today and helping customers plan for the emerging technologies of tomorrow.
 For more information, visit www.spglobal.com/mobility .
 
What’s In It For You?
 
Our Purpose: 
Progress is not a self-starter. It requires a catalyst to be set in motion. Information, imagination, people, technology–the right combination can unlock possibility and change the world.
 
Our world is in transition and getting more complex by the day. We push past expected observations and seek out new levels of understanding so that we can help companies, governments and individuals make an impact on tomorrow. At S&P Global we transform data into Essential Intelligence®, pinpointing risks and opening possibilities. We Accelerate Progress.
 
Our People: 
We're more than 35,000 strong worldwide—so we're able to understand nuances while having a broad perspective. Our team is driven by curiosity and a shared belief that Essential Intelligence can help build a more prosperous future for us all.
 
From finding new ways to measure sustainability to analyzing energy transition across the supply chain to building workflow solutions that make it easy to tap into insight and apply it. We are changing the way people see things and empowering them to make an impact on the world we live in. We’re committed to a more equitable future and to helping our customers find new, sustainable ways of doing business. We’re constantly seeking new solutions that have progress in mind. Join us and help create the critical insights that truly make a difference.
 
Our Values: 
Integrity, Discovery, Partnership
 
At S&P Global, we focus on Powering Global Markets. Throughout our history, the world's leading organizations have relied on us for the Essential Intelligence they need to make confident decisions about the road ahead. We start with a foundation of integrity in all we do, bring a spirit of discovery to our work, and collaborate in close partnership with each other and our customers to achieve shared goals.
 
Benefits: 
We take care of you, so you can take care of business. We care about our people. That’s why we provide everything you—and your career—need to thrive at S&P Global.
 
Our benefits include: 
Health & Wellness: Health care coverage designed for the mind and body.
 
Flexible Downtime: Generous time off helps keep you energized for your time on.
 
Continuous Learning: Access a wealth of resources to grow your career and learn valuable new skills.
 
Invest in Your Future: Secure your financial future through competitive pay, retirement planning, a continuing education program with a company-matched student loan contribution, and financial wellness programs.
 
Family Friendly Perks: It’s not just about you. S&P Global has perks for your partners and little ones, too, with some best-in class benefits for families.
 
Beyond the Basics: From retail discounts to referral incentive awards—small perks can make a big difference.
 For more information on benefits by country visit: https://spgbenefits.com/benefit-summaries
 
Global Hiring and Opportunity at S&P Global: 
At S&P Global, we are committed to fostering a connected and engaged workplace where all individuals have access to opportunities based on their skills, experience, and contributions. Our hiring practices emphasize fairness, transparency, and merit, ensuring that we attract and retain top talent. By valuing different perspectives and promoting a culture of respect and collaboration, we drive innovation and power global markets.
 
Recruitment Fraud Alert: 
If you receive an email from a spglobalind.com domain or any other regionally based domains, it is a scam and should be reported to reportfraud@spglobal.com . S&P Global never requires any candidate to pay money for job applications, interviews, offer letters, “pre-employment training” or for equipment/delivery of equipment. Stay informed and protect yourself from recruitment fraud by reviewing our guidelines, fraudulent domains, and how to report suspicious activity here .
 
-----------------------------------------------------------
 
Equal Opportunity Employer
 S&P Global is an equal opportunity employer and all qualified candidates will receive consideration for employment without regard to race/ethnicity, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, marital status, military veteran status, unemployment status, or any other status protected by law. Only electronic job submissions will be considered for employment.
 
If you need an accommodation during the application process due to a disability, please send an email to: EEO.Compliance@spglobal.com and your request will be forwarded to the appropriate person.
 
US Candidates Only: The EEO is the Law Poster http://www.dol.gov/ofccp/regs/compliance/posters/pdf/eeopost.pdf describes discrimination protections under federal law. Pay Transparency Nondiscrimination Provision - https://www.dol.gov/sites/dolgov/files/ofccp/pdf/pay-transp_%20English_formattedESQA508c.pdf
 
----------------------------------------------------------- 20 - Professional (EEO-2 Job Categories-United States of America), IFTECH202.1 - Middle Professional Tier I (EEO Job Group), SWP Priority – Ratings - (Strategic Workforce Planning)
 
Job ID: 317941
 Posted On: 2025-10-03
 Location: Bangalore, Karnataka, India

Senior Software Engineer ( UI Backend and Data Pipeline Engineer)

Pavago

Job Title: Data Engineer
 Position Type: Full-Time, Remote
 Working Hours: U.S. client business hours (with flexibility for pipeline monitoring and data refresh cycles)
 
About the Role: 
Our client is seeking a Data Engineer to design, build, and maintain reliable data pipelines and infrastructure that deliver clean, accessible, and actionable data. This role requires strong software engineering fundamentals, experience with modern data stacks, and an eye for quality and scalability. The Data Engineer ensures data flows seamlessly from source systems to warehouses and BI tools, powering decision-making across the business.
 
Responsibilities:Pipeline Development: 
<ul>
<li>Build and maintain ETL/ELT pipelines using Python, SQL, or Scala.</li>
<li>Orchestrate workflows with Airflow, Prefect, Dagster, or Luigi.</li>
<li>Ingest structured and unstructured data from APIs, SaaS platforms, relational databases, and streaming sources.</li>
</ul>
Data Warehousing: 
<ul>
<li>Manage data warehouses (Snowflake, BigQuery, Redshift).</li>
<li>Design schemas (star/snowflake) optimized for analytics.</li>
<li>Implement partitioning, clustering, and query performance tuning.</li>
</ul>
Data Quality & Governance: 
<ul>
<li>Implement validation checks, anomaly detection, and logging for data integrity.</li>
<li>Enforce naming conventions, lineage tracking, and documentation (dbt, Great Expectations).</li>
<li>Maintain compliance with GDPR, HIPAA, or industry-specific regulations.</li>
</ul>
Streaming & Real-Time Data: 
<ul>
<li>Develop and monitor streaming pipelines with Kafka, Kinesis, or Pub/Sub.</li>
<li>Ensure low-latency ingestion for time-sensitive use cases.</li>
</ul>
Collaboration: 
<ul>
<li>Partner with analysts and data scientists to provide curated, reliable datasets.</li>
<li>Support BI teams in building dashboards (Tableau, Looker, Power BI).</li>
<li>Document data models and pipelines for knowledge transfer.</li>
</ul>
Infrastructure & DevOps: 
<ul>
<li>Containerize data services with Docker and orchestrate in Kubernetes.</li>
<li>Automate deployments via CI/CD pipelines (GitHub Actions, Jenkins, GitLab CI).</li>
<li>Manage cloud infrastructure using Terraform or CloudFormation.</li>
</ul>
What Makes You a Perfect Fit: 
<ul>
<li>Passion for clean, reliable, and scalable data.</li>
<li>Strong problem-solving skills with debugging mindset.</li>
<li>Balance of software engineering rigor and data intuition.</li>
<li>Collaborative communicator who thrives in cross-functional environments.</li>
</ul>
Required Experience & Skills (Minimum): 
<ul>
<li>3+ years in data engineering or back-end development.</li>
<li>Strong Python and SQL skills.</li>
<li>Experience with at least one major data warehouse (Snowflake, Redshift, BigQuery).</li>
<li>Familiarity with pipeline orchestration tools (Airflow, Prefect).</li>
</ul>
Ideal Experience & Skills: 
<ul>
<li>Experience with dbt for transformations and data modeling.</li>
<li>Streaming data experience (Kafka, Kinesis, Pub/Sub).</li>
<li>Cloud-native data platforms (AWS Glue, GCP Dataflow, Azure Data Factory).</li>
<li>Background in regulated industries (healthcare, finance) with strict compliance.</li>
</ul>
What Does a Typical Day Look Like?
 A Data Engineer’s day revolves around keeping pipelines running, improving reliability, and enabling teams with high-quality data. You will:
 <ul>
<li>Check pipeline health in Airflow/Prefect and resolve any failed jobs.</li>
<li>Ingest new data sources, writing connectors for APIs or SaaS platforms.</li>
<li>Optimize SQL queries and warehouse performance to reduce costs and latency.</li>
<li>Collaborate with analysts/data scientists to deliver clean datasets for dashboards and models.</li>
<li>Implement validation checks to prevent downstream reporting issues.</li>
<li>Document and monitor pipelines so they’re reproducible, scalable, and audit-ready.</li>
</ul>
In essence: you ensure the business has accurate, timely, and trustworthy data powering every decision.
 
Key Metrics for Success (KPIs): 
<ul>
<li>Pipeline uptime ≥ 99%.</li>
<li>Data freshness within agreed SLAs (hourly, daily, weekly).</li>
<li>Zero critical data quality errors reaching BI/analytics.</li>
<li>Cost-optimized queries and warehouse performance.</li>
<li>Positive feedback from data consumers (analysts, scientists, leadership).</li>
</ul>
Interview Process: 
<ul>
<li>Initial Phone Screen</li>
<li>Video Interview with Pavago Recruiter</li>
<li>Technical Task (e.g., build a small ETL pipeline or optimize a SQL query)</li>
<li>Client Interview with Engineering/Data Team</li>
<li>Offer & Background Verification</li>
</ul>

Data Engineer

Aark Connect

Job Title: Data Engineer with Strong Communication SkillsData Engineer who not only excels in building scalable and efficient data pipelines, but also possesses exceptional communication and presentation skills. This role bridges technical execution with cross-functional collaboration, requiring the ability to explain complex data infrastructure in simple terms to technical and non-technical stakeholders alike.Key Responsibilities<ul><li>Data Pipeline Development: Design, build, and maintain robust ETL/ELT pipelines using modern data engineering tools (e.g., Spark, Airflow, dbt).</li><li>Data Modeling: Develop and optimize data models (star/snowflake schemas) that support analytics and reporting use cases.</li><li>Collaboration &amp; Communication:</li><li>Work closely with data analysts, data scientists, and business teams to understand data needs.</li><li>Deliver clear, engaging presentations of data architecture, pipeline performance, and data quality issues.</li><li>Translate business requirements into scalable engineering solutions and explain trade-offs in plain language.</li><li>Data Quality &amp; Governance: Implement and monitor data quality checks and ensure compliance with data governance policies.</li><li>Documentation: Produce clear technical documentation and present data lineage and infrastructure to non-technical teams.</li></ul>Required Qualifications<ul><li>Bachelor's or Master&rsquo;s degree in Computer Science, Data Engineering, Information Systems, or a related field.</li><li>2+ years of experience in data engineering or a related role.</li><li>Proficiency in SQL, Python, and at least one big data platform (e.g., Hadoop, Spark, Snowflake, Redshift).</li><li>Experience with orchestration tools (e.g., Apache Airflow) and cloud data services (AWS, GCP, or Azure).</li><li>Exceptional verbal and written communication skills.</li><li>Comfortable speaking in team meetings, executive briefings, and client-facing settings.</li></ul>Preferred Qualifications<ul><li>Experience working in a cross-functional, agile team environment.</li><li>Familiarity with data privacy regulations (GDPR, CCPA).</li><li>Previous experience presenting at internal town halls, data literacy sessions, or industry meetups.</li><li>Experience with visualization tools (e.g., Looker, Tableau) is a plus.</li></ul>What Sets You Apart<ul><li>You're as confident writing efficient SQL as you are explaining pipeline latency to a product manager.</li><li>You thrive at the intersection of data infrastructure and human communication.</li><li>You believe that data engineering should be as accessible and transparent as it is powerful.</li></ul>Benefits<ul><li>Competitive salary and stock options</li><li>Flexible work environment</li><li>Health, dental, and vision insurance</li><li>Learning &amp; development budget</li><li>Opportunities to present at internal or external conferences</li></ul>Job Type: Full-timePay: &#8377;1,000,000.00 - &#8377;1,800,000.00 per yearApplication Question(s):<ul><li>can you work onsite?</li><li>can you work night/evening shift?</li><li>immediate joiner??</li></ul>Work Location: In person

Data Engineer (Excellent communication skills)

Alpha Beta Partners Ltd (UK based company)

Role: Data EngineerEducation: BS Computer Science (Preferred)Skills: Required: Python (Pandas, NumPy, Polaris), SQLPreferred: Power Automate, Power Query and SparkExperience: 2 Years (minimum)You are expected to:<ul><li>Automate the flow of data using Power Automate and Data Pipeline in python;</li><li>Have knowledge of data warehousing and data modeling;</li><li>Extract, transform, and load data using SQL, Python and UI based tools;</li><li>Document, monitor, and maintain the work done;</li><li>Collaborate with other IT professionals;</li><li>Bring new ideas to the table to further enhance the management of data.</li></ul>You are a good fit if you are:<ul><li>Seeking a career in Data Engineering;</li><li>Desiring to learn Python, Microsoft Fabric, and other tools mentioned above;</li><li>Willing to drive change;</li><li>Passionate about developing a data culture in an organization;</li><li>Staying abreast with the latest developments in the field;</li><li>Okay with working at night. ( 5pm-2am )</li></ul>Job Type: Full-timeWork Location: In person

Weekday

<div><div>This role is for one of our clients</div><div></div><div> 
Company Name: Muoro.io</div><div>
Industry: Technology, Information and Media</div><div>
Seniority level: Mid-Senior level</div><div></div><div> 
Min Experience: 5 years</div><div>
Location: NCR</div><div>
JobType: full-time</div> 
<div></div>
<div>
We are seeking an experienced Data Engineer with deep expertise in the AWS data ecosystem to design, build, and optimize scalable data pipelines and platforms. The ideal candidate will have strong analytical skills, hands-on experience in modern data architectures, and a passion for delivering high-quality, reliable data solutions.</div><div></div><div> 
Key Responsibilities</div><div>
Design, develop, and maintain robust data pipelines and ETL workflows using AWS tools such as Glue, Kinesis, Redshift, S3, and Data Lake architectures.</div><div>
Work extensively with SQL and Databricks to process, transform, and analyze large datasets.</div><div>
Implement data orchestration and workflow automation using tools like Airflow or Step Functions.</div><div>
Ensure data quality, integrity, and consistency across systems through rigorous validation and monitoring practices.</div><div>
Collaborate with cross-functional teams, including data analysts and software engineers, to support data-driven decision-making.</div><div>
Optimize data storage, retrieval, and processing for scalability and performance in a cloud-native environment.</div><div></div><div> 
Requirements</div><div>
5+ years of hands-on experience in data engineering.</div><div>
Proven expertise in the AWS ecosystem (Glue, Kinesis, Redshift, S3).</div><div>
Proficiency in SQL, Databricks, and data pipeline orchestration tools.</div><div>
Strong understanding of data analysis, data quality management, and data modeling principles.</div><div>
Experience working with data lakes and modern data architectures.</div><div>
Excellent problem-solving, communication, and collaboration skills.</div><div></div><div> 
Core Skills</div><div>
SQL</div><div>
AWS (Glue, Kinesis, Redshift, S3)</div><div>
Databricks</div><div>
Data Pipeline Development</div><div>
Airflow / Step Functions</div><div>
Data Quality Management</div><div>
Data Lake Architecture</div></div>

Data Engineer (AWS)

ixceed

Role: AWS Data EngineerLocation: PuneMode: HybridType: PermanentJob Description:We are seeking a talented and motivated Data Engineer with requisite years of hands-on experience to join our growing data team. The ideal candidate will have experience working with large datasets, building data pipelines, and utilizing AWS public cloud services to support the design, development, and maintenance of scalable data architectures. This is an excellent opportunity for individuals who are passionate about data engineering and cloud technologies and want to make an impact in a dynamic and innovative environment.Key Responsibilities:<ul><li>Data Pipeline Development: Design, develop, and optimize end-to-end data pipelines for extracting, transforming, and loading (ETL) large volumes of data from diverse sources into data warehouses or lakes.</li><li>Cloud Infrastructure Management: Implement and manage data processing and storage solutions in AWS (Amazon Web Services) using services like S3, Redshift, Lambda, Glue, Kinesis, and others.</li><li>Data Modeling: Collaborate with data scientists, analysts, and business stakeholders to define data requirements and design optimal data models for reporting and analysis.</li><li>Performance Tuning &amp; Optimization: Identify bottlenecks and optimize query performance, pipeline processes, and cloud resources to ensure cost-effective and scalable data workflows.</li><li>Automation &amp; Scripting: Develop automated data workflows and scripts to improve operational efficiency using Python, SQL, or other scripting languages.</li><li>Collaboration &amp; Documentation: Work closely with data analysts, data scientists, and other engineering teams to ensure data availability, integrity, and quality. Document processes, architectures, and solutions clearly.</li><li>Data Quality &amp; Governance: Ensure the accuracy, consistency, and completeness of data. Implement and maintain data governance policies to ensure compliance and security standards are met.</li><li>Troubleshooting &amp; Support: Provide ongoing support for data pipelines and troubleshoot issues related to data integration, performance, and system reliability.</li></ul>Qualifications:Essential Skills:<ul><li>Experience: 8+ years of professional experience as a Data Engineer, with a strong background in building and optimizing data pipelines and working with large-scale datasets.</li><li>AWS Experience: Hands-on experience with AWS cloud services, particularly S3, Lambda, Glue, Redshift, RDS, and EC2.</li><li>ETL Processes: Strong understanding of ETL concepts, tools, and frameworks. Experience with data integration, cleansing, and transformation.</li><li>Programming Languages: Proficiency in Python, SQL, and other scripting languages (e.g., Bash, Scala, Java).</li><li>Data Warehousing: Experience with relational and non-relational databases, including data warehousing solutions like AWS Redshift, Snowflake, or similar platforms.</li><li>Data Modeling: Experience in designing data models, schema design, and data architecture for analytical systems.</li><li>Version Control &amp; CI/CD: Familiarity with version control tools (e.g., Git) and CI/CD pipelines.</li><li>Problem-Solving: Strong troubleshooting skills, with an ability to optimize performance and resolve technical issues across the data pipeline.</li></ul>Desirable Skills:<ul><li>Big Data Technologies: Experience with Hadoop, Spark, or other big data technologies.</li><li>Containerization &amp; Orchestration: Knowledge of Docker, Kubernetes, or similar containerization/orchestration technologies.</li><li>Data Security: Experience implementing security best practices in the cloud and managing data privacy requirements.</li><li>Data Streaming: Familiarity with data streaming technologies such as AWS Kinesis or Apache Kafka.</li><li>Business Intelligence Tools: Experience with BI tools (Tableau, Quicksight) for visualization and reporting.</li><li>Agile Methodology: Familiarity with Agile development practices and tools (Jira, Trello, etc.)</li></ul>Job Type: PermanentPay: &#8377;3,000,000.00 - &#8377;3,500,000.00 per yearBenefits:<ul><li>Work from home</li></ul>Experience:<ul><li>AWS: 4 years (Required)</li><li>Data Engineering: 6 years (Required)</li><li>Python: 3 years (Required)</li><li>Pyspark/Spark: 3 years (Required)</li></ul>Work Location: In person

AWS Data Engineer

Search 10000+ Data pipeline engineer jobs near you this October. New roles added regularly across leading industries. Apply today.

10000 data pipeline engineer jobs NEAR_ME

Data Pipeline Engineer

Senior Software Engineer ( UI Backend and Data Pipeline Engineer)

Data Engineer

Data Engineer (Excellent communication skills)

Data Engineer

Data Engineer (AWS)

AWS Data Engineer