data pipeline engineer

Everforce Software Pvt. Ltd.

Job Title: Senior Data Pipeline EngineerLocation: Remote | Chennai, IndiaDuration: 6+ months (extendable based on performance)Experience Required: 8+ yearsWork Timing: Flexible (supporting US &amp; India)Job OverviewWe are seeking an experienced Senior Data Pipeline Engineer with strong expertise in modern data platforms. The ideal candidate will have hands-on experience in building, optimising, and scaling large-scale batch and real-time data pipelines using cutting-edge technologies. This role involves working with cross-functional teams to deliver reliable, high-quality data solutions for enterprise-grade applications.Key Responsibilities- Design, develop, and maintain scalable and reliable data pipelines for both batch and real-time workloads.- Implement real-time event streaming and data ingestion using Apache Kafka.- Build and optimise ETL/ELT workflows on Databricks and integrate with Snowflake for analytics and reporting.- Write efficient, production-ready code in Python, Node.js, and Java.- Collaborate with data scientists, analysts, and business stakeholders to deliver consistent data solutions.- Ensure data quality, governance, and security across platforms.- Monitor, debug, and optimise distributed data systems for high performance.- Lead and mentor junior engineers, promoting best practices in data engineering.Required Qualifications- 8+ years of professional experience in data engineering or software engineering.- Strong expertise in PySpark for distributed data processing.- Hands-on experience with Apache Kafka for data streaming.- Proficiency with Databricks for big data and ML-driven workloads.- Experience with Snowflake for cloud data warehousing.- Advanced programming skills in Python, with proficiency in Node.js and Java.- Solid knowledge of ETL/ELT frameworks, data modelling, and pipelines.- Strong understanding of distributed systems and cloud-based architectures.- Bachelor&rsquo;s or Master&rsquo;s degree in Computer Science, Engineering, or a related field.Preferred Skills- Experience with cloud platforms (AWS, Azure, GCP).- Knowledge of CI/CD pipelines and containerization (Docker, Kubernetes).- Familiarity with MLOps and data science workflows.- Strong system design and debugging skills in large-scale environments.- Excellent communication and problem-solving abilities.What We Offer- Opportunity to design and scale enterprise-grade data infrastructure.- Competitive compensation with career growth opportunities.- Flexible and collaborative work environment.Job Type: Contractual / Temporary Contract length: 6 monthsWork Location: Remote

Senior Data Pipeline Engineer

Encora

<div>Data Engineer II

Important Information

Location: Bangalore, Hyderabad, Pune, Chennai, Ahmadabad, noida

Job Summary
<div>
AWe are seeking a Data Engineer with a strong background in building and optimizing data pipelines. The ideal candidate will have the following skills: 
<ul><li>Data Pipeline Development: Expertise in creating and maintaining a data pipeline to move data from MongoDB to Iceberg.</li><li>AWS Glue: Hands-on experience with AWS Glue for building ETL jobs.</li><li>Data Transformation: Proficient in data transformation</li><li>Iceberg: Good working knowledge of the Iceberg table format, including:</li></ul>
o Troubleshooting performance and data integrity issues. 
o Managing catalogs and schema evolution. 
o Implementing upsert functionality for slowly changing dimensions. 
<ul><li>Snowflake: Good experience in working with query build &amp; optimization</li></ul></div>

Experience: 5+ Years

About Encora

Encora is a global company that offers Software and Digital Engineering solutions. Our practices include Cloud Services, Product Engineering &amp; Application Modernization, Data &amp; Analytics, Digital Experience &amp; Design Services, DevSecOps, Cybersecurity, Quality Engineering, AI &amp; LLM Engineering, among others.

At Encora, we hire professionals based solely on their skills and do not discriminate based on age, disability, religion, gender, sexual orientation, socioeconomic status, or nationality.</div>

Data Engineer

Thermo Fisher Scientific

<div>Work Schedule
Standard (Mon-Fri)
Environmental Conditions
Office
Job Description
Job Summary:
We are seeking a skilled and detail-oriented Data Engineer to join our data team. The ideal candidate will be responsible for building and maintaining scalable data pipelines, extracting data from diverse sources including APIs, databases, and flat files, and ensuring high data quality and reliability. You will work closely with analysts, data scientists, and engineers to power data-driven decision-making across the organization.
Key Responsibilities:<ul><li>
Design, develop, and maintain scalable and robust data pipelines for both batch and real-time processing.</li><li>
Extract, transform, and load (ETL) data from a wide variety of structured and unstructured data sources including:<ul><li>
RESTful and SOAP APIs</li><li>
Databases (SQL, NoSQL)</li><li>
Cloud storage (e.g., S3, Google Cloud Storage)</li><li>
File formats (e.g., JSON, CSV, XML, Parquet)</li><li>
Web scraping tools where appropriate</li></ul></li><li>
Build reusable data connectors and integration solutions to automate data ingestion.</li><li>
Collaborate with internal stakeholders to understand data requirements and ensure accessibility and usability.</li><li>
Monitor and optimize pipeline performance and troubleshoot data flow issues.</li><li>
Ensure data governance, security, and quality standards are applied across all pipelines.</li><li>
Experience with data manipulation and analysis libraries such as Pandas, Polars, or Dask for handling large datasets efficiently.</li><li>
Design and create data flow and architecture diagrams to visually represent data pipelines, system integrations, and data models, ensuring clarity and alignment among technical and non-technical stakeholders.</li></ul>
Requirements:
Technical Skills:<ul><li>
Proficiency in SQL and at least one programming language (Python, Java, Scala).</li><li>
Experience with data pipeline and workflow tools (e.g., Apache Airflow, AWS Data Pipeline).</li><li>
Knowledge of relational and non-relational databases. (e.g., Oracle, SqlServer, MongoDB).</li><li>
Strong data modeling and data warehousing skills.</li></ul>
Education &amp; Experience:<ul><li>
Bachelor&rsquo;s degree in Computer Science, Engineering, Information Systems, or related field (Master&rsquo;s a plus).</li><li>
5+ years of experience in a data engineering or similar role.</li></ul>
Soft Skills:<ul><li>
Strong analytical and problem-solving abilities.</li><li>
Excellent communication and collaboration skills.</li><li>
Detail-oriented and proactive mindset.</li></ul></div>

Apex One

Position: Data Engineer / Big Data EngineerExperience:10Yrs Location: Pune &ndash; HybridJD (Short):<ul><li>Strong in SQL, Python, Spark, Snowflake, Databricks, AWS.</li><li>Hands-on with ETL/ELT tools (DBT, Airflow, StreamSets) &amp; data pipeline design.</li><li>Expertise in data modeling, warehousing, performance tuning &amp; security (GDPR/HIPAA).</li><li>Skilled in relational &amp; NoSQL databases (MS SQL, MySQL) with advanced query optimization.</li><li>Proven track record in delivering scalable data solutions &amp; actionable insights.</li></ul>Job Types: Full-time, PermanentPay: Up to &#8377;1,681,411.40 per yearWork Location: In person

Senior Data Engineer

Amerisure

<div><div>Amerisure creates exceptional value for its partners, policyholders, and employees. As a property and casualty insurance company, Amerisure&rsquo;s promise to our partner agencies and policyholders begins with a comprehensive line of insurance products designed to protect businesses, as well as the health and safety of every employee. With an A.M. Best &ldquo;A&rdquo; (Excellent) rating, Amerisure serves mid-sized commercial enterprises focused in construction, manufacturing and healthcare. Ranked as one of the top 100 Property &amp; Casualty companies in the United States, we proudly manage nearly $1 Billion of Direct Written Premium and maintain $1.21 billion in surplus.</div><div></div><div>
Amerisure is currently recruiting for a Data Engineer II that can do a 2-day hybrid approach onsite in our Farmington Hills office. The ideal candidate will possess the following skill set.</div><div></div><div>
Summary Statement</div><div>
The Data Engineer II will drive designing, building, and optimizing advanced data pipelines and models, ensuring their scalability and performance. This role includes managing and supporting complex production environments, identifying opportunities for system enhancements, and mentoring junior team members. The Data Engineer II will work with tools such as Databricks, AzureGit, and others, driving innovation and efficiency in a collaborative, Agile environment.</div><div></div><div>
Essential Tasks/Major Duties</div><ul><li>
Create and maintain scalable, efficient, and high-performing data pipelines to meet business needs.</li><li>
Analyze complex datasets to uncover data quality issues, patterns, and potential improvements.</li><li>
Develop and maintain robust data models that align with business and technical requirements.</li><li>
Implement monitoring, logging, and alerting systems to ensure pipeline reliability and identify performance bottlenecks.</li><li>
Ensure the metadata repository is accurate and supports data governance and traceability efforts.</li><li>
Manage the deployment and testing of pipeline updates and improvements using CI/CD practices.</li><li>
Engage in Agile ceremonies, such as stand-ups, sprint planning, and retrospectives, to support iterative development.</li><li>
Resolve operational issues for data pipelines and participate in an on-call rotation to maintain system reliability.</li></ul><div></div><div>
Knowledge, Skills &amp; Abilities</div><ul><li>
Bachelor&rsquo;s degree in computer science, Data Engineering, or a related field (or equivalent experience).</li><li>
2 years of data engineering experience delivering high-quality solutions.</li><li>
Proficiency with tools like Databricks, AzureGit, AWS, Jira, ER/Studio, and VS Code.</li><li>
2 years in data pipeline development, modeling, and transformation.</li><li>
3 years of SQL, at least one programming language (e.g., Python), and big data technologies (e.g., Spark, Hadoop).</li><li>
1 year experience with cloud platforms (Azure, AWS), CI/CD processes, DevOps practices, and real-time data streaming.</li><li>
Experience with data governance frameworks, metadata management, and pipeline automation.</li><li>
Data querying, filtering, and joining using performant languages like SQL or Python.</li><li>
Application of data quality concepts, including master and reference data for survivorship.</li><li>
Data security principles, such as classification, roles, and privileges across data stacks.</li><li>
Data ingestion and storage design, optimizing pipelines, file structures, and partitioning for performance and cost efficiency.</li><li>
Data modeling, including creating data dictionaries and conceptual models.</li><li>
Strong understanding of Agile and DevOps methodologies.</li><li>
Ability to mentor junior colleagues, collaborate effectively in teams, and foster a positive work environment.</li><li>
Skilled in capturing client needs and delivering value through Agile practices.</li><li>
Strong analytical and problem-solving skills, with the ability to manage complex tasks independently.</li><li>
Excellent communication skills to convey technical concepts to non-technical stakeholders.</li><li>
Familiarity with insurance terminology and practices, applied to data organization and communication, preferred.</li><li>
Experience supporting and troubleshooting platform systems and environments.</li></ul><div>
#LI-BR1</div><div></div><div>
Just as we are committed to creating exceptional value for our Partners For Success&reg; agencies and policyholders, Amerisure also remains committed to being an employer of choice. We reinforce this commitment by adhering to an Employee Value Proposition that, in part, is provided through a competitive total rewards package. This package includes competitive base pay, performance-based incentive pay, comprehensive health and welfare benefits, a 401(k) savings plan with profit sharing, and generous paid time off programs. We also offer flexible work arrangements to promote work-life balance. Recognized as one of the Best and Brightest&reg; Companies to Work For in the Nation and one of Business Insurance magazine&rsquo;s Best Places to Work in Insurance, we provide a workplace that fosters excellence and professional growth. If you are looking for a collaborative and rewarding career, Amerisure is looking for you.</div><div></div><div>
Amerisure Insurance provides equal employment opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, sex, national origin, age, disability or genetics. In addition to federal law requirements, Amerisure Insurance complies with applicable state and local laws governing nondiscrimination in employment in every location in which the company has facilities. This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation and training. Amerisure Insurance expressly prohibits any form of workplace harassment based on race, color, religion, gender, sexual orientation, gender identity or expression, national origin, age, genetic information, disability, or veteran status. Improper interference with the ability of Amerisure&rsquo;s employees to perform their job duties may result in discipline up to and including discharge.</div></div>

Data Engineer II

ixceed

Role: AWS Data EngineerLocation: PuneMode: HybridType: PermanentJob Description:We are seeking a talented and motivated Data Engineer with requisite years of hands-on experience to join our growing data team. The ideal candidate will have experience working with large datasets, building data pipelines, and utilizing AWS public cloud services to support the design, development, and maintenance of scalable data architectures. This is an excellent opportunity for individuals who are passionate about data engineering and cloud technologies and want to make an impact in a dynamic and innovative environment.Key Responsibilities:<ul><li>Data Pipeline Development: Design, develop, and optimize end-to-end data pipelines for extracting, transforming, and loading (ETL) large volumes of data from diverse sources into data warehouses or lakes.</li><li>Cloud Infrastructure Management: Implement and manage data processing and storage solutions in AWS (Amazon Web Services) using services like S3, Redshift, Lambda, Glue, Kinesis, and others.</li><li>Data Modeling: Collaborate with data scientists, analysts, and business stakeholders to define data requirements and design optimal data models for reporting and analysis.</li><li>Performance Tuning &amp; Optimization: Identify bottlenecks and optimize query performance, pipeline processes, and cloud resources to ensure cost-effective and scalable data workflows.</li><li>Automation &amp; Scripting: Develop automated data workflows and scripts to improve operational efficiency using Python, SQL, or other scripting languages.</li><li>Collaboration &amp; Documentation: Work closely with data analysts, data scientists, and other engineering teams to ensure data availability, integrity, and quality. Document processes, architectures, and solutions clearly.</li><li>Data Quality &amp; Governance: Ensure the accuracy, consistency, and completeness of data. Implement and maintain data governance policies to ensure compliance and security standards are met.</li><li>Troubleshooting &amp; Support: Provide ongoing support for data pipelines and troubleshoot issues related to data integration, performance, and system reliability.</li></ul>Qualifications:Essential Skills:<ul><li>Experience: 8+ years of professional experience as a Data Engineer, with a strong background in building and optimizing data pipelines and working with large-scale datasets.</li><li>AWS Experience: Hands-on experience with AWS cloud services, particularly S3, Lambda, Glue, Redshift, RDS, and EC2.</li><li>ETL Processes: Strong understanding of ETL concepts, tools, and frameworks. Experience with data integration, cleansing, and transformation.</li><li>Programming Languages: Proficiency in Python, SQL, and other scripting languages (e.g., Bash, Scala, Java).</li><li>Data Warehousing: Experience with relational and non-relational databases, including data warehousing solutions like AWS Redshift, Snowflake, or similar platforms.</li><li>Data Modeling: Experience in designing data models, schema design, and data architecture for analytical systems.</li><li>Version Control &amp; CI/CD: Familiarity with version control tools (e.g., Git) and CI/CD pipelines.</li><li>Problem-Solving: Strong troubleshooting skills, with an ability to optimize performance and resolve technical issues across the data pipeline.</li></ul>Desirable Skills:<ul><li>Big Data Technologies: Experience with Hadoop, Spark, or other big data technologies.</li><li>Containerization &amp; Orchestration: Knowledge of Docker, Kubernetes, or similar containerization/orchestration technologies.</li><li>Data Security: Experience implementing security best practices in the cloud and managing data privacy requirements.</li><li>Data Streaming: Familiarity with data streaming technologies such as AWS Kinesis or Apache Kafka.</li><li>Business Intelligence Tools: Experience with BI tools (Tableau, Quicksight) for visualization and reporting.</li><li>Agile Methodology: Familiarity with Agile development practices and tools (Jira, Trello, etc.)</li></ul>Job Type: PermanentPay: &#8377;3,000,000.00 - &#8377;3,800,000.00 per yearBenefits:<ul><li>Work from home</li></ul>Experience:<ul><li>Data Engineering: 6 years (Required)</li><li>Python: 3 years (Required)</li><li>Pyspark/Spark: 3 years (Required)</li><li>AWS: 5 years (Required)</li></ul>Work Location: In person

Data Engineer (AWS)

Buzzworks Business Services Pvt Ltd

<ul><li>Job Title : Data Engineer</li><li>Job Type :Contract To Hire (9 Months) After 9 months payroll fromClientSide</li><li>Yrs of Experience : 4 - 14 yrs</li><li>Notice Period : Immediate - 30 Days</li><li>Interview Date : 22/08/25 (Friday)</li><li>Primary Skill Set : GCP, Big Query, Data Pipeline Development, SQL, Python/ Java </li></ul>About the RoleWe are seeking a highly skilled Data Engineer to design, develop, and optimize scalable data pipelines and solutions on Google Cloud Platform (GCP). The ideal candidate will have strong expertise in BigQuery, SQL, Python/Java, and hands-on experience in building robust data pipelines to support advanced analytics and business intelligence.Key Responsibilities<ul><li>Design, develop, and maintain scalable data pipelines and ETL/ELT processes.</li><li>Work extensively on GCP services including BigQuery, Cloud Storage, Dataflow, Pub/Sub, and Composer.</li><li>Optimize data models and queries for performance and cost efficiency in BigQuery.</li><li>Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements.</li><li>Ensure data quality, governance, and security across all data pipelines and storage systems.</li><li>Troubleshoot and resolve data pipeline issues in real time.</li><li>Contribute to architecture design discussions and provide best practices for data engineering.</li></ul>Required Skills<ul><li>Strong hands-on experience with Google Cloud Platform (GCP), particularly BigQuery.</li><li>Proven expertise in building and maintaining data pipelines.</li><li>Strong SQL skills for query optimization and large-scale data manipulation.</li><li>Proficiency in Python or Java for developing scalable ETL/ELT solutions.</li><li>Good understanding of data modeling, partitioning, and performance tuning.</li><li>Experience with workflow orchestration tools (e.g., Airflow/Cloud Composer) is a plus.</li><li>Familiarity with CI/CD, version control (Git), and agile methodologies.</li></ul>Good to Have<ul><li>Exposure to streaming data pipelines (Pub/Sub, Kafka).</li><li>Knowledge of machine learning pipelines integration.</li><li>Experience with other cloud platforms (AWS/Azure).</li></ul>Soft Skills<ul><li>Strong analytical and problem-solving skills.</li><li>Excellent communication and collaboration abilities.</li><li>Ability to work independently and as part of a team in a fast-paced environment.</li></ul>Job Type: Contractual / TemporaryPay: &#8377;500,000.00 - &#8377;4,000,000.00 per yearBenefits:<ul><li>Food provided</li><li>Health insurance</li><li>Provident Fund</li></ul>Work Location: In person

GCP Data Engineer/ Senior data engineer

Search 10000+ Data pipeline engineer jobs near you this September. New roles added regularly across leading industries. Apply today.

10000 data pipeline engineer jobs NEAR_ME

Senior Data Pipeline Engineer

Data Engineer

Data Engineer

Senior Data Engineer

Data Engineer II

Data Engineer (AWS)

GCP Data Engineer/ Senior data engineer