data pipeline engineer

Caregenix Solutions

We're Hiring: ETL Engineer (Remote &ndash; Full Time)Company: CareGenix SolutionsLocation: RemoteExperience: 3+ YearsType: Full-TimeCaregenix Solutions, a fast-growing healthcare tech company, is seeking a skilled ETL Engineer to design and maintain robust data pipelines that power real-time insights and analytics.Key Responsibilities:Build and manage scalable ETL/ELT pipelines using AWS services (Glue, Lambda, S3, EMR, Redshift, etc.).Design and optimize data models and databases (Redshift, Athena, RDS).Ingest data from diverse sources and ensure quality through validation checks.Collaborate with teams to deliver reliable, insight-driven data solutions.Monitor pipeline performance and troubleshoot as needed.Follow best practices for data security and governance.Requirements6+ years in Data Engineering or similar roles.Strong in SQL and Python or Scala.Hands-on experience with AWS data tools: Glue, Redshift, EMR, Athena, Kinesis, etc.Experience with workflow/orchestration tools (Airflow, Step Functions).Solid knowledge of data warehousing, performance tuning, and big data.Familiarity with CI/CD, Git, and IaC tools (Terraform, CloudFormation).Bachelor&rsquo;s in Computer Science, Engineering, or related field.To Apply Send your resume to careers@caregenix.solutionsSubject Line: ETL EngineerBe part of a team transforming healthcare through data. We&rsquo;d love to hear from youJob Type: Full-timePay: Up to Rs400,000.00 per monthApplication Question(s):<ul><li>what is your expected pay?</li></ul>Experience:<ul><li>Data/ ETL : 2 years (Required)</li></ul>Work Location: Remote

Data Engineer

MOBIZ

<div>About Mobiz

Mobiz is an information technology firm, and we do three things exceptionally well: cloud and network automation, digital transformation, and data protection. Double-digit growth year-over-year has driven us to grow our team and ensure we stay ahead of the curve. 
What Can You Expect?

Every day at Mobiz we work with a deep sense of purpose. We continuously innovate. Our mission is to empower our clients to do more through transformation. You&rsquo;ll work in a collaborative environment alongside highly talented people that improve client operations and exceed expectations. We strive to simplify technology challenges, and no less. 


Who Are We Looking For?

We are looking for a skilled and motivated Data Engineer to join our growing data team. In this role, you will be responsible for designing, building, and operating scalable, secure, and high-performing data solutions that enable analytics, reporting, and data-driven decision-making across the organization. You will collaborate closely with cross-functional teams and leverage modern cloud technologies to transform raw data into trusted, accessible insights. 


Key Responsibilities
<ul>
<li>Design, build, test, deploy, and maintain robust data pipelines using Databricks, Azure Data Factory, and Microsoft Fabric, supporting both batch and streaming workloads.</li>
<li>Ingest, integrate, and harmonize data from multiple cloud and on-premises sources to deliver a unified, reliable, and trusted data view for analytics and reporting.</li>
<li>Use Python and SQL to clean, transform, validate, and model data to meet business, reporting, and analytical requirements.</li>
<li>Contribute to data architecture and platform design decisions, ensuring solutions are scalable, secure, cost-effective, and aligned with long-term business and technical strategy.</li>
<li>Build and maintain metadata-driven pipelines that gracefully adapt to schema changes, new data sources, and evolving ingestion patterns.</li>
<li>Proactively identify data quality issues, pipeline failures, and performance bottlenecks; perform root-cause analysis and implement durable, long-term solutions.</li>
<li>Continuously optimize data pipeline performance, reliability, and scalability to ensure fast, consistent, and dependable access to data.</li>
<li>Implement automated data quality checks, validation rules, and monitoring dashboards to ensure accuracy, consistency, and trust in delivered datasets.</li>
<li>Monitor, analyze, and manage data pipeline and compute costs across Azure, Databricks, and Microsoft Fabric, recommending and implementing cost-optimization strategies without compromising performance or availability.</li>
<li>Automate routine and repetitive data engineering tasks, applying best practices for CI/CD, deployment, testing, and operational support.</li>
<li>Design, implement, and maintain secure data access controls using role-based access, audits, and governance best practices to ensure appropriate data security and compliance.</li>
<li>Maintain clear, accurate, and up-to-date technical documentation, and provide ongoing operational support as data platforms and business needs evolve.</li>
<li>Stay current with emerging data engineering technologies, cloud services, and analytics best practices, and evaluate new tools through proofs of concept when appropriate.</li>
<li>Collaborate closely with product managers, software engineers, analysts, and operations teams to embed high-quality data solutions into core business processes.</li>
<li>Promote and enforce data engineering best practices including version control, modular and reusable pipeline design, documentation standards, and scalable architecture patterns.</li>
<li>Enable and support data democratization by providing well-modeled, secure, and self-service-ready datasets through Power BI, Microsoft Fabric, or custom analytics solutions.</li>
<li>Deliver measurable outcomes such as highly reliable pipelines with minimal manual intervention, on-time pipeline execution within service windows, rapid resolution of critical incidents, and consistently high data quality.</li>
<li>Ensure reusable and standardized pipeline components are leveraged across teams to accelerate delivery, improve consistency, and reduce operational overhead.</li>
<li>Provide stakeholders with reliable, well-modeled datasets that can be confidently explored without requiring ongoing engineering support.</li>
<li>Ensure all data solutions comply with organizational security, access, and governance policies while maintaining positive feedback from internal and external users.</li></ul> 


 Required Skills &amp; Qualifications 
<ul>
<li>3&ndash;5 years of hands-on experience in data engineering or related roles, with proven experience building, supporting, and operating production-grade data pipelines.</li>
<li>Strong hands-on experience with Databricks, including development, optimization, and operational support of Databricks-based data solutions.</li>
<li>Proven experience with Azure Data Factory (ADF) and Microsoft Fabric for data ingestion, orchestration, and transformation.</li>
<li>Advanced proficiency in SQL for querying, data modeling, performance tuning, and troubleshooting.</li>
<li>Strong programming skills in Python for data transformation, automation, scripting, and pipeline development.</li>
<li>Hands-on experience with Microsoft Azure cloud services related to data storage, compute, networking, and security.</li>
<li>Solid understanding of ETL/ELT patterns, data integration strategies, and real-time or near-real-time data processing concepts.</li>
<li>Experience designing metadata-driven and reusable pipeline components that scale across multiple teams and projects.</li>
<li>Strong understanding of data governance, data quality frameworks, monitoring practices, and automated validation techniques.</li>
<li>Experience supporting high-availability data platforms with SLAs, monitoring, alerting, and incident response processes.</li>
<li>Demonstrated ability to optimize cloud compute usage and manage costs efficiently across Azure, Databricks, and Microsoft Fabric.</li>
<li>Experience using version control systems such as Git and following modern software engineering best practices.</li>
<li>Ability to design secure, scalable, and cost-effective data architectures aligned with business needs.</li>
<li>Strong analytical, problem-solving, and troubleshooting skills with a proactive and ownership-driven mindset.</li>
<li>Excellent communication skills, with the ability to explain complex data concepts to both technical and non-technical stakeholders.</li>
<li>Ability to work independently as well as collaboratively in cross-functional team environments.</li>
<li>High attention to detail with a strong focus on data accuracy, reliability, and documentation quality.</li>
<li>Bachelor&rsquo;s degree in Computer Science, Engineering, or a related field.</li>
<li>Relevant certifications are required or strongly preferred, including Databricks Certified Data Engineer, Microsoft Certified: Azure Data Engineer Associate, and Microsoft Certified: Fabric Data Engineer Associate.</li></ul> 


What We Offer
<ul>
<li>A team of bright, hard-working, and innovative people that will contribute to your growth.</li>
<li>Competitive Salary and comprehensive benefits plan.</li>
<li>A dynamic and collaborative work environment with opportunity to work with cutting-edge technology and innovative solutions.</li></ul> 


Other 
This is a full-time office based job in Islamabad, Pakistan. 
Closing

Mobiz embraces diversity and is committed to building a team that represents a variety of backgrounds, perspectives, and skills. The more inclusive we are, the better we are. We are an equal opportunity employer.

We thank all applicants for their interest. We will contact you if you are selected to move forward in the recruiting process.</div>

Match Made Tech

<div><div>Job Title: Data Engineer
LOCATION: Irvine, CA (onsite). Monday through Thursday onsite, Fridays remote.
COMPENSATION: $50-95 an hour. This is a 2-year contract that will convert to full-time.
ABOUT US: We are on a mission to develop innovative AI solutions that will revolutionize our workforce. As we embark on an exciting new greenfield AI project, our data team is forward-thinking and data-driven dedicated to transforming data into actionable insights. We are seeking a skilled Data Engineer to join our team, responsible for leveraging Python, Java, PostgreSQL, Snowflake and AWS to design, develop, and maintain data pipelines and systems that support our analytics and decision-making processes.
JOB DESCRIPTION:
Position Overview: As a Data Engineer, you will play a pivotal role in the development and management of our data infrastructure. You will work closely with cross-functional teams to ensure data availability, reliability, and quality, enabling data-driven decision-making across the organization.
RESPONSIBILITIES:
Data Pipeline Development:
<ul><li>Design, implement, and maintain data pipelines using Python, PostgreSQL, and AWS technologies.
</li><li>Develop efficient and scalable ETL processes to extract, transform, and load data from various sources into our Snowflake data warehouses.
</li></ul>Database Management:
<ul><li>Design and maintain PostgreSQL databases, including schema design, indexing, and performance optimization.
</li><li>Monitor database performance, troubleshoot issues, and implement improvements as needed.
</li><li>AWS Integration:
</li><li>Utilize AWS services, such as AWS Glue, S3, Redshift, and EMR, for data storage, processing, and analytics.
</li><li>Collaborate with AWS solutions architects to optimize data-related AWS infrastructure.
</li></ul>Data Quality and Governance:
<ul><li>Implement data quality checks and validation processes to ensure data accuracy and consistency.
</li><li>Enforce data governance policies and security measures to protect sensitive data.
</li></ul>Documentation and Collaboration:
<ul><li>Maintain comprehensive documentation of data pipelines, database schemas, and ETL processes.
</li><li>Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver actionable insights.
</li></ul>Performance Optimization:
<ul><li>Continuously monitor and optimize data pipelines and database performance for scalability and efficiency.
</li></ul>QUALIFICATIONS:
<ul><li>Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.
</li><li>Proven experience (3+ years) as a data engineer with expertise in Python, Java, PostgreSQL, and AWS.
</li><li>Strong SQL skills and proficiency in data modeling and database design.
</li><li>Hands-on experience with AWS services, particularly Glue, S3, Redshift, and EMR.
</li><li>Knowledge of ETL processes and data integration techniques.
</li><li>Excellent problem-solving skills and attention to detail.
</li><li>Effective communication and collaboration skills to work within a team.
</li><li>A commitment to staying current with industry trends and best practices in data engineering.
</li></ul>WHAT MAKES THIS EXCITING:
<ul><li>Be part of a dynamic team in a data-driven organization.
</li><li>Technical ownership on greenfield projects
</li><li>Contribute to projects that have a meaningful impact on our business and industry.
</li><li>Competitive compensation and opportunities for professional growth.
</li><li>Work in a collaborative and innovative environment.
</li></ul>If you are a skilled Data Engineer with expertise in Python, Java, PostgreSQL, and AWS and are passionate about turning data into insights, we encourage you to apply. Join us in our mission to harness the power of data!</div></div>

Data Engineer (Hybrid)

Johns Hopkins University

<div>The Pathology Department is seeking a Data Engineer I to assist with the creation and maintenance of complex data pipelines from raw acquisition of data to visualization. The Data Engineer I will support the design, production, and maintenance of software infrastructure to automatically extract, transform, and load data from diverse sources. The Data Engineer I will help input/output data from databases and realize complex queries. The Data Engineer I will support the creation of scripts for data manipulation, cleaning, filtering, and preparation of data outputs to be used for data visualization or web display. This will also include creating parts of systems to monitor and guarantee data quality, perform daily data quality assurance tasks, maintain, and troubleshoot the infrastructure, and audit data sources. The Data Engineer I will help explore and analyze data to answer policy-related questions as needed. 
Specific Duties &amp; Responsibilities<ul><li>
Assist with the design, production, and maintenance of data pipelines for data acquisition, management, and transformation, and back-end code development to power data web applications and convert raw data into usable information.</li><li>
Contribute to writing and maintenance of ETL/ELTs that operate on a variety of structured and unstructured sources.</li><li>
Assist with the development and maintenance of web data scraping systems for automatic data acquisition.</li><li>
Assist design data architecture and provide ongoing support.</li><li>
Assist with input/output data from databases and perform queries.</li><li>
Assist in the creation of scripts to clean, transform, and analyze data.</li><li>
Contribute to production data pipelines using data warehousing systems.</li><li>
Create and implement production software to monitor data quality and detect data anomalies.</li><li>
Perform daily manual data quality assurance tasks.</li><li>
Support, maintain, and troubleshoot the software infrastructure.</li><li>
Source data, conduct analyses, visualize data, and generate insights to support ongoing research projects and other requests across the organization.</li><li>
Assist developers, analysts, data scientists, researchers, policy experts, and other partners.</li><li>
Communicate with division leadership, and others on the team.</li><li>
Collaborate with external partners, contractors, and vendors.</li><li>
Other duties as assigned.</li></ul> 

Minimum Qualifications 
<ul><li>
Bachelor&rsquo;s Degree.</li><li>
Three years of related experience.</li><li>
Additional education may substitute for required experience, and additional related experience may substitute for required education beyond a high school diploma/graduation equivalent, to the extent permitted by the JHU equivalency formula.</li></ul> 

Preferred Qualifications 
Technical Skills &amp; Expected Level of Proficiency<ul><li>
Data Pipeline Architecture &amp; ETL/ELT Development - Intermediate</li><li>
Database Querying - Intermediate</li><li>
Data Tools and Platforms - Intermediate</li><li>
Data Validation and Quality Assurance - Intermediate</li><li>
Data Warehousing &amp; Architecture - Intermediate</li><li>
Project Management - Intermediate</li><li>
Programming Languages - Intermediate</li><li>
Troubleshooting - Intermediate</li><li>
Web Scraping and Data Acquisition - Intermediate</li></ul> 
The core technical skills listed are most essential; additional technical skills may be required based on specific division or department needs. 
Classified Title: Data Engineer I 
Job Posting Title (Working Title): Data Engineer I- Pathology
 
Role/Level/Range: ATP/04/PF
 
Starting Salary Range: $85,500 - $149,800 Annually ($87,000 targeted; Commensurate w/exp.) 
Employee group: Full Time 
Schedule: Monday-Friday 8:30am-5:00pm 
FLSA Status: Exempt 
Location: School of Medicine Campus 
Department name: SOM Pat Molecular Diagnostics Lab 
Personnel area: School of Medicine</div>

Data Engineer I- Pathology

GSSTech Group

<div>We are seeking a highly skilled Data Engineer with strong expertise in PySpark and the Cloudera Data Platform (CDP). The ideal candidate will design, develop, and maintain scalable data pipelines while ensuring high data quality, performance, and availability across the organisation.
This role requires hands-on experience in big data ecosystems, cloud-native technologies, and advanced data processing frameworks. You will collaborate with cross-functional teams to build reliable and high-performance data solutions that drive business insights.<h3 class="jobSectionHeader">
Key Responsibilities</h3><h3 class="jobSectionHeader">
1. Data Pipeline Development</h3><ul>
<li>Design, develop, and maintain scalable ETL/ELT pipelines using PySpark on CDP
</li><li>Ensure data integrity, reliability, and performance optimisation
</li></ul><h3 class="jobSectionHeader">2. Data Ingestion</h3><ul>
<li>Develop ingestion frameworks to collect data from relational databases, APIs, streaming sources, and file systems
</li><li>Load structured and unstructured data into Data Lake/Data Warehouse environments
</li></ul><h3 class="jobSectionHeader">3. Data Transformation &amp; Processing</h3><ul>
<li>Process, cleanse, and transform large-scale datasets using PySpark
</li><li>Build reusable data processing components
</li></ul><h3 class="jobSectionHeader">4. Performance Optimisation</h3><ul>
<li>Tune Spark jobs and Cloudera components for optimal performance
</li><li>Optimise memory, partitioning, and execution plans
</li><li>Reduce ETL runtime and improve cluster efficiency
</li></ul><h3 class="jobSectionHeader">5. Data Quality &amp; Validation</h3><ul>
<li>Implement data validation checks and monitoring mechanisms
</li><li>Ensure end-to-end data quality and governance standards
</li></ul><h3 class="jobSectionHeader">6. Automation &amp; Orchestration</h3><ul>
<li>Automate workflows using tools such as Apache Oozie, Apache Airflow, or similar orchestration frameworks
</li><li>Maintain CI/CD integration for data pipelines
</li></ul><h3 class="jobSectionHeader">7. Monitoring &amp; Support</h3><ul>
<li>Monitor pipeline health and troubleshoot failures
</li><li>Provide production support and continuous improvements
</li></ul><h3 class="jobSectionHeader">Required Skills &amp; Qualifications</h3><ul>
<li>5+ years of experience in Data Engineering
</li><li>Strong hands-on experience in PySpark
</li><li>Experience working on Cloudera Data Platform (CDP)
</li><li>Strong knowledge of Hadoop ecosystem (HDFS, Hive, Impala, YARN)
</li><li>Proficiency in SQL and data modelling concepts
</li><li>Experience with workflow orchestration tools (Airflow, Oozie, etc.)
</li><li>Good understanding of data warehousing concepts
</li><li>Experience with performance tuning and optimisation
</li></ul><h3 class="jobSectionHeader">Good to Have</h3><ul>
<li>Experience with cloud platforms (AWS, Azure, GCP)
</li><li>Knowledge of streaming tools (Kafka, Spark Streaming)
</li><li>Exposure to DevOps practices and CI/CD pipelines
</li><li>Banking/Financial Services domain experience</li></ul></div>

Data Engineer (PySpark)

Accenture

Project Role : Data Engineer 
Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems.
 
Must have skills : Data Analysis &amp; Interpretation 
Good to have skills : Oracle Procedural Language Extensions to SQL (PLSQL), Data Analytics, Tableau 
Minimum 3 year(s) of experience is required 
Educational Qualification : 15 years full time education 
 
Summary: As a Data Engineer, you will engage in the design, development, and maintenance of data solutions that facilitate data generation, collection, and processing. Your typical day will involve creating data pipelines, ensuring data quality, and implementing ETL processes to effectively migrate and deploy data across various systems. You will collaborate with team members to address challenges and enhance data workflows, contributing to the overall efficiency and effectiveness of data management within the organization. Roles &amp; Responsibilities: - Expected to perform independently and become an SME. - Required active participation/contribution in team discussions. - Contribute in providing solutions to work related problems. - Assist in the optimization of data processing workflows to enhance performance. - Collaborate with cross-functional teams to identify data needs and develop appropriate solutions. Professional &amp; Technical Skills: - Must To Have Skills: Proficiency in Data Analysis &amp; Interpretation, Data Analytics, Tableau, Oracle Procedural Language Extensions to SQL (PLSQL), Power BI and Excel. - Good To Have Skills: Experience with Data Analytics, Tableau, Oracle Procedural Language Extensions to SQL (PLSQL). - Good knowledge of the Data Modelling, Data Controls &amp; Assurance. - Have experience in Python for Data Manipulation &amp; Automation - Have experience with ETL/ELT and Data Pipeline Orchestration - Have understanding with Machine Learning Concepts - Strong understanding of data modeling and database design principles. Additional Information: - The candidate should have minimum 3 years of experience in Data Analysis &amp; Interpretation. - This position is based at our Gurugram office. - A 15 years full time education is required.
 
15 years full time education

ibex

<div>
 We are seeking a highly motivated and responsible
 
 Data Engineer
 
 to join our dynamic team. This role requires hands-on experience in building scalable data pipelines, managing structured and unstructured databases, and supporting end-to-end data operations. The ideal candidate will bring strong problem-solving skills, a collaborative mindset, and a commitment to data excellence.
 
 
 
 Responsibilities
 
 
 
 <ul>
 <li>
 Design, develop, and manage reliable and scalable data pipelines.
 </li>
 <li>
 Clean, validate, and transform data from multiple sources for analytics and operational use.
 </li>
 <li>
 Work with large datasets and ensure high data quality and availability.
 </li>
 <li>
 Develop and maintain ETL workflows using standard tools and scripting.
 </li>
 <li>
 Automate data extraction, transformation, and loading processes across systems.
 </li>
 <li>
 Monitor and optimize performance of data workflows for speed and efficiency.
 </li>
 <li>
 Develop, maintain, and optimize complex SQL queries, stored procedures, functions, and views.
 </li>
 <li>
 Design and implement relational database schemas and indexing strategies.
 </li>
 <li>
 Ensure data integrity, security, and compliance within SQL Server environments.
 </li>
 <li>
 Work with MongoDB for storing and querying unstructured or semi-structured data.
 </li>
 <li>
 Design document-based data models and manage performance tuning for MongoDB.
 </li>
 <li>
 Handle replication and backup strategies where applicable.
 </li>
 <li>
 Analyze complex data requirements and translate them into efficient technical solutions.
 </li>
 <li>
 Identify data-related issues, perform root-cause analysis, and deliver timely resolutions.
 </li>
 <li>
 Work closely with data analysts, software engineers, and business stakeholders to understand requirements.
 </li>
 <li>
 Participate in code reviews, team planning sessions, and cross-functional meetings.
 </li>
 <li>
 Take ownership of assigned tasks with accountability and attention to detail.
 </li>
 <li>
 Exhibit a proactive approach to delivering solutions and supporting team goals.
 </li>
 <li>
 Contribute to architectural discussions and recommend improvements based on industry best practices.
 
 
 </li>
 </ul>
 
 Qualifications
 
 
 
 <ul>
 <li>
 Bachelor&rsquo;s degree in Computer Science, Information Systems, or a related field.
 </li>
 <li>
 2&ndash;3 years of experience in data engineering or database development roles.
 </li>
 <li>
 Strong proficiency in SQL Server (T-SQL, indexing, optimization).
 </li>
 <li>
 Hands-on experience with MongoDB or other NoSQL technologies.
 </li>
 <li>
 Familiarity with data pipeline tools and ETL development.
 </li>
 <li>
 Working knowledge of Python or similar scripting languages.
 </li>
 <li>
 Understanding of data modeling, normalization, and data warehousing concepts.
 </li>
 <li>
 Strong analytical thinking and problem-solving skills.
 </li>
 <li>
 Excellent communication and teamwork abilities.
 </li>
 </ul>
</div>

Data Engineer, Tech

Search 9754+ Data pipeline engineer jobs near you this February. New roles added regularly across leading industries. Apply today.

9754 data pipeline engineer jobs NEAR_ME

Data Engineer

Data Engineer

Data Engineer (Hybrid)

Data Engineer I- Pathology

Data Engineer (PySpark)

Data Engineer

Data Engineer, Tech