data pipeline engineer

Caregenix Solutions

We're Hiring: ETL Engineer (Remote &ndash; Full Time)Company: CareGenix SolutionsLocation: RemoteExperience: 3+ YearsType: Full-TimeCaregenix Solutions, a fast-growing healthcare tech company, is seeking a skilled ETL Engineer to design and maintain robust data pipelines that power real-time insights and analytics.Key Responsibilities:Build and manage scalable ETL/ELT pipelines using AWS services (Glue, Lambda, S3, EMR, Redshift, etc.).Design and optimize data models and databases (Redshift, Athena, RDS).Ingest data from diverse sources and ensure quality through validation checks.Collaborate with teams to deliver reliable, insight-driven data solutions.Monitor pipeline performance and troubleshoot as needed.Follow best practices for data security and governance.Requirements6+ years in Data Engineering or similar roles.Strong in SQL and Python or Scala.Hands-on experience with AWS data tools: Glue, Redshift, EMR, Athena, Kinesis, etc.Experience with workflow/orchestration tools (Airflow, Step Functions).Solid knowledge of data warehousing, performance tuning, and big data.Familiarity with CI/CD, Git, and IaC tools (Terraform, CloudFormation).Bachelor&rsquo;s in Computer Science, Engineering, or related field.To Apply Send your resume to careers@caregenix.solutionsSubject Line: ETL EngineerBe part of a team transforming healthcare through data. We&rsquo;d love to hear from youJob Type: Full-timePay: Up to Rs400,000.00 per monthApplication Question(s):<ul><li>what is your expected pay?</li></ul>Experience:<ul><li>Data/ ETL : 2 years (Required)</li></ul>Work Location: Remote

Data Engineer

MOBIZ

<div>About Mobiz

Mobiz is an information technology firm, and we do three things exceptionally well: cloud and network automation, digital transformation, and data protection. Double-digit growth year-over-year has driven us to grow our team and ensure we stay ahead of the curve. 
What Can You Expect?

Every day at Mobiz we work with a deep sense of purpose. We continuously innovate. Our mission is to empower our clients to do more through transformation. You&rsquo;ll work in a collaborative environment alongside highly talented people that improve client operations and exceed expectations. We strive to simplify technology challenges, and no less. 


Who Are We Looking For?

We are looking for a skilled and motivated Data Engineer to join our growing data team. In this role, you will be responsible for designing, building, and operating scalable, secure, and high-performing data solutions that enable analytics, reporting, and data-driven decision-making across the organization. You will collaborate closely with cross-functional teams and leverage modern cloud technologies to transform raw data into trusted, accessible insights. 


Key Responsibilities
<ul>
<li>Design, build, test, deploy, and maintain robust data pipelines using Databricks, Azure Data Factory, and Microsoft Fabric, supporting both batch and streaming workloads.</li>
<li>Ingest, integrate, and harmonize data from multiple cloud and on-premises sources to deliver a unified, reliable, and trusted data view for analytics and reporting.</li>
<li>Use Python and SQL to clean, transform, validate, and model data to meet business, reporting, and analytical requirements.</li>
<li>Contribute to data architecture and platform design decisions, ensuring solutions are scalable, secure, cost-effective, and aligned with long-term business and technical strategy.</li>
<li>Build and maintain metadata-driven pipelines that gracefully adapt to schema changes, new data sources, and evolving ingestion patterns.</li>
<li>Proactively identify data quality issues, pipeline failures, and performance bottlenecks; perform root-cause analysis and implement durable, long-term solutions.</li>
<li>Continuously optimize data pipeline performance, reliability, and scalability to ensure fast, consistent, and dependable access to data.</li>
<li>Implement automated data quality checks, validation rules, and monitoring dashboards to ensure accuracy, consistency, and trust in delivered datasets.</li>
<li>Monitor, analyze, and manage data pipeline and compute costs across Azure, Databricks, and Microsoft Fabric, recommending and implementing cost-optimization strategies without compromising performance or availability.</li>
<li>Automate routine and repetitive data engineering tasks, applying best practices for CI/CD, deployment, testing, and operational support.</li>
<li>Design, implement, and maintain secure data access controls using role-based access, audits, and governance best practices to ensure appropriate data security and compliance.</li>
<li>Maintain clear, accurate, and up-to-date technical documentation, and provide ongoing operational support as data platforms and business needs evolve.</li>
<li>Stay current with emerging data engineering technologies, cloud services, and analytics best practices, and evaluate new tools through proofs of concept when appropriate.</li>
<li>Collaborate closely with product managers, software engineers, analysts, and operations teams to embed high-quality data solutions into core business processes.</li>
<li>Promote and enforce data engineering best practices including version control, modular and reusable pipeline design, documentation standards, and scalable architecture patterns.</li>
<li>Enable and support data democratization by providing well-modeled, secure, and self-service-ready datasets through Power BI, Microsoft Fabric, or custom analytics solutions.</li>
<li>Deliver measurable outcomes such as highly reliable pipelines with minimal manual intervention, on-time pipeline execution within service windows, rapid resolution of critical incidents, and consistently high data quality.</li>
<li>Ensure reusable and standardized pipeline components are leveraged across teams to accelerate delivery, improve consistency, and reduce operational overhead.</li>
<li>Provide stakeholders with reliable, well-modeled datasets that can be confidently explored without requiring ongoing engineering support.</li>
<li>Ensure all data solutions comply with organizational security, access, and governance policies while maintaining positive feedback from internal and external users.</li></ul> 


 Required Skills &amp; Qualifications 
<ul>
<li>3&ndash;5 years of hands-on experience in data engineering or related roles, with proven experience building, supporting, and operating production-grade data pipelines.</li>
<li>Strong hands-on experience with Databricks, including development, optimization, and operational support of Databricks-based data solutions.</li>
<li>Proven experience with Azure Data Factory (ADF) and Microsoft Fabric for data ingestion, orchestration, and transformation.</li>
<li>Advanced proficiency in SQL for querying, data modeling, performance tuning, and troubleshooting.</li>
<li>Strong programming skills in Python for data transformation, automation, scripting, and pipeline development.</li>
<li>Hands-on experience with Microsoft Azure cloud services related to data storage, compute, networking, and security.</li>
<li>Solid understanding of ETL/ELT patterns, data integration strategies, and real-time or near-real-time data processing concepts.</li>
<li>Experience designing metadata-driven and reusable pipeline components that scale across multiple teams and projects.</li>
<li>Strong understanding of data governance, data quality frameworks, monitoring practices, and automated validation techniques.</li>
<li>Experience supporting high-availability data platforms with SLAs, monitoring, alerting, and incident response processes.</li>
<li>Demonstrated ability to optimize cloud compute usage and manage costs efficiently across Azure, Databricks, and Microsoft Fabric.</li>
<li>Experience using version control systems such as Git and following modern software engineering best practices.</li>
<li>Ability to design secure, scalable, and cost-effective data architectures aligned with business needs.</li>
<li>Strong analytical, problem-solving, and troubleshooting skills with a proactive and ownership-driven mindset.</li>
<li>Excellent communication skills, with the ability to explain complex data concepts to both technical and non-technical stakeholders.</li>
<li>Ability to work independently as well as collaboratively in cross-functional team environments.</li>
<li>High attention to detail with a strong focus on data accuracy, reliability, and documentation quality.</li>
<li>Bachelor&rsquo;s degree in Computer Science, Engineering, or a related field.</li>
<li>Relevant certifications are required or strongly preferred, including Databricks Certified Data Engineer, Microsoft Certified: Azure Data Engineer Associate, and Microsoft Certified: Fabric Data Engineer Associate.</li></ul> 


What We Offer
<ul>
<li>A team of bright, hard-working, and innovative people that will contribute to your growth.</li>
<li>Competitive Salary and comprehensive benefits plan.</li>
<li>A dynamic and collaborative work environment with opportunity to work with cutting-edge technology and innovative solutions.</li></ul> 


Other 
This is a full-time office based job in Islamabad, Pakistan. 
Closing

Mobiz embraces diversity and is committed to building a team that represents a variety of backgrounds, perspectives, and skills. The more inclusive we are, the better we are. We are an equal opportunity employer.

We thank all applicants for their interest. We will contact you if you are selected to move forward in the recruiting process.</div>

Match Made Tech

<div><div>Job Title: Data Engineer
LOCATION: Irvine, CA (onsite). Monday through Thursday onsite, Fridays remote.
COMPENSATION: $50-95 an hour. This is a 2-year contract that will convert to full-time.
ABOUT US: We are on a mission to develop innovative AI solutions that will revolutionize our workforce. As we embark on an exciting new greenfield AI project, our data team is forward-thinking and data-driven dedicated to transforming data into actionable insights. We are seeking a skilled Data Engineer to join our team, responsible for leveraging Python, Java, PostgreSQL, Snowflake and AWS to design, develop, and maintain data pipelines and systems that support our analytics and decision-making processes.
JOB DESCRIPTION:
Position Overview: As a Data Engineer, you will play a pivotal role in the development and management of our data infrastructure. You will work closely with cross-functional teams to ensure data availability, reliability, and quality, enabling data-driven decision-making across the organization.
RESPONSIBILITIES:
Data Pipeline Development:
<ul><li>Design, implement, and maintain data pipelines using Python, PostgreSQL, and AWS technologies.
</li><li>Develop efficient and scalable ETL processes to extract, transform, and load data from various sources into our Snowflake data warehouses.
</li></ul>Database Management:
<ul><li>Design and maintain PostgreSQL databases, including schema design, indexing, and performance optimization.
</li><li>Monitor database performance, troubleshoot issues, and implement improvements as needed.
</li><li>AWS Integration:
</li><li>Utilize AWS services, such as AWS Glue, S3, Redshift, and EMR, for data storage, processing, and analytics.
</li><li>Collaborate with AWS solutions architects to optimize data-related AWS infrastructure.
</li></ul>Data Quality and Governance:
<ul><li>Implement data quality checks and validation processes to ensure data accuracy and consistency.
</li><li>Enforce data governance policies and security measures to protect sensitive data.
</li></ul>Documentation and Collaboration:
<ul><li>Maintain comprehensive documentation of data pipelines, database schemas, and ETL processes.
</li><li>Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver actionable insights.
</li></ul>Performance Optimization:
<ul><li>Continuously monitor and optimize data pipelines and database performance for scalability and efficiency.
</li></ul>QUALIFICATIONS:
<ul><li>Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.
</li><li>Proven experience (3+ years) as a data engineer with expertise in Python, Java, PostgreSQL, and AWS.
</li><li>Strong SQL skills and proficiency in data modeling and database design.
</li><li>Hands-on experience with AWS services, particularly Glue, S3, Redshift, and EMR.
</li><li>Knowledge of ETL processes and data integration techniques.
</li><li>Excellent problem-solving skills and attention to detail.
</li><li>Effective communication and collaboration skills to work within a team.
</li><li>A commitment to staying current with industry trends and best practices in data engineering.
</li></ul>WHAT MAKES THIS EXCITING:
<ul><li>Be part of a dynamic team in a data-driven organization.
</li><li>Technical ownership on greenfield projects
</li><li>Contribute to projects that have a meaningful impact on our business and industry.
</li><li>Competitive compensation and opportunities for professional growth.
</li><li>Work in a collaborative and innovative environment.
</li></ul>If you are a skilled Data Engineer with expertise in Python, Java, PostgreSQL, and AWS and are passionate about turning data into insights, we encourage you to apply. Join us in our mission to harness the power of data!</div></div>

Data Engineer (Hybrid)

GSSTech Group

<div>We are seeking a highly skilled Data Engineer with strong expertise in PySpark and the Cloudera Data Platform (CDP). The ideal candidate will design, develop, and maintain scalable data pipelines while ensuring high data quality, performance, and availability across the organisation.
This role requires hands-on experience in big data ecosystems, cloud-native technologies, and advanced data processing frameworks. You will collaborate with cross-functional teams to build reliable and high-performance data solutions that drive business insights.<h3 class="jobSectionHeader">
Key Responsibilities</h3><h3 class="jobSectionHeader">
1. Data Pipeline Development</h3><ul>
<li>Design, develop, and maintain scalable ETL/ELT pipelines using PySpark on CDP
</li><li>Ensure data integrity, reliability, and performance optimisation
</li></ul><h3 class="jobSectionHeader">2. Data Ingestion</h3><ul>
<li>Develop ingestion frameworks to collect data from relational databases, APIs, streaming sources, and file systems
</li><li>Load structured and unstructured data into Data Lake/Data Warehouse environments
</li></ul><h3 class="jobSectionHeader">3. Data Transformation &amp; Processing</h3><ul>
<li>Process, cleanse, and transform large-scale datasets using PySpark
</li><li>Build reusable data processing components
</li></ul><h3 class="jobSectionHeader">4. Performance Optimisation</h3><ul>
<li>Tune Spark jobs and Cloudera components for optimal performance
</li><li>Optimise memory, partitioning, and execution plans
</li><li>Reduce ETL runtime and improve cluster efficiency
</li></ul><h3 class="jobSectionHeader">5. Data Quality &amp; Validation</h3><ul>
<li>Implement data validation checks and monitoring mechanisms
</li><li>Ensure end-to-end data quality and governance standards
</li></ul><h3 class="jobSectionHeader">6. Automation &amp; Orchestration</h3><ul>
<li>Automate workflows using tools such as Apache Oozie, Apache Airflow, or similar orchestration frameworks
</li><li>Maintain CI/CD integration for data pipelines
</li></ul><h3 class="jobSectionHeader">7. Monitoring &amp; Support</h3><ul>
<li>Monitor pipeline health and troubleshoot failures
</li><li>Provide production support and continuous improvements
</li></ul><h3 class="jobSectionHeader">Required Skills &amp; Qualifications</h3><ul>
<li>5+ years of experience in Data Engineering
</li><li>Strong hands-on experience in PySpark
</li><li>Experience working on Cloudera Data Platform (CDP)
</li><li>Strong knowledge of Hadoop ecosystem (HDFS, Hive, Impala, YARN)
</li><li>Proficiency in SQL and data modelling concepts
</li><li>Experience with workflow orchestration tools (Airflow, Oozie, etc.)
</li><li>Good understanding of data warehousing concepts
</li><li>Experience with performance tuning and optimisation
</li></ul><h3 class="jobSectionHeader">Good to Have</h3><ul>
<li>Experience with cloud platforms (AWS, Azure, GCP)
</li><li>Knowledge of streaming tools (Kafka, Spark Streaming)
</li><li>Exposure to DevOps practices and CI/CD pipelines
</li><li>Banking/Financial Services domain experience</li></ul></div>

Data Engineer (PySpark)

Accenture

Project Role : Data Engineer 
Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems.
 
Must have skills : Data Analysis &amp; Interpretation 
Good to have skills : Oracle Procedural Language Extensions to SQL (PLSQL), Data Analytics, Tableau 
Minimum 3 year(s) of experience is required 
Educational Qualification : 15 years full time education 
 
Summary: As a Data Engineer, you will engage in the design, development, and maintenance of data solutions that facilitate data generation, collection, and processing. Your typical day will involve creating data pipelines, ensuring data quality, and implementing ETL processes to effectively migrate and deploy data across various systems. You will collaborate with team members to address challenges and enhance data workflows, contributing to the overall efficiency and effectiveness of data management within the organization. Roles &amp; Responsibilities: - Expected to perform independently and become an SME. - Required active participation/contribution in team discussions. - Contribute in providing solutions to work related problems. - Assist in the optimization of data processing workflows to enhance performance. - Collaborate with cross-functional teams to identify data needs and develop appropriate solutions. Professional &amp; Technical Skills: - Must To Have Skills: Proficiency in Data Analysis &amp; Interpretation, Data Analytics, Tableau, Oracle Procedural Language Extensions to SQL (PLSQL), Power BI and Excel. - Good To Have Skills: Experience with Data Analytics, Tableau, Oracle Procedural Language Extensions to SQL (PLSQL). - Good knowledge of the Data Modelling, Data Controls &amp; Assurance. - Have experience in Python for Data Manipulation &amp; Automation - Have experience with ETL/ELT and Data Pipeline Orchestration - Have understanding with Machine Learning Concepts - Strong understanding of data modeling and database design principles. Additional Information: - The candidate should have minimum 3 years of experience in Data Analysis &amp; Interpretation. - This position is based at our Gurugram office. - A 15 years full time education is required.
 
15 years full time education

ibex

<div>
 We are seeking a highly motivated and responsible
 
 Data Engineer
 
 to join our dynamic team. This role requires hands-on experience in building scalable data pipelines, managing structured and unstructured databases, and supporting end-to-end data operations. The ideal candidate will bring strong problem-solving skills, a collaborative mindset, and a commitment to data excellence.
 
 
 
 Responsibilities
 
 
 
 <ul>
 <li>
 Design, develop, and manage reliable and scalable data pipelines.
 </li>
 <li>
 Clean, validate, and transform data from multiple sources for analytics and operational use.
 </li>
 <li>
 Work with large datasets and ensure high data quality and availability.
 </li>
 <li>
 Develop and maintain ETL workflows using standard tools and scripting.
 </li>
 <li>
 Automate data extraction, transformation, and loading processes across systems.
 </li>
 <li>
 Monitor and optimize performance of data workflows for speed and efficiency.
 </li>
 <li>
 Develop, maintain, and optimize complex SQL queries, stored procedures, functions, and views.
 </li>
 <li>
 Design and implement relational database schemas and indexing strategies.
 </li>
 <li>
 Ensure data integrity, security, and compliance within SQL Server environments.
 </li>
 <li>
 Work with MongoDB for storing and querying unstructured or semi-structured data.
 </li>
 <li>
 Design document-based data models and manage performance tuning for MongoDB.
 </li>
 <li>
 Handle replication and backup strategies where applicable.
 </li>
 <li>
 Analyze complex data requirements and translate them into efficient technical solutions.
 </li>
 <li>
 Identify data-related issues, perform root-cause analysis, and deliver timely resolutions.
 </li>
 <li>
 Work closely with data analysts, software engineers, and business stakeholders to understand requirements.
 </li>
 <li>
 Participate in code reviews, team planning sessions, and cross-functional meetings.
 </li>
 <li>
 Take ownership of assigned tasks with accountability and attention to detail.
 </li>
 <li>
 Exhibit a proactive approach to delivering solutions and supporting team goals.
 </li>
 <li>
 Contribute to architectural discussions and recommend improvements based on industry best practices.
 
 
 </li>
 </ul>
 
 Qualifications
 
 
 
 <ul>
 <li>
 Bachelor&rsquo;s degree in Computer Science, Information Systems, or a related field.
 </li>
 <li>
 2&ndash;3 years of experience in data engineering or database development roles.
 </li>
 <li>
 Strong proficiency in SQL Server (T-SQL, indexing, optimization).
 </li>
 <li>
 Hands-on experience with MongoDB or other NoSQL technologies.
 </li>
 <li>
 Familiarity with data pipeline tools and ETL development.
 </li>
 <li>
 Working knowledge of Python or similar scripting languages.
 </li>
 <li>
 Understanding of data modeling, normalization, and data warehousing concepts.
 </li>
 <li>
 Strong analytical thinking and problem-solving skills.
 </li>
 <li>
 Excellent communication and teamwork abilities.
 </li>
 </ul>
</div>

Data Engineer, Tech

Deutsche Bank

<div><h2 class="jobSectionHeader">Job Description:
</h2>Job Title: GCP Data Engineer, AS
Location: Pune, India
Role Description
It&rsquo;s a GCP Data Engineer role, performing end to end development of integrations, data preparation/synthesization/transformation, data quality, data storage workloads, along with usage of the prepared data for advanced data analytics (may include some of the AI use cases). It encompasses the L3 support activities as well for the implemented workloads.
What we&rsquo;ll offer you
As part of our flexible scheme, here are just some of the benefits that you&rsquo;ll enjoy
<ul><li>Best in class leave policy.
</li><li>Gender neutral parental leaves
</li><li>100% reimbursement under childcare assistance benefit (gender neutral)
</li><li>Sponsorship for Industry relevant certifications and education
</li><li>Employee Assistance Program for you and your family members
</li><li>Comprehensive Hospitalization Insurance for you and your dependents
</li><li>Accident and Term life Insurance
</li><li>Complementary Health screening for 35 yrs. and above
</li></ul>Your key responsibilities
<ul><li>Design, develop and maintain data pipelines using Python and SQL programming language on GCP.
</li><li>Expertise on API development and integration
</li><li>Experience on writing SQL queries and optimizing it.
</li><li>Strong programming skills with Python.
</li><li>Experience with GCP Composer or Apache airflow.
</li><li>Hands-on experience on streaming data pipelines.
</li><li>Worked on near real-time services like Pub Sub/Kafka.
</li><li>Proven experience on app development and hosting over cloud platform with help of Docker (preferably GCP, cloud run).
</li><li>A willingness to accept failures, learn and try again.
</li><li>Experience in Agile methodologies, ETL, ELT, Data movement and Data processing skills.
</li><li>Work with Cloud Composer to manage and process batch data jobs efficiently.
</li><li>Develop and optimize complex SQL queries for data analysis, extraction, and transformation.
</li><li>Develop and deploy google cloud services using Terraform.
</li><li>Consume and Hosting REST API using Python.
</li><li>Build and deploy AI and RAG pipelines
</li><li>Design, build and deploy agentic workflows
</li><li>Operationalize AI applications
</li><li>Knowledge of RAG, vector database, multi-agent system, agentic framework &ndash; ADK along with a2a protocol and agent cards
</li><li>Experienced in Vertex AI platform
</li><li>Implement CI CD pipeline using GitHub Action
</li><li>Monitor and troubleshoot data pipelines, resolving any issues in a timely manner.
</li><li>Ensure team collaboration using Jira, Confluence, and other tools.
</li><li>Ability to quickly learn new any existing technologies Strong problem-solving skills.
</li><li>Write advanced SQL and Python scripts.
</li><li>Strategize novel approaches for designing, implementing, deploying robust and scalable ai systems
</li><li>Certification on Professional Google Cloud Data engineer will be an added advantage.
</li></ul>Your skills and experience
<ul><li>6-10 years of IT experience, as a hands-on technologist.
</li><li>Proficient in Python
</li><li>Proficient in SQL.
</li><li>Hands on experience on GCP Cloud Composer, Data Flow, Big Query, Cloud Function, Cloud Run, Google ADK and good to have GKE
</li><li>Hands on experience in building and managing AI workloads and REST API hosting and consumptions.
</li><li>Proficient in Terraform/ Hashicorp.
</li><li>Experienced in GitHub and Git Actions
</li><li>Experienced in CI-CD
</li><li>Experience in automating ETL testing using python and SQL.
</li><li>Good to have APIGEE.
</li><li>Good to have Bit Bucket
</li><li>Understanding of LLM(Gemini) and embedding models
</li><li>Experienced in prompt engineering
</li></ul>How we&rsquo;ll support you
<ul><li>Training and development to help you excel in your career
</li><li>Coaching and support from experts in your team
</li><li>A culture of continuous learning to aid progression
</li><li>A range of flexible benefits that you can tailor to suit your needs
</li></ul>About us and our teams
Please visit our company website for further information:
https://www.db.com/company/company.html
We strive for a culture in which we are empowered to excel together every day. This includes acting responsibly, thinking commercially, taking initiative and working collaboratively.
<div><div><div><div>Together we share and celebrate the successes of our people. Together we are Deutsche Bank Group.
We welcome applications from all people and promote a positive, fair and inclusive work environment.</div></div></div></div></div>

GCP Data Engineer, AS

Search 9418+ Data pipeline engineer jobs near you this February. New roles added regularly across leading industries. Apply today.

9418 data pipeline engineer jobs NEAR_ME

Data Engineer

Data Engineer

Data Engineer (Hybrid)

Data Engineer (PySpark)

Data Engineer

Data Engineer, Tech

GCP Data Engineer, AS