data pipeline engineer

Pavago

<b>Job Title:</b> Data Engineer
<br><b>Position Type:</b> Full-Time, Remote
<br><b>Working Hours:</b> U.S. client business hours (with flexibility for pipeline monitoring and data refresh cycles)
<br><br>
<b>About the Role:</b><br>
Our client is seeking a Data Engineer to design, build, and maintain reliable data pipelines and infrastructure that deliver clean, accessible, and actionable data. This role requires strong software engineering fundamentals, experience with modern data stacks, and an eye for quality and scalability. The Data Engineer ensures data flows seamlessly from source systems to warehouses and BI tools, powering decision-making across the business.
<br><br>
<b>Responsibilities:</b><b>Pipeline Development:</b><br>
<ul>
<li>Build and maintain ETL/ELT pipelines using Python, SQL, or Scala.</li>
<li>Orchestrate workflows with Airflow, Prefect, Dagster, or Luigi.</li>
<li>Ingest structured and unstructured data from APIs, SaaS platforms, relational databases, and streaming sources.</li>
</ul>
<b>Data Warehousing:</b><br>
<ul>
<li>Manage data warehouses (Snowflake, BigQuery, Redshift).</li>
<li>Design schemas (star/snowflake) optimized for analytics.</li>
<li>Implement partitioning, clustering, and query performance tuning.</li>
</ul>
<b>Data Quality & Governance:</b><br>
<ul>
<li>Implement validation checks, anomaly detection, and logging for data integrity.</li>
<li>Enforce naming conventions, lineage tracking, and documentation (dbt, Great Expectations).</li>
<li>Maintain compliance with GDPR, HIPAA, or industry-specific regulations.</li>
</ul>
<b>Streaming & Real-Time Data:</b><br>
<ul>
<li>Develop and monitor streaming pipelines with Kafka, Kinesis, or Pub/Sub.</li>
<li>Ensure low-latency ingestion for time-sensitive use cases.</li>
</ul>
<b>Collaboration:</b><br>
<ul>
<li>Partner with analysts and data scientists to provide curated, reliable datasets.</li>
<li>Support BI teams in building dashboards (Tableau, Looker, Power BI).</li>
<li>Document data models and pipelines for knowledge transfer.</li>
</ul>
<b>Infrastructure & DevOps:</b><br>
<ul>
<li>Containerize data services with Docker and orchestrate in Kubernetes.</li>
<li>Automate deployments via CI/CD pipelines (GitHub Actions, Jenkins, GitLab CI).</li>
<li>Manage cloud infrastructure using Terraform or CloudFormation.</li>
</ul>
<b>What Makes You a Perfect Fit:</b><br>
<ul>
<li>Passion for clean, reliable, and scalable data.</li>
<li>Strong problem-solving skills with debugging mindset.</li>
<li>Balance of software engineering rigor and data intuition.</li>
<li>Collaborative communicator who thrives in cross-functional environments.</li>
</ul>
<b>Required Experience & Skills (Minimum):</b><br>
<ul>
<li>3+ years in data engineering or back-end development.</li>
<li>Strong Python and SQL skills.</li>
<li>Experience with at least one major data warehouse (Snowflake, Redshift, BigQuery).</li>
<li>Familiarity with pipeline orchestration tools (Airflow, Prefect).</li>
</ul>
<b>Ideal Experience & Skills:</b><br>
<ul>
<li>Experience with dbt for transformations and data modeling.</li>
<li>Streaming data experience (Kafka, Kinesis, Pub/Sub).</li>
<li>Cloud-native data platforms (AWS Glue, GCP Dataflow, Azure Data Factory).</li>
<li>Background in regulated industries (healthcare, finance) with strict compliance.</li>
</ul>
What Does a Typical Day Look Like?
<br>A Data Engineer’s day revolves around keeping pipelines running, improving reliability, and enabling teams with high-quality data. You will:
<br><ul>
<li>Check pipeline health in Airflow/Prefect and resolve any failed jobs.</li>
<li>Ingest new data sources, writing connectors for APIs or SaaS platforms.</li>
<li>Optimize SQL queries and warehouse performance to reduce costs and latency.</li>
<li>Collaborate with analysts/data scientists to deliver clean datasets for dashboards and models.</li>
<li>Implement validation checks to prevent downstream reporting issues.</li>
<li>Document and monitor pipelines so they’re reproducible, scalable, and audit-ready.</li>
</ul>
<b>In essence:</b> you ensure the business has accurate, timely, and trustworthy data powering every decision.
<br><br>
<b>Key Metrics for Success (KPIs):</b><br>
<ul>
<li>Pipeline uptime ≥ 99%.</li>
<li>Data freshness within agreed SLAs (hourly, daily, weekly).</li>
<li>Zero critical data quality errors reaching BI/analytics.</li>
<li>Cost-optimized queries and warehouse performance.</li>
<li>Positive feedback from data consumers (analysts, scientists, leadership).</li>
</ul>
<b>Interview Process:</b><br>
<ul>
<li>Initial Phone Screen</li>
<li>Video Interview with Pavago Recruiter</li>
<li>Technical Task (e.g., build a small ETL pipeline or optimize a SQL query)</li>
<li>Client Interview with Engineering/Data Team</li>
<li>Offer & Background Verification</li>
</ul>


Data Engineer

Princeton IT America

<p>Data Engineer</p><p>Type: Permanent / Contractual (C2H)</p><p>Experience: 8+ Years (Minimum 5 years of relevant experience in Databricks)</p><p>Location: Noida / Hyderabad</p><p>Work Mode: Onsite (5 Days a Week)</p><p>Notice Period: Immediate</p><p>Position Overview</p><p>We are seeking a highly skilled Data Engineer to design and implement scalable data platforms leveraging Databricks. The ideal candidate will have deep expertise in data architecture, pipeline development, and integration of diverse data sources including SQL Server, MongoDB, and InfluxDB. This requires proficiency in both real-time and batch data processing, with a strong foundation in cloud data solutions (preferably Azure). This position offers the opportunity to work on advanced analytics, machine learning enablement, and enterprise-scale data solutions that drive business insights and innovation.</p><p>Key Responsibilities</p><p>&#61623; Design, build, and maintain a robust data platform on Databricks.</p><p>&#61623; Develop scalable ETL/ELT pipelines s to ingest data from multiple sources (SQL Server, MongoDB, InfluxDB) into Databricks Delta Lake.</p><p>&#61623; Implement both real-time and batch data ingestion strategies using Kafka, Azure Event Hubs, or equivalent tools.</p><p>&#61623; Optimize data storage and processing for performance, scalability, and cost efficiency.</p><p>&#61623; Build and maintain data models supporting BI, analytics, and machine learning use cases. &#61623; Collaborate closely with Data Scientists, Analysts, and Product Teams to define and deliver data requirements.</p><p>&#61623; Ensure data quality, security, and governance across all pipelines and data repositories.</p><p>&#61623; Conduct performance tuning, monitoring, and troubleshooting to ensure reliability of data workflows.</p><p>Required Skills &amp; Qualifications</p><p>&#61623; Proven hands-on experience in Databricks, including Delta Lake, Spark, PySpark, and SQL. &#61623; Strong understanding of data integration from heterogeneous systems &mdash; SQL Server, MongoDB, and InfluxDB.</p><p>&#61623; Expertise in ETL/ELT pipeline development and workflow orchestration using tools like Apache Airflow, Azure Data Factory, or similar.</p><p>&#61623; Proficiency in data modeling, data warehousing, and performance optimization techniques.</p><p>&#61623; Experience in real-time data streaming using Kafka, Azure Event Hubs, or related technologies.</p><p>&#61623; Advanced programming skills in Python and SQL.</p><p>&#61623; Working knowledge of Azure Cloud and its data services.</p><p>&#61623; Experience with Change Data Capture (CDC) techniques for incremental data processing. &#61623; Excellent problem-solving, debugging, and analytical skills.</p><p>Job Type: Full-time</p><p>Pay: &#8377;2,000,000.00 - &#8377;2,400,000.00 per year</p><p>Experience:</p><ul><li>Data Engineer: 8 years (Required)</li><li>Delta Lake: 6 years (Required)</li></ul><p>Work Location: In person</p>

JSR Tech Consulting

<div><div>Looking for a senior <b>Data Engineer.</b><br>
AWS, Glue, Python, PySpark, SQL<br>
Data Warehousing experience.<br>
Data Modeling<br>
FABRIC is a plus.<br>
3 days hybrid in Newark, NJ<br>
Potential contract to hire.
<br>
<br>
Building a new AWS infrastructure/applications, taking ownership of an existing DB2 based legacy application.<br>
Build entire end to end ETL pipeline.<br>
<br>
payrate: 65 - 75 per hour depending on experience.<br>
<br>
#LI-LB1</div></div>

Quadrabay Verification Services

<p><b>Position Summary</b></p><p>We are seeking a skilled <b>Data Engineer</b> with strong expertise in building and managing <b>ETL/ELT pipelines using PySpark</b> within an <b>AWS cloud environment</b>. The ideal candidate will be responsible for designing, implementing, and maintaining scalable data workflows to support multiple data sources, while ensuring seamless integration with <b>Power BI</b> for reconciliation, reporting, and advanced analytics.</p><p><b>Key Responsibilities</b></p><p>&middot; <b>Data Pipeline Development</b></p><p>o Design, develop, and maintain scalable <b>ETL/ELT pipelines</b> using <b>PySpark</b>.</p><p>o Implement data ingestion from multiple data sources (Finance, CRM, Payment Gateways, etc.) into AWS Data Lake/Warehouse.</p><p>o Ensure efficient <b>data partitioning, schema management, and optimization</b> for large datasets.</p><p>&middot; <b>Cloud &amp; Infrastructure (AWS)</b></p><p>o Leverage <b>AWS services</b> (S3, Glue, EMR, Redshift, Lambda, Step Functions) for data engineering workflows.</p><p>o Implement <b>data security, access control, and compliance</b> in line with organizational standards.</p><p>o Automate deployments using <b>CI/CD pipelines</b>.</p><p>&middot; <b>Data Integration &amp; Reporting</b></p><p>o Prepare curated datasets for <b>Power BI dashboards</b> and reports.</p><p>o Enable reconciliation processes by providing granular, auditable datasets.</p><p>o Collaborate with BI developers to ensure data accuracy, integrity, and performance in reporting.</p><p>&middot; <b>Collaboration &amp; Process</b></p><p>o Work closely with <b>Business Analysts, Data Scientists, and ERP functional teams</b> to understand requirements.</p><p>o Participate in <b>data governance initiatives</b>, ensuring metadata, lineage, and quality controls are implemented.</p><p>o Provide support for <b>ad-hoc analytics requests</b> from stakeholders.</p><p><b>Required Skills &amp; Qualifications</b></p><p>&middot; <b>Technical Skills</b></p><p>o Strong programming in <b>Python</b> with deep knowledge of <b>PySpark</b>.</p><p>o Proven experience in <b>ETL/ELT pipeline development</b> in distributed environments.</p><p>o Expertise in <b>AWS data services</b> (S3, Glue, EMR, Redshift, Athena, Lambda, Kinesis).</p><p>o Proficiency in <b>SQL (analytical queries, optimization, tuning)</b>.</p><p>o Familiarity with <b>Power BI integration</b> (Direct Query, Dataflows, Gateways).</p><p>&middot; <b>Data Engineering Practices</b></p><p>o Experience with <b>data lakes, warehouses, and star/snowflake schema design</b>.</p><p>o Knowledge of <b>data quality frameworks, reconciliation processes, and auditability controls</b>.</p><p>o Hands-on experience with <b>workflow orchestration tools</b>.</p><p>&middot; <b>Soft Skills</b></p><p>o Strong problem-solving and analytical skills.</p><p>o Ability to collaborate across IT, business, and analytics teams.</p><p>o Clear communication for technical and non-technical stakeholders.</p><p><b>Preferred Qualifications</b></p><p>&middot; Experience with <b>ERP data models</b> (Finance, CRM - Reconciliation).</p><p>&middot; Exposure to <b>machine learning data preparation</b> (feature engineering, training/test datasets).</p><p>&middot; Certification(s):</p><p>o AWS Certified Data Analytics &ndash; Specialty</p><p>o Databricks/Spark Developer Certification</p><p>o Microsoft Power BI Data Analyst Associate</p><p>Job Type: Full-time</p><p>Ability to commute/relocate:</p><ul><li>Ajman: Reliably commute or planning to relocate before starting work (Preferred)</li></ul><p>Application Question(s):</p><ul><li>Availability to start immediately </li><li>Visa Status</li></ul><p>Experience:</p><ul><li>Data Engineer: 3 years (Preferred)</li></ul><p>Language:</p><ul><li>English  (Preferred)</li></ul>

Relevance Labs

<div><p>As an Azure Databricks Engineer, you will be responsible for designing, developing, and maintaining data pipelines and solutions on the Azure Databricks platform. You will work closely with data scientists, data engineers, and business stakeholders to understand requirements, implement solutions, and ensure the reliability and performance of data infrastructure. Responsibilities:</p>
<ul><li>Design, develop, and maintain data pipelines using Azure Databricks.</li>
<li>Implement data ingestion, transformation, and loading processes.</li>
<li>Optimize Databricks clusters for performance and cost efficiency.</li>
<li>Collaborate with data scientists and engineers to build and deploy machine learning models.</li>
<li>Monitor and troubleshoot data pipelines and infrastructure.</li>
<li>Implement data security and governance best practices.</li>
<li>Automate data pipeline deployments and infrastructure provisioning.</li>
<li>Stay up-to-date with the latest Azure Databricks features and best practices.</li></ul></div>

Azure Databricks Engineer

Pragma Edge

<div></div><div><div><div><div><div><div><div><div><div><div><div><div><div><ul><div><div>Job Category - IT</div>
</div><div></div><div><div>Job Type - Full Time</div>
</div><div></div><div><div>No of openings: 01</div>
</div><div></div><div><div>Experience : 6 to 8 Years</div>
</div><div></div><div><div>Job Location - Hyderabad</div>
</div></ul></div></div></div></div></div></div></div></div></div></div><div></div><div><div><div><div><div><div><div>Senior IT Data Engineer</div>
</div></div><div><div><p>The Senior IT Data Engineers are experts in data streaming, building data pipelines that support real-time data refreshes and are cost-optimized for computing resources. They deeply understand data security, implementing row-level and column-level security measures.
</p><p>This role typically involves leading the implementation of complex projects, using advanced big data technologies, and ensuring robust data pipeline orchestration across multiple systems.</p>
</div></div><div><div><div>Job Responsibilities</div>
</div></div><div><div><ul><li>Lead the design and orchestration of complex data pipelines and ETL processes, focusing on optimizing data models.</li><li>
Architect scalable data solutions, including data warehousing and big data technologies, and ensure alignment with organizational goals.</li><li>
Implement advanced data governance frameworks and ensure robust data security measures.</li><li>
Optimize cloud costs by strategically using cloud resources and efficient data architectures.</li><li>
Lead the development of data visualization strategies, ensuring they contribute to data-driven decision-making at a strategic level.</li><li>
Design and deploy enterprise-scale data visualization solutions, optimizing performance within tools.</li><li>
Oversee complex data pipeline orchestration and ensure seamless integration across systems.</li><li>
Interact with 3rd party vendors to incorporate new functionality and evaluate tools.</li><li>
Resolve complex escalations and situations, driving effective resolutions.</li><li>
Ensure adherence to data privacy and security practices to protect sensitive information.</li><li>
Communicate effectively with stakeholders to understand and address their needs through data visualization.</li><li>
Perform other assigned job-related duties that align with our organization&rsquo;s vision, mission, and values and fall within your scope of practice.</li></ul>
</div></div><div><div><div>Qualifications</div>
</div></div><div><div><ul><li>Education: Bachelor&rsquo;s Degree or relevant experience.</li><li>
Preferred Certification(s): Any relevant IT Certification.</li><li>
Experience: 5+ years of relevant and practical experience.</li></ul><p><b>
Special Skills:</b></p><ul><li>
Advanced skills in data streaming and real-time data processing.</li><li>
Expertise in designing complex data architectures and scalable systems.</li><li>
Deep knowledge of big data technologies and extensive experience with data security measures.</li><li>
Ability to lead data architecture reviews and approvals.</li></ul><p><b>
Soft Skills:</b></p><ul><li>
Project Management: Learning and understanding complex data initiatives from conception to completion.</li><li>
Mentorship: Guiding junior team members in their professional development.</li><li>
Strategic Thinking: Aligning data solutions with broader organizational goals.</li><li>
Communication: Articulating data strategies to both technical and non-technical stakeholders.</li><li>
Problem-Solving: Resolving complex data challenges and escalations.</li><li>
Adaptability: Staying current with rapidly evolving data technologies and best practices.</li><li>
Creativity: Innovating new approaches to data pipeline design and optimization.</li></ul></div></div></div></div></div></div></div></div></div><p></p>

COZMALABS

<p>We are looking for an experienced&#8239; Data Engineer&#8239;to join our team full-time. This role requires a strong background in building and optimizing data pipelines, with a focus on supporting AI-driven solutions. The candidate will work closely with software engineers to ensure reliable data flows for AI models and contribute to cloud-based data integration using Microsoft Azure.</p><p>Key Responsibilities:</p><ul><li>Design, develop, and maintain scalable data pipelines and ETL processes to support AI-driven projects</li><li>Build and optimize data infrastructure to ensure efficient data management, processing, and storage in cloud environments (Azure)</li><li>Collaborate with AI engineers to facilitate data preparation for machine learning models and AI solutions</li><li>Ensure data pipelines are secure, reliable, and meet performance standards</li><li>Monitor and troubleshoot data pipeline performance to ensure timely and accurate data delivery</li><li>Implement data governance policies and ensure compliance with data security standards</li></ul><p>Required Qualifications:</p><ul><li>Proven experience as a Data Engineer, with a focus on building and maintaining complex data pipelines for AI projects</li><li>Proficiency in SQL and experience with relational databases (e.g., PostgreSQL, SQL Server) and NoSQL databases (e.g., MongoDB)</li><li>Expertise in cloud-based data solutions, specifically Microsoft Azure (e.g., Azure Data Factory, Azure Synapse, Azure Databricks)</li><li>Experience with data integration tools and ETL frameworks (e.g., Apache Airflow)</li><li>Proficiency in at least one programming language for data manipulation (e.g., Python)</li><li>Knowledge of data warehousing solutions and best practices</li><li>Experience in working with data pipelines for AI or machine learning workflows</li><li>Strong problem-solving and analytical skills</li><li>Bachelor&rsquo;s degree in Computer Science, Engineering, or a related field</li><li>Proficiency in English (written and spoken)</li></ul><p>Desirable Qualifications:</p><ul><li>Familiarity with machine learning models and data preprocessing for AI solutions</li><li>Knowledge of containerization tools (Docker, Kubernetes) for data pipeline deployments</li></ul><p>Job Type: Full-time</p>

Search 10000+ Data pipeline engineer jobs near you this November. New roles added regularly across leading industries. Apply today.

10000 data pipeline engineer jobs NEAR_ME

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Azure Databricks Engineer

Data Engineer

Data Engineer