data pipeline engineer

Hexegic

<div>Hexegic is a leading technical consultancy providing agile multi-disciplinary teams to high performing organisations. The company promises exciting, engaging and rewarding projects for those that are keen to develop and build a successful career.
The Role
The Data Pipeline Engineers ensure that our customer work with updated and correct data in our data analytics tool. You are the first to respond to data health failures on key pipelines. You will be comfortable reading code to identify fixes and making changes.
Core Responsibilities
<ul><li>Maintain build schedules so that pipelines run effectively.
</li><li>Setting up and maintaining health checks on different data pipelines
</li><li>Respond, Triage, and debug the data pipeline when broken.
</li><li>Reading code and writing code changes and/or modifying the monitoring set-up
</li><li>Knowing and understanding how to navigate the data pipelines and documentation.
</li><li>Following Standard Operating Procedures to contact other teams and data providers when data is incorrect or not received on time.
</li><li>Communicating outages with the end users of a data pipeline
</li></ul>What We Value
<ul><li>Comfortable reading and writing code in Python, Pyspark</li><li>
Basic understanding of Spark and interested in learning the basics of tuning Spark jobs.
</li><li>Data pipeline monitoring team members should be able to use and navigate pipeline development tools.
</li><li>Experience supporting data integration technologies.</li></ul></div>

Data Pipeline Engineer

Netbuilder

Location: New York (office-based) with travel to client sites as requiredSalary: Up to $140,000 per year (dependent on experience)Role type: Full-time, permanentDepartment: Technology Consulting &ndash; Observability PracticeNETbuilder is building an Observability Practice, and this is a senior delivery hire into it. You will join as a senior data pipeline engineer, working hands-on inside enterprise client teams to architect, deploy and operate the observability platforms and telemetry pipelines that sit at the heart of modern observability.This is primarily a hands-on delivery role &ndash; a senior individual-contributor position rather than a line-management one. Your focus is engineering across two fronts: designing and deploying the observability platforms themselves, and building the data pipelines that collect, route, transform, enrich and deliver telemetry reliably and cost-effectively across complex enterprise environments. You will be the senior technical authority on observability platform architecture and data pipelines for the engagements you work on. While there is no permanent team to manage, on specific engagements you will provide technical leadership &ndash; mentoring junior engineers, planning the workstream and allocating tasks across the delivery team.Consulting experience is preferred but not essential &ndash; what matters most is deep, current, hands-on expertise with industry-standard observability platforms and their pipeline and transformation tooling. If you would rather solve hard data-engineering and platform problems and lead through expertise than carry permanent people-management responsibility, this is the role.Key Responsibilities:Pipeline Engineering &amp; Delivery<ul><li>Design, build and operate telemetry pipelines that collect, route, transform, enrich, reduce and mask logs, metrics and traces at enterprise scale</li><li>Develop and maintain data transformation logic &ndash; parsing, restructuring, filtering and normalising telemetry across formats and schemas</li><li>Optimise pipelines for data quality, cost control and performance &ndash; reducing noise and volume without losing signal</li><li>Onboard complex applications and data sources onto observability platforms</li><li>Build, test and document reusable pipeline configurations, processors and templates</li><li>Monitor pipeline health and throughput, and troubleshoot ingestion, routing and delivery issues</li></ul>Platform Architecture &amp; Deployment<ul><li>Architect observability platform deployments &ndash; topology, sizing, high availability, resilience and scaling &ndash; to meet enterprise client requirements</li><li>Deploy, configure and operate observability platforms and their collection/agent fleets across cloud, hybrid and on-premises environments</li><li>Define deployment patterns and infrastructure-as-code for repeatable, automated platform rollouts</li><li>Integrate observability platforms with client infrastructure, data sources and downstream destinations</li><li>Own platform health, upgrades and capacity planning, and keep architecture and deployment documentation current</li></ul>Client Delivery<ul><li>Deliver observability platform and pipeline engagements directly with enterprise clients as a senior practitioner</li><li>Own delivery outcomes for the platform and pipeline workstreams on assigned engagements</li><li>Act as senior technical authority and escalation point for platform architecture, deployment and data pipeline issues</li><li>Lead troubleshooting and root-cause analysis across enterprise observability platforms and telemetry pipelines</li><li>Support pre-sales with technical scoping and solution design for platform and pipeline work</li><li>Translate client requirements into platform and pipeline designs that balance coverage, cost and compliance</li></ul>Engagement Leadership &amp; MentoringThis is engagement-scoped technical leadership rather than permanent line management &ndash; on specific engagements you will:<ul><li>Provide technical leadership as the senior practitioner, setting direction and quality standards for the platform and pipeline workstream</li><li>Plan the workstream &ndash; scope the work, sequence the tasks and allocate them across the engagement delivery team where required</li><li>Mentor and coach junior engineers and consultants working alongside you, sharing platform and pipeline expertise on the job</li><li>Co-ordinate with the wider engagement team and report progress on the workstream</li></ul>Tooling &amp; Practice Contribution<ul><li>Work hands-on across the observability tool landscape (see preferred tools below), applying each platform's architecture, deployment and data-pipelining capabilities</li><li>Stay current with platform capabilities, roadmaps and certifications</li><li>Contribute to NETbuilder's platform and pipeline methodology, toolkits and reusable assets</li></ul>Qualifications and Experience:<ul><li>Senior data pipeline engineering experience: demonstrable hands-on track record designing, building and operating telemetry/observability data pipelines at enterprise scale</li><li>Platform architecture and deployment experience: hands-on experience designing, deploying and operating observability platforms (topology, sizing, high availability and scaling) across cloud, hybrid and on-premises environments</li><li>Data transformation expertise: strong practical experience with parsing, routing, filtering, enrichment and normalisation of log, metric and trace data</li><li>Tooling experience: hands-on experience with the architecture, deployment and data-pipelining functionality of one or more industry-standard observability platforms (see preferred tools below)</li><li>Engagement leadership: comfortable mentoring junior engineers and planning and allocating tasks within a delivery team on an engagement, without formal line-management authority</li><li>Consulting experience: preferred but not essential &ndash; comfortable working directly within enterprise client teams</li></ul>Preferred Tool Experience:Hands-on experience with the architecture, deployment and data-pipelining functionality of one or more of the following platforms, in order of preference:<ul><li>Bindplane (ObservIQ / Google)</li><li>Dynatrace (OpenPipeline)</li><li>OpenTelemetry (Collector &ndash; receivers, processors and exporters)</li><li>Cribl (Stream)</li><li>Elastic (ingestpipelines)</li><li>Splunk</li></ul>Preferred Certifications:Certification in one or more of the following is preferred:<ul><li>Dynatrace Professional (Dynatrace Master advantageous)</li><li>OpenTelemetry Certified Associate (OTCA) &ndash; also covers Bindplane, which is OpenTelemetry-native</li><li>Cribl Certified Engineer (CC Engineer) or Cribl Certified Observability Engineer &ndash; Admin (CCOE Admin)</li><li>Elastic Certified Engineer</li><li>Splunk Enterprise Certified Architect</li></ul>Key Skills:<ul><li>Platform architecture and deployment: designing observability platform topology (sizing, HA, scaling) and deploying/automating platform and pipeline infrastructure as code</li><li>Pipeline and transformation tooling: deep expertise in one or more observability platforms and their transformation languages and processors</li><li>OpenTelemetry: strong working knowledge of the OpenTelemetry Collector (receivers, processors, exporters) and OTLP</li><li>Scripting and automation: Python, Bash</li><li>Data formats and parsing: JSON, regex, syslog, and common log, metric and trace formats</li><li>Cloud platform knowledge: AWS, Azure, and/or GCP</li><li>Container and orchestration: Kubernetes, Docker, OpenShift</li><li>Familiar with delivery and project tooling: JIRA, Confluence, ServiceNow</li></ul>About You:<ul><li>A hands-on engineer who would rather architect and build a solution than sit in a management meeting</li><li>Happy to lead through expertise &ndash; mentoring others, planning a workstream and giving direction on an engagement, even without a formal management title</li><li>Comfortable working directly inside enterprise client teams and engaging senior stakeholders</li><li>Rigorous about data quality, cost and reliability &ndash; you care about getting the right signal at the right price</li><li>Self-managing, delivery-focused and comfortable in a fast-moving environment</li><li>Naturally curious &ndash; you keep pace with a fast-evolving observability tooling landscape</li></ul>About NETbuilder:NETbuilder is a leading provider of digital solutions, software, consulting and managed services. We work across multiple sectors with specialist expertise in financial services, government, and commercial markets. Since 1999 we have been providing end-to-end solutions across Digital Delivery, Development and Technology. At our core we are a Digital Transformation consultancy with capabilities for onsite, onshore and offshore delivery.You will join a world-class team of experienced consultants, helping to build a new observability practice within the NETbuilder group.Why NETbuilder:<ul><li>Join a new observability practice as a senior technical specialist, working with cutting-edge observability platforms</li><li>Work with enterprise clients across multiple sectors</li><li>Competitive base salary: $120,000 - $140,000 (dependent on experience)</li><li>Continuous investment in professional development and certification</li><li>High-performance, close-knit team culture where individual contribution is recognised</li><li>Office-based New York, with travel to enterprise client sites</li></ul>Pay: Up to $140,000.00 per yearBenefits:<ul><li>Health insurance</li><li>Paid time off</li></ul>Application Question(s):<ul><li>Do you have a demonstrable hands-on track record of designing, building and operating telemetry/observability data pipelines?</li></ul>Ability to Commute:<ul><li>New York, NY 10036 (Required)</li></ul>Willingness to travel:<ul><li>25% (Required)</li></ul>Work Location: In person

Senior Data Pipeline Engineer

Everything Managed Group

<div>Data Engineer: 
The Everything Managed Group is a dynamic, innovative and technology driven business committed to leveraging data-driven insights to optimise operations and drive strategic decision-making. We are now seeking a talented Data Engineer to play a significant part in the design, delivery and ongoing evolution of the EMG data platform. 
Summary:

As a Data Engineer, you will play a critical role in transforming data into actionable insights and supporting our data infrastructure. You will collaborate with cross-functional teams to gather requirements, design data pipelines and transformations, and develop analytical solutions to address business challenges. 
Responsibilities:
<div>
<ul><li>Collaborate with cross-functional teams, including system administrators, developers / engineers, external partners, and business stakeholders, to understand requirements and deliver solutions.</li></ul>
<ul><li>Design, build and maintain scalable data pipelines to ingest, process and transform structured and unstructured data from diverse sources.</li></ul>
<ul><li>Optimise solutions for performance, reliability, and scalability, ensuring efficient data processing and storage.</li></ul>
<ul><li>Perform exploratory profiling and validation on new data sources to assess quality, structure and fitness for purpose prior to pipeline development.</li></ul>
<ul><li>Implement and manage orchestration workflows to automate data pipeline scheduling monitoring and error handling.</li></ul>
<ul><li>Design and implement data quality and consistency schemes to ensure the integrity and consistency of data, management information and reporting.</li></ul>
<ul><li>Design and implement well-structured optimised data models that enable accurate, performant reporting and analysis by downstream stakeholders both within the team and across the wider organisation.</li></ul>
<ul><li>Ensure the data platform is fully documented, maintaining up-to-date information requirements, data models, data dictionaries, and ETL mappings.</li></ul>
<ul><li>Provide ad-hoc deep dive analysis, reporting requests and management information to business users.</li></ul>
<ul><li>Communicate technical concepts and findings to non-technical audiences in a clear and understandable manner.</li></ul>
<ul><li>Provide training and support to others, sharing best practices and fostering culture of continuous learning.</li></ul>
<ul><li>Plan and prioritise work to ensure on time delivery of tasks.</li></ul></div> 


Skills and Experience:
<ul>
<li>Degree in Mathematics, Statistics, Computer Science, Engineering or a related field or equivalent practical experience in a data engineering role. .</li>
<li>Ideally experienced as a Data Engineer with hands-on experience designing and delivering data pipelines using industry-standard tools and cloud technologies.</li>
<li>Knowledge of data engineering concepts, data pipeline development, ETL processes, data lake and data warehousing technologies.</li>
<li>Proficiency in working with at least one major cloud data platform or lakehouse technology.</li>
<li>Data modelling skills (including 3NF and dimensional modelling).</li>
<li>Good knowledge of relational database technologies and concepts.</li>
<li>Excellent analytical and problem-solving skills with a keen attention to detail.</li>
<li>Good verbal and written communication skills with the ability to collaborate across teams and present findings to stakeholders.</li>
<li>Experience with data transformation and/or distributed processing concepts, technologies and best practices is a benefit.</li>
<li>Proficiency in Python for data engineering tasks (e.g. pipeline scripting, data transformation, automation)</li>
<li>Experience using version control (Git) and familiarity with CI/CD practices for data engineering workflows.</li>
<li>Ability to work both independently and collaboratively in a dynamic environment.</li>
<li>Strong customer focus and stakeholder skills.</li>
<li>Ability to prioritise tasks and work effectively in a fast-paced environment.</li></ul></div>

Data Engineer

Nous Infosystems

<div>
 
 
 
 Role: Data Engineer
 
 
 
 
 Location: London, UK (2 days/week onsite)
 
 
 
 
 
 
 
 
 
 
 
 
 Role Overvie
 
 
 
 w:Looking for a Data Engineer to build and maintain scalable data pipelines and support large-scale analytics systems. You&rsquo;ll work with high-volume data using modern orchestration frameworks and big data technologie
 
 
 
 
 
 
 s.
 
 Key Responsibiliti
 
 
 <ul>
 <li>
 es:Build and maintain data pipelines using Pyt
 </li>
 <li>
 honDevelop and optimize ETL workfl
 </li>
 <li>
 owsWrite complex SQL queries (Presto / Spark S
 </li>
 <li>
 QL)Work with Spark for large-scale data process
 </li>
 <li>
 ingEnsure data quality, monitoring, and reliabil
 </li>
 <li>
 ityCollaborate with cross-functional teams (Data Science, Product, Analyti
 </li>
 </ul>
 
 
 
 
 
 cs)
 
 Tech S
 
 
 <ul>
 <li>
 tackLanguages: Python,
 </li>
 <li>
 SQLTools: Dataswarm (Airflow-like), Spark, Presto,
 </li>
 <li>
 HiveOthers: Data catalog, dashboarding, lineage &amp; monitoring t
 </li>
 </ul>
 
 
 
 
 
 ools
 
 Required S
 
 
 <ul>
 <li>
 killsStrong Python + SQL expe
 </li>
 <li>
 rtiseExperience with data pipeline/orchestration tools (Airflow or sim
 </li>
 <li>
 ilar)Hands-on with Spark / big data proce
 </li>
 <li>
 ssingGood understanding of data warehousing con
 </li>
 <li>
 ceptsExperience working with large dat
 </li>
 <li>
 asetsExperience in FAANG-scale environ
 </li>
 <li>
 mentsExposure to data lineage / metrics frame
 </li>
 <li>
 worksFamiliarity with real-time monitoring
 </li>
 </ul>
 
 
 
 tools
</div>

Roke

<div>
 
 
 Data Engineer - National Security Business
 
 
 
 
 
 
 Be part of a growing and highly trusted supplier into the NS domain working to deliver mission critical solutions helping to keep the nation safe, secure and prosperous.
 
 
 
 
 
 
 
 
 Working on leading edge technology solutions including AI/DS, Cyber, Cloud, DevOPs/SRE, Platform Engineering
 
 
 
 
 
 
 
 We have secured long term work, across the full spectrum, on the latest framework with the client, which provides the springboard for our ongoing growth and development in this domain, so join us on what will be an incredible growth journey.
 
 
 
 
 
 
 
 
 Role
 
 
 
 
 
 
 
 
 As a Data Engineer, you&rsquo;ll be actively involved in development of mission critical technical solutions that focus on data services for our National Security customers.
 
 
 
 
 
 You will work alongside our customers to solve their complex and unique challenges.
 
 
 
 
 
 As our next Data Engineer, you&rsquo;ll be managing and developing data pipelines and analytics that transform raw data into valuable insights for Roke&rsquo;s National Security customers, enabling downstream analytics and reporting. You&rsquo;ll be working with diverse data sources, applying distributed compute techniques to handle large datasets.
 
 
 
 
 
 
 
 
 
 
 
 You will be responsible for
 
 
 
 
 
 
 
 <ul>
 <li>
 Data pipeline development - Data ingestion and pipeline orchestration design and tooling.
 </li>
 <li>
 Data integration - Integrating and enriching data from various sources, ensuring data consistency and quality.
 </li>
 <li>
 Data analytics &ndash; Extracting valuable insights from data to provide customers with actionable intelligence.
 </li>
 <li>
 Data security - Implementing data security measures to protect sensitive information.
 </li>
 <li>
 Data systems &ndash; Monitoring for performance issues and making any necessary optimisations/updates.
 </li>
 <li>
 Helping the scrum team decompose user requests and key results into epics and stories.
 </li>
 <li>
 Writing clean, secure code following a test-driven approach.
 </li>
 <li>
 Creating code that is open by default and easy for others to reuse.
 </li>
 </ul>
 
 
 
 
 
 
 
 Required Skills
 
 
 
 
 
 
 
 <ul>
 <li>
 Experience with Big Data technologies, such as Apache Hadoop, Spark or Pig
 </li>
 <li>
 AWS
 </li>
 <li>
 Apache NiFi
 </li>
 <li>
 Python or Java
 </li>
 </ul>
 
 
 
 
 
 
 Built on over a 70 year heritage, Roke offers specialist knowledge in sensors, communications, cyber, and AI and ML. We change the way organisations think and act &ndash; through dynamic insights from the analysis of multiple layers of data. We take care of the innovative, technical stuff that keeps everyone safe &ndash; that&rsquo;s our mission, passion, and motivation.
 
 
 
 
 
 
 
 
 Where you&rsquo;ll work&hellip;
 
 
 
 
 
 
 
 
 
 GLOUCESTER
 
 - Alongside hybrid and flexible working options, you&rsquo;ll find our Gloucester site in a business park two minutes from junction 11A of the M5; The site allows easy access to our local customer base. Set on the outskirts of the Cotswolds, you are never far from a picturesque view or lunch time walk.
 
 
 
 
 
 
 
 
 Clearances&hellip;
 
 
 
 Due to the nature of this role, we require you to be eligible to achieve DV clearance.
 
 
 
 
 
 
 
 
 The Next Step..
 
 .
 
 
 Click apply, submitting an up-to-date CV. We look forward to hearing from you.
 
</div>

Programming.com

Data Engineer &ndash; Mulehunter.aiJob Location<ul><li>Mumbai, Maharashtra (On-site / Hybrid)</li></ul>About Mulehunter.ai &amp; Why This Role MattersMulehunter.ai is an advanced AI/ML-based system designed to detect mule accounts across the banking ecosystem. Reliable, well-structured data pipelines are the critical foundation on which our modeling, alerting, and reporting workflows operate.In this role, you will design and execute robust data workflows to support advanced analytics and machine learning initiatives. Operating from our Mumbai office, you will collaborate closely with data scientists and business stakeholders to manage, transform, and operationalize large-scale financial and transactional datasets.What You Will Work On<ul><li>Data pipelines and infrastructure for Mulehunter.ai, managing large-scale banking datasets built to power AI/ML fraud detection at scale.</li></ul>What You Will Do<ul><li>Data Integration: Extract structured and unstructured data from diverse banking databases, ensuring secure, high-throughput, and efficient data flow into modeling pipelines.</li><li>Data Preprocessing: Clean raw financial datasets, handle missing values, normalize formats, and prepare complex account-level and transaction-level data for downstream ML applications.</li><li>BRE Query Development: Design and implement optimized SQL-based queries and logic for the Business Rule Engine (BRE) based on exploratory data analysis (EDA) insights.</li><li>ML Collaboration: Partner tightly with the Data Science team to support complex data preparation, automated feature engineering, and live ML experiments.</li><li>Pipeline Management: Build, maintain, schedule, and monitor production-grade ETL/ELT workflows to guarantee system reliability.</li><li>Security &amp; Compliance: Enforce data security protocol, strict access controls, and complete data auditability at every layer of the data pipeline.</li></ul>What We Are Looking For<ul><li>Experience: 3&ndash;8 years of core experience in data engineering, preferably within the BFSI, financial crime, or AML/fraud domains.</li><li>Core Technical Skills: Strong mastery of SQL, advanced data query optimization, and proficiency in Python for data manipulation (pandas, NumPy, etc.).</li><li>Database Systems: Hands-on experience with relational databases (PostgreSQL, MySQL, Oracle) and data warehousing solutions (Snowflake, Redshift, BigQuery, or Azure Synapse).</li><li>Big Data &amp; Orchestration: Solid experience with big data frameworks (Apache Spark, Hadoop) and pipeline automation/orchestration tools (Kafka, Apache Airflow).</li><li>Domain Understanding: Prior exposure to handling highly sensitive financial datasets, specifically account-level and transaction-level ledger data.</li></ul>What Success Looks Like<ul><li>Robust Infrastructure: Production-grade, highly reliable data pipelines consistently serving the machine learning modeling team.</li><li>Data Readiness: Clean, well-modeled, and structured datasets readily optimized for real-time analysis and model training.</li><li>Scalable Performance: Efficient, low-latency data infrastructure engineered to support high-volume fraud detection across the banking ecosystem.</li></ul>Pay: ₹510,440.24 - ₹1,868,453.81 per yearBenefits:<ul><li>Health insurance</li><li>Paid sick time</li><li>Provident Fund</li></ul>Work Location: In person

Search 10000+ Data pipeline engineer jobs near you this July. New roles added regularly across leading industries. Apply today.

10000 data pipeline engineer jobs Near Me

Data Pipeline Engineer

Senior Data Pipeline Engineer

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Data Engineer