data pipeline engineer

Seeka Technology

<div>Company Description 
We believe strong early career roles should do more than give someone a job title. They should help people grow into the kind of professionals they want to become. At SEEKA Technologies, we give junior team members meaningful work that builds real capability, sharpens problem-solving, and develops practical experience in fast-moving business and technology environments. Our goal is to help emerging talent strengthen their skills, expand their potential, and prepare for a future shaped by constant innovation across business and IT.
SEEKA Technologies (Not Seeka Limited) is a project under its parent organisation, called Fresh Futures Australia, which is an education consultant based in both Australia and Malaysia. We are developing and creating a platform that utilises A.I. to help match students and job seekers to the right opportunities relevant to them from Kindergarten up to the University, along with vocational training centres and language schools, and of course to businesses and companies who need the right candidates. Our mission is to make it easier for anyone to find, filter and apply to educational institutions and companies in a more seamless manner.
We are currently looking to hire a junior Backend AI &amp; Data Pipeline Engineer who wants to build real-world experience in backend systems, data processing, scraping, retrieval, and cloud-based infrastructure. This role is ideal for someone who already has hands-on technical experience and wants to grow further by working on meaningful engineering challenges that support Yuzee&rsquo;s intelligent matching platform. You will contribute to the systems that process data, power search and matching, and improve the efficiency, reliability, and scalability of our platform.
Below are the important details you will need to take note of:<ul><li>
English is the primary language used in the role</li><li>
This is a full-time remote/work-from-home position</li><li>
We welcome both local and international candidates</li><li>
Candidates should have a degree or proven practical experience relevant to the role</li></ul> 

Job Description 
About the role
We are looking for a Backend AI &amp; Data Pipeline Engineer to own the end-to-end data processing infrastructure that powers Yuzee's intelligent course and job matching platform. You will design and maintain scalable, event-driven pipelines that process tens of thousands of daily records, generate semantic embeddings, and feed a growing knowledge graph used for personalised career pathway recommendations.
What you'll do<ul><li>
Design and maintain three distinct processing pipelines &mdash; scheduled job ingestion, event-driven course processing, and a periodic knowledge graph builder &mdash; each with independent trigger logic and cost controls</li><li>
Generate and manage semantic embeddings via Amazon Bedrock (Titan v2), index them in MongoDB Atlas Vector Search, and calibrate similarity thresholds to ensure match accuracy</li><li>
Build and maintain a knowledge graph linking jobs, courses, skills, and industries using FP-Growth association rules and archetype-to-SOC code mapping</li><li>
Build and improve a two-stage discovery and matching API on AWS Lambda &mdash; vector retrieval first, then deep eligibility scoring with LLM re-ranking</li><li>
Right-size Fargate Spot instances and design resumable processing loops that tolerate interruption, keeping infrastructure costs under control as data volume scales</li><li>
Maintain and improve daily job scrapers across multiple sources and build institution data scrapers with robust HTML cleaning pipelines</li></ul>
What we're looking for<ul><li>
1+ years of backend engineering experience focused on data pipelines, ML infrastructure, or search systems</li><li>
Hands-on experience with AWS serverless and container services &mdash; Lambda, ECS Fargate, EventBridge, and Step Functions</li><li>
Strong Python skills &mdash; Pandas, async processing, bulk database operations, and text cleaning</li><li>
Familiarity with vector databases and semantic similarity search; MongoDB Atlas Vector Search experience is a strong plus</li><li>
Cost-conscious infrastructure mindset &mdash; you think in per-record compute costs, free tiers, Spot resilience, and right-sizing</li><li>
Ability to document and communicate complex architecture clearly to both technical and non-technical stakeholders</li></ul>
Nice to have<ul><li>
Experience with knowledge graphs or association rule mining (FP-Growth, Apriori)</li><li>
Experience using LLMs for re-ranking or eligibility assessment on top of vector retrieval results</li><li>
Background in edtech, jobtech, or recommendation/matching systems</li></ul> 

Qualifications 
Degree or existing proven experience 
Additional Information 
Use this instead:
Benefits<ul><li>
Fully remote / work-from-home role</li><li>
Flexible working hours within the team&rsquo;s expected schedule and business needs</li><li>
Opportunity to work on real backend, data, and AI infrastructure projects</li><li>
Exposure to practical engineering challenges in scraping, pipelines, retrieval, and cloud systems</li><li>
Ongoing growth and development within a fast-moving technology environment</li><li>
Opportunity to build long-term value and grow with the company based on performance, including progression and increased responsibility over time</li></ul>
A slightly more polished version:
Benefits<ul><li>
Fully remote / work-from-home position</li><li>
Some flexibility in working hours, depending on team requirements and deliverables</li><li>
Hands-on experience working on meaningful backend, data pipeline, and AI-related systems</li><li>
Opportunity to contribute to a growing platform with real product and engineering challenges</li><li>
Professional growth in a practical, fast-paced environment</li><li>
Strong potential for long-term progression based on performance, regardless of location</li></ul>
If you want it to sound more attractive for hiring, the strongest version would be to add things like:<ul><li>
ownership</li><li>
real product impact</li><li>
career growth</li><li>
direct exposure to architecture and scaling decisions</li></ul></div>

Backend AI & Data Pipeline Engineer

Emonics LLC

<div>
 
 Job Overview:
 
 
 We are seeking a Data Engineer to support data pipeline development and ETL processes.
 
 
 
 
 Responsibilities:
 
 
 
 
 Build and maintain ETL pipelines
 
 
 
 
 Optimize database performance
 
 
 
 
 Work with structured and unstructured data
 
 
 
 
 Ensure data integrity
 
 
 
 
 Requirements:
 
 
 
 
 Degree in Data Engineering or related field
 
 
 
 
 Knowledge of SQL, Python
 
 
 
 
 Familiarity with data warehousing concepts
 
</div>

Data Engineer

KDA CONSULTING INC

<div>
 KDA Consulting Inc. is seeking a highly skilled
 
 Data Engineer
 
 to design, build, and maintain scalable data pipelines and architectures supporting mission-critical programs within the Intelligence Community (IC). This role focuses on enabling efficient data processing, analytics, and decision-making by delivering reliable, high-quality data solutions in cloud environments.
 
 
 The ideal candidate will have strong experience with
 
 AWS cloud services, data engineering best practices, and big data technologies
 
 , with the ability to operate in a fast-paced, collaborative environment supporting complex mission systems.
 
 
 
 Data Pipeline Development &amp; Engineering
 
 
 
 <ul>
 <li>
 Design, develop, and maintain scalable ETL/ELT pipelines to ingest, transform, and load data from multiple sources
 </li>
 <li>
 Build and optimize data workflows supporting both batch and real-time processing
 </li>
 <li>
 Ensure high availability and reliability of data pipelines across enterprise systems
 
 
 </li>
 </ul>
 
 AWS Cloud &amp; Data Services
 
 
 
 <ul>
 <li>
 Leverage AWS services such as S3, Glue, Lambda, Redshift, EMR, and Kinesis to support data engineering solutions
 </li>
 <li>
 Design and manage cloud-based data architectures that are secure, scalable, and cost-efficient
 </li>
 <li>
 Implement best practices for data storage, partitioning, and lifecycle management
 
 
 </li>
 </ul>
 
 Data Optimization &amp; Performance
 
 
 
 <ul>
 <li>
 Optimize data storage and retrieval for performance, scalability, and cost efficiency
 </li>
 <li>
 Implement indexing, partitioning, and query optimization strategies
 </li>
 <li>
 Monitor and troubleshoot data pipeline performance issues
 
 
 </li>
 </ul>
 
 Data Quality &amp; Governance
 
 
 
 <ul>
 <li>
 Ensure data accuracy, consistency, and integrity across all data pipelines and systems
 </li>
 <li>
 Implement validation, monitoring, and error-handling mechanisms
 </li>
 <li>
 Support data governance practices and compliance with security and IC standards
 
 
 </li>
 </ul>
 
 Collaboration &amp; Integration
 
 
 
 <ul>
 <li>
 Work closely with data scientists, software engineers, and mission stakeholders to support analytics and operational needs
 </li>
 <li>
 Integrate data solutions with enterprise applications, APIs, and downstream analytics platforms
 </li>
 <li>
 Participate in Agile development processes and contribute to continuous improvement efforts
 
 
 </li>
 </ul>
 
 Requirements
 
 
 
 Active TS/ SCI W/ Polygraph Required.
 
 
 Bachelor's degree in Computer Science, Engineering, or a related technical field (or equivalent experience)
 
 
 Demonstrated experience in
 
 data engineering and pipeline development
 
 
 
 Strong experience with
 
 AWS data services
 
 (e.g., S3, Glue, Redshift, Lambda, EMR)
 
 
 Proficiency in
 
 SQL and Python
 
 for data processing and transformation
 
 
 Experience working with
 
 large-scale data sets and big data technologies
 
 
 
 Strong analytical and problem-solving skills
</div>

Sunset

<div>
 
 About Sunset
 
 
 
 Sunset is building the data layer for real-world AI training. We work with frontier labs to turn messy, multi-modal enterprise data into the highest-quality training data on the market &mdash; sourced from the hundreds of venture-backed startups we've helped wind down.
 
 
 We're a fast-growing team based in-person in Dumbo, Brooklyn. Backed by Floodgate, Afore Capital, Hustle Fund, and incredible entrepreneurs.
 
 
 
 The Role
 
 
 
 As a Data Engineer at Sunset, you'll own the pipeline that turns raw, chaotic enterprise data into the highest-quality training data on the market. One of our core technical challenges is entity resolution and de-identification across different sources and modalities. An even deeper challenge is understanding the node structures and linkages well enough to effectively reconstruct the business world this data comes from. All of this happens on sensitive data, which means security and privacy aren't a separate workstream but are built into every pipeline, system, and decision we make.
 
 
 
 What You'll Work On
 
 
 
 You'll own problems end-to-end. Some examples of what you might tackle in your first 90 days:
 
 
 <ul>
 <li>
 Designing the de-identification layer that replaces PII with stable pseudonyms while preserving every relationship across every source
 </li>
 <li>
 Building coreference resolution across Slack threads, email chains, and Linear comments so that "me," "him," and first-name mentions all resolve to the right canonical entity
 </li>
 <li>
 Hardening how we ingest, store, and process sensitive client data &mdash; from encryption and access controls to audit trails and isolation boundaries
 </li>
 <li>
 Extending our entity resolution pipeline to handle new modalities &mdash; think audio, video, design files, or embedded references inside documents
 
 
 </li>
 </ul>
 
 You Might Be a Fit If
 
 
 
 <ul>
 <li>
 You are a product minded engineer and have shipped data pipelines at scale
 </li>
 <li>
 You have strong Python and are comfortable across NER, record linkage, and coreference
 </li>
 <li>
 You take security and privacy seriously and have built systems where getting it wrong wasn't an option
 </li>
 <li>
 You want to own a hard, ambiguous problem end-to-end rather than wait for a PRD
 </li>
 <li>
 AI is deeply integrated into your workflow and life
 
 
 </li>
 </ul>
 
 This Role Might Not Be a Fit If
 
 
 
 <ul>
 <li>
 You want to work remote or hybrid &mdash; we're in-person 5 days/week in Dumbo
 </li>
 <li>
 You want to do novel ML research &mdash; this role is applied, not research
 </li>
 <li>
 You prefer long planning cycles or narrow ownership
 
 
 </li>
 </ul>
 
 Our Stack
 
 
 
 Python, Postgres, Redis, AWS. We pick tools based on the problem, not the other way around.
 
 
 
 Compensation &amp; Benefits
 
 
 
 <ul>
 <li>
 $180K&ndash;$280K base + meaningful equity
 </li>
 <li>
 100% covered medical, dental, and vision
 </li>
 <li>
 Unlimited PTO
 </li>
 <li>
 $500 in-office setup allowance
 
 
 </li>
 </ul>
 
 How We Hire
 
 
 
 <ul>
 <li>
 Intro Chat (20 min) &ndash; mutual fit and interests
 </li>
 <li>
 Technical Session (1hr) &ndash; collaborative problem-solving
 </li>
 <li>
 Onsite (2&ndash;3 hrs) &ndash; product deep dive, system design, meet the team
 </li>
 <li>
 Quick references &rarr; Offer
 
 
 </li>
 </ul>
 Compensation Range: $180K - $280K
 
 
</div>

CVS Health

<div>We&rsquo;re building a world of health around every individual &mdash; shaping a more connected, convenient and compassionate health experience. At CVS Health&reg;, you&rsquo;ll be surrounded by passionate colleagues who care deeply, innovate with purpose, hold ourselves accountable and prioritize safety and quality in everything we do. Join us and be part of something bigger &ndash; helping to simplify health care one person, one family and one community at a time.<h1 class="jobSectionHeader">
Position Summary</h1>
If you&rsquo;re eager to make a real impact in the health care industry through your own meaningful contributions, explore a role in technology with CVS Health. Our journey calls for technical innovators and data visionaries: come help us pave the way.
At CVS Health, we possess an extensive repository of healthcare data that spans over 150 million individuals, providing an unparalleled foundation for ambitious Data Engineers. In this role, you will engage with complex business challenges, harnessing modern tools and technologies to securely store, process, transform, and enrich terabyte to petabyte scale healthcare data. Your work will underpin data-driven business decisions and contribute to our mission of delivering industry-best data products / software with a customer-first mindset and team-oriented approach.
As a Data Engineer, you will be instrumental in designing, developing, and maintaining optimal data pipelines to assemble large and intricate datasets, catering to the business requirements of various CVS lines of business. Collaborating closely with teams, you will craft tools to provide actionable insights and integrate them with consumer touchpoints.
As a Data Engineer, you will:<ul><li>
Build and optimize analytical data models in BigQuery. Implement partitioning, clustering, and materialized views for performance and cost efficiency. Ensure compliance with data governance, access controls, and IAM best practices.</li><li>
Develop integrations with external systems (APIs, flat files etc.) using GCP-native or hybrid approaches. Utilize tools like Dataflow or custom Python/Java services on Cloud Functions or Cloud Run to handle transformations and ingestion logic.</li><li>
Build automated CI/CD pipeline using Cloud Build, GitHub Actions, or Jenkins for deploying data pipeline code and workflows. Set up observability using Cloud Monitoring, Cloud Logging, and Error Reporting to ensure pipeline reliability.</li></ul>
As leaders in healthcare, our analytics and engineering teams deliver innovative solutions to business problems by collaborating with cross-functional teams in a dynamic and agile environment. You will be part of a team that values collaboration and encourages innovative thinking at all levels. You will be intellectually challenged to solve problems associated with large scale complex, structured and unstructured data, that will allow you to grow your technical skills and engineering expertise.
This role may work onsite 2 days a week in one of CVS Health's core hubs in: NYC, Hartford, CT or Wellesley, MA. Full-time Remote candidates will also be considered.
Required Qualifications<ul><li>
1+ years of experience with SQL, NoSQL</li><li>
1+ years of experience with Python (or a comparable scripting language)</li><li>
1+ years of experience with Data warehouses (such as data modeling and technical architectures) and infrastructure components</li><li>
1+ years of experience with ETL/ELT, and building high-volume data pipelines</li><li>
1+ years of experience with reporting/analytic tools</li><li>
1+ years of experience with query optimization, data structures, transformation, metadata, dependency, and workload management</li><li>
1+ years of experience with Big data and cloud architecture</li><li>
1+ years of hands-on experience building modern data pipelines within a major cloud platform (GCP)</li><li>
1+ years of experience with deployment/scaling of apps on containerized environment (i.e. Kubernetes, AKS)</li><li>
1+ years of experience with real-time and streaming technology (i.e. Kafka, Spark Streaming)</li><li>
1+ year(s) of soliciting complex requirements and managing relationships with key stakeholders</li></ul>
Preferred Qualifications
<ul><li>Experience with complex systems and solving challenging analytical problems</li><li>
Strong collaboration and communication skills within and across teams</li><li>
Knowledge of data visualization and reporting</li><li>
Experience in designing and building data engineering solutions in cloud environments (preferably GCP)</li><li>
Experience with Git, CI/CD pipeline, and other DevOps principles/best practices</li><li>
Experience with bash shell scripts, UNIX utilities &amp; UNIX Commands</li><li>
Understanding of software development methodologies including waterfall and agile</li><li>
Ability to leverage multiple tools and programming languages to analyze and manipulate data sets from disparate data sources</li><li>
Knowledge of API development</li><li>
Experience with schema design and dimensional data modeling</li><li>
Google Professional Data Engineer Certification</li><li>
Knowledge of microservices and SOA</li><li>
Formal SAFe and/or agile experience. Previous healthcare experience and domain knowledge</li><li>
Experience designing, building, and maintaining data processing systems</li><li>
Experience architecting and building data warehouse and data lakes</li></ul>
Education<ul><li>
Bachelor&rsquo;s Degree or equivalent work experience in Computer Science, Information Systems, Data Engineering, Data Analytics, Machine Learning, or related field required</li><li>
Master&rsquo;s Degree preferred</li></ul>
Anticipated Weekly Hours
40
Time Type
Full time
Pay Range
The typical pay range for this role is:
$64,890.00 - $173,040.00
This pay range represents the base hourly rate or base annual full-time salary for all positions in the job grade within which this position falls. The actual base salary offer will depend on a variety of factors including experience, education, geography and other relevant factors. This position is eligible for a CVS Health bonus, commission or short-term incentive program in addition to the base pay range listed above. 
Our people fuel our future. Our teams reflect the customers, patients, members and communities we serve and we are committed to fostering a workplace where every colleague feels valued and that they belong.
Great benefits for great people
We take pride in offering a comprehensive and competitive mix of pay and benefits that reflects our commitment to our colleagues and their families. 
This full‑time position is eligible for a comprehensive benefits package designed to support the physical, emotional, and financial well‑being of colleagues and their families. The benefits for this position include medical, dental, and vision coverage, paid time off, retirement savings options, wellness programs, and other resources, based on eligibility. 
Additional details about available benefits are provided during the application process and on Benefits Moments. 
We anticipate the application window for this opening will close on: 05/11/2026
Qualified applicants with arrest or conviction records will be considered for employment in accordance with all federal, state and local laws.</div>

Radley James

<div>
 
 A fast-paced, technology-driven trading firm based in Chicago is seeking a Data Engineer to design, build, and maintain robust data pipelines and infrastructure that powers the trading platforms.
 
 
 
 
 
 
 Key Responsibilities
 
 
 <ul>
 <li>
 Design and develop scalable data pipelines for real-time and batch processing
 </li>
 <li>
 Build and maintain data infrastructure to support trading, analytics, and research
 </li>
 <li>
 Ensure data quality, integrity, and availability across systems
 </li>
 <li>
 Optimize data workflows for performance and low latency
 </li>
 <li>
 Collaborate with traders, quants, and software engineers to deliver data solutions
 </li>
 <li>
 Integrate and manage large datasets from multiple market data providers
 </li>
 <li>
 Monitor and troubleshoot data systems in a high-availability environment
 </li>
 </ul>
 
 
 Requirements
 
 
 <ul>
 <li>
 Bachelor&rsquo;s or Master&rsquo;s degree in Computer Science, Engineering, or a related field
 </li>
 <li>
 3+ years of experience in data engineering or a similar role
 </li>
 <li>
 Strong proficiency in Python
 </li>
 <li>
 Experience with data pipeline tools
 </li>
 <li>
 Solid understanding of SQL and database systems
 </li>
 <li>
 Experience working with large-scale, high-frequency datasets preferred
 </li>
 <li>
 Strong problem-solving skills and attention to detail
 </li>
 </ul>
 
 
 Nice to Have
 
 
 <ul>
 <li>
 Experience in financial services or trading environments
 </li>
 <li>
 Knowledge of market data feeds and financial instruments
 </li>
 <li>
 Experience with time-series databases
 </li>
 <li>
 Understanding of low-latency systems and distributed architectures
 </li>
 </ul>
</div>

Search 10000+ Data pipeline engineer jobs near you this April. New roles added regularly across leading industries. Apply today.

10000 data pipeline engineer jobs Near Me

Backend AI & Data Pipeline Engineer

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Data Engineer