data pipeline engineer

On-Demand Group

<div>
 <p>
  Hiring: Data Engineer | Contract-to-Hire
 </p>
 <p>
  We&rsquo;re partnering with an organization building next-generation AI and analytics platforms and looking for a Data Engineer to help drive greenfield data initiatives focused on personalization, recommendation engines, and user journey analytics.
 </p>
 <p>
  This role will focus on:
 </p>
 <p>
  &bull; Building and maintaining scalable Python-based data pipelines
 </p>
 <p>
  &bull; Supporting AI-driven recommendation and personalization systems
 </p>
 <p>
  &bull; Developing graph-based user journey models across web, mobile, and engagement platforms
 </p>
 <p>
  &bull; Designing analytics-focused data models and reporting solutions
 </p>
 <p>
  What we&rsquo;re looking for:
 </p>
 <p>
  &bull; Strong end-to-end Data Engineering experience (ingestion, transformation, modeling, analytics delivery)
 </p>
 <p>
  &bull; Experience with modern cloud data platforms
 </p>
 <p>
  &bull; Python pipeline development experience
 </p>
 <p>
  &bull; Strong dimensional/data modeling background
 </p>
 <p>
  &bull; Experience supporting scalable production data workflows
 </p>
 <p>
  &bull; Ability to thrive in fast-moving, evolving environments
 </p>
 <p>
  <br/>
 </p>
 <p>
  <br/>
 </p>
 <p>
  <em>
   The projected hourly range for this position is $75.00 to $85.00.
  </em>
 </p>
 <p>
  <em>
   On-Demand Group (ODG) provides employee benefits which includes healthcare, dental, and vision insurance. ODG is an equal opportunity employer that does not discriminate on the basis of race, color, religion, gender, sexual orientation, age, national origin, disability, or any other characteristic protected by law.
  </em>
 </p>
 <p>
  <br/>
 </p>
 <p>
  Location: Hybrid &ndash; Minneapolis, MN
 </p>
 <p>
  Type: Contract-to-Hire
 </p>
</div>


Data Engineer

Valkyrie Enterprises

<div><div><b>Data Engineer</b></div>
<div><b>Purpose:</b></div>
<ul><li>Valkyrie Enterprises has need for a <b>Data Engineer </b>in <b>Virginia Beach, VA.</b></li>
<li><b>This role is Hybrid.</b></li>
<li>The Data Engineer supports Valkyrie&rsquo;s Data &amp; AI initiatives by designing, building, and maintaining reliable enterprise data pipelines, integrations, and curated datasets that enable reporting, automation, analytics, and future AI-enabled capabilities.</li>
<li>This role is responsible for helping establish and scale the organization&rsquo;s enterprise data foundation by transforming raw operational data into trusted, structured, and reusable data assets.</li>
<li>The Data Engineer works across enterprise systems, data platforms, and operational workflows to improve data accessibility, reliability, and consistency throughout the organization.</li>
<li>The role focuses on developing scalable data pipelines, supporting enterprise reporting and planning needs, improving data quality and observability, and preparing data structures that support automation and AI initiatives.</li>
<li>This position works closely with IT Services and stakeholders across the enterprise to support the successful delivery, expansion, and ongoing improvement of enterprise data capabilities.</li>
</ul><div><b>Job Description:</b></div>
<ul><li>The Data Engineer is responsible for designing, developing, maintaining, and supporting enterprise data pipelines, integrations, and curated data solutions across Valkyrie&rsquo;s business systems and operational platforms.</li>
</ul><div>This role will:</div>
<ul><li>Develop and maintain scalable ETL/ELT pipelines across enterprise systems and operational data sources.</li>
<li>Build and maintain curated datasets that support reporting, planning, analytics, automation, and AI initiatives.</li>
<li>Design and support enterprise data warehouse models and data integration workflows.</li>
<li>Improve data quality, validation, monitoring, lineage, and operational reliability across the data platform.</li>
<li>Support ingestion and integration of new enterprise systems, business applications, and operational data sources.</li>
<li>Troubleshoot pipeline failures, performance issues, and data inconsistencies while driving root-cause resolution.</li>
<li>Support AI and automation initiatives by preparing structured, reusable, and model-ready datasets.</li>
<li>Partner with stakeholders to translate operational and reporting needs into scalable technical solutions.</li>
<li>Contribute to engineering standards, documentation, deployment practices, and operational support procedures.</li>
<li>Collaborate with technical leadership and product stakeholders on backlog priorities, sequencing, and delivery planning.</li>
</ul><div><b>Qualifications:</b></div>
<ul><li>Must have a bachelor&rsquo;s degree in Information Systems, Computer Science, Engineering, Data Analytics, or a related field and at least 5 years of relevant professional experience, or an equivalent combination of education and experience.</li>
<li>Must have hands-on experience developing and supporting ETL/ELT pipelines, data integrations, or enterprise data solutions.</li>
<li>Must demonstrate strong SQL and Python skills in production or enterprise environments.</li>
<li>Must have experience supporting modern data warehouse or cloud-based data platforms.</li>
<li>Must demonstrate experience working with structured and unstructured data sources across enterprise systems.</li>
<li>Must have experience working in a defense contractor, government, or regulated compliance environment.</li>
<li>Must demonstrate the ability to troubleshoot data quality, integration, and pipeline performance issues.</li>
<li>Must possess strong written and verbal communication skills and the ability to work effectively across technical and non-technical teams.</li>
<li>Must demonstrates foundational ability to translate business and operational requirements into technical solutions.</li>
<li>Must be capable of working independently with appropriate guidance, exercising sound judgement and contributing to multiple concurrent initiatives.</li>
</ul><div><b>Desired Qualifications</b></div>
<ul><li>Experience with cloud data platforms and services such as Snowflake and Azure, preferred.</li>
<li>Experience supporting reporting, analytics, automation, or AI-enabled business initiatives, preferred.</li>
<li>Experience working with ERP, HRIS, finance, workforce, or operational business systems, preferred.</li>
<li>Familiarity with orchestration, monitoring, observability, or data validation frameworks, preferred.</li>
<li>Experience supporting machine learning, intelligent automation, or AI-related data preparation workflows, preferred.</li>
<li>Experience with version control, testing, and deployment practices for data pipelines, preferred.</li>
</ul><div><b>Security Requirements:</b></div>
<ul><li>Must have the ability to obtain a U.S. Government security clearance.</li>
<li>Must comply with all Valkyrie Enterprises security policies and procedures.</li>
</ul> <div><b>Travel Requirements:</b></div>
<ul><li>Travel may be required on an as-needed basis to support stakeholder engagement, workshops, project coordination, or key business initiatives.</li>
<li>If position requires travel by domestic flight or access to secure federal facilities/military bases, candidate must be able to obtain (by start of position) and maintain appropriate identification credentials, such as REAL ID. (More information regarding REAL ID can be found: https://www.dhs.gov/real-id )</li>
</ul><div><b>Physical Requirements:</b></div>
<ul><li>Remaining in a stationary position, often standing, or sitting for prolonged periods.</li>
<li>Able to sit and work on a computer for long periods of time.</li>
<li>Moving about to accomplish tasks or moving from one worksite to another.</li>
<li>Light work that includes moving objects up to 20 pounds.</li>
</ul><div><b>Valkyrie strictly adheres to a policy of equal employment opportunity. This policy is based on Valkyrie&rsquo;s commitment to hire and retain qualified employees consistent with position requirements; and to seek, employ, promote and treat all employees and applicants for employment without regard to race, color, religious creed, national origin, ancestry, citizenship status, pregnancy, childbirth, physical disability, mental disability, age, military status or protected veteran status, marital status, registered domestic partner or civil union status, gender (including sex stereotyping and gender identity or expression), medical condition, genetic information or sexual orientation or other protected characteristics</b></div>
<div></div><div>Additionally, Valkyrie Enterprises provides a variety of benefits to eligible employees to support your best health, wellness, and future, to include medical/dental/vision options, company paid life and disability insurances, 401k with match, education reimbursement, as well as company paid holidays and paid time off (PTO)</div></div><div></div>

Data Engineer (Job ID:4342)

Agile Resources, Inc.

<div>
 <strong>
  Job Title:
 </strong>
 SQL Data Engineer
 <br/>
 <br/>
 <strong>
  Location:
 </strong>
 Denver, CO (Hybrid &ndash; 2+ days onsite required)
 <br/>
 <br/>
 <strong>
  Job Type:
 </strong>
 Contract (12 months, potential for full-time conversion)
 <br/>
 <br/>
 <strong>
  Overview
  <br/>
  <br/>
 </strong>
 We are seeking an experienced SQL Data Engineer to join our team in the Denver area. This role will focus on building and maintaining scalable data pipelines, integrating data across enterprise systems, and supporting high-volume data processing environments. The ideal candidate thrives in a fast-paced setting and can work cross-functionally with both technical and business stakeholders.
 <br/>
 <br/>
 <strong>
  What You&rsquo;ll Do
  <br/>
  <br/>
 </strong>
 <ul>
  <li>
   Build and maintain ETL/ELT pipelines and data integrations
  </li>
  <li>
   Develop and optimize data models and data warehouse structures
  </li>
  <li>
   Integrate data across systems, including Salesforce, APIs, and enterprise platforms
  </li>
  <li>
   Ensure data quality, accuracy, and performance across pipelines
  </li>
  <li>
   Troubleshoot data issues and pipeline failures
  </li>
  <li>
   Support CI/CD and deployment processes for data pipelines
  </li>
  <li>
   Collaborate with business and technical teams to translate requirements into scalable data solutions
   <br/>
   <br/>
  </li>
 </ul>
 <strong>
  Required Qualifications
  <br/>
  <br/>
 </strong>
 <ul>
  <li>
   4&ndash;7+ years of experience in Data Engineering
  </li>
  <li>
   Strong SQL expertise, specifically with Microsoft SQL Server
  </li>
  <li>
   Hands-on experience with ETL tools, particularly SSIS
  </li>
  <li>
   Solid understanding of data warehousing and data modeling concepts
  </li>
  <li>
   Experience working with large datasets and high-volume systems
  </li>
  <li>
   Familiarity with API-based integrations
  </li>
  <li>
   Strong troubleshooting and problem-solving abilities
   <br/>
   <br/>
  </li>
 </ul>
 <strong>
  Preferred Qualifications
  <br/>
  <br/>
 </strong>
 <ul>
  <li>
   Experience with cloud platforms such as AWS, Azure, Snowflake, or Redshift
  </li>
  <li>
   Experience working with Salesforce data or CRM integrations
  </li>
  <li>
   Exposure to MuleSoft or similar integration platforms
  </li>
  <li>
   Familiarity with CI/CD or DevOps practices for data pipelines
  </li>
  <li>
   Experience in enterprise or large-scale environments
   <br/>
   <br/>
  </li>
 </ul>
 <strong>
  Soft Skills
  <br/>
  <br/>
 </strong>
 <ul>
  <li>
   Ability to work independently in a fast-paced environment
  </li>
  <li>
   Strong communication skills with both technical and non-technical stakeholders
  </li>
  <li>
   Comfortable collaborating across multiple teams, including Data, Salesforce, and Integration teams
   <br/>
   <br/>
  </li>
 </ul>
 <strong>
  Engagement Details
  <br/>
  <br/>
 </strong>
 <ul>
  <li>
   Duration: 12 months
  </li>
  <li>
   Conversion: Potential to convert to full-time
  </li>
  <li>
   Work Model: Hybrid (Denver, CO &ndash; 2+ days onsite required)
   <br/>
   <br/>
  </li>
 </ul>
 #AB-2
 <br/>
 <br/>
 #ADCC
</div>


SQL Data Engineer

Bennett Family of Companies

<div><h1 class="jobSectionHeader"><b>Data Platform Engineer</b></h1><div><p><b>
Department: </b>Data &amp; Analytics</p><p><b><br>
Location: </b>Hybrid</p><p><b><br>
Employment Type: </b>Full-Time</p></div><br>
<p>
</p><p></p><h2 class="jobSectionHeader"><b>Position Overview</b></h2><p>
We are seeking an experienced Data Engineer to join our growing team and play a central role in shaping our data strategy. This individual will be responsible for building, optimizing, and governing our Databricks-based data platform hosted on AWS connecting disparate data sources, enabling reliable data pipelines, and ensuring that high-quality data is available to support business intelligence, analytics, and machine learning initiatives.</p><p>
</p><h2 class="jobSectionHeader"><b>Key Responsibilities</b></h2><p><b>
Data Strategy &amp; Platform Governance</b></p><ul><li>
Partner with leadership to define and execute a comprehensive data strategy aligned with business goals.</li><li>
Establish standards and best practices for data ingestion, transformation, storage, and access across the organization.</li><li>
Drive adoption of data governance frameworks including data cataloging, lineage tracking, and quality monitoring.</li><li>
Champion a culture of data-driven decision-making by enabling self-service analytics capabilities.</li></ul><p>
</p><p><b>Databricks Platform Administration</b></p><ul><li>
Architect, configure, and maintain the Databricks environment hosted on AWS (EMR, S3, IAM, VPC, etc.).</li><li>
Manage Databricks workspaces, clusters, Unity Catalog, and access controls.</li><li>
Optimize compute and storage costs through cluster sizing, auto-scaling, and lifecycle management.</li><li>
Implement and maintain CI/CD pipelines for data workflows using tools
</li></ul><p></p><p><b>Data Integration &amp; Pipeline Development</b></p><ul><li>
Design and build scalable ETL/ELT pipelines to ingest data from a variety of sources including SaaS applications, databases, APIs, and streaming platforms.</li><li>
Leverage Delta Lake, Delta Live Tables, and Databricks Workflows to build reliable, incremental data processing solutions.</li><li>
Integrate cloud-native AWS services (Glue, Kinesis, EventBridge, Lambda) with the Databricks ecosystem.</li><li>
Ensure data pipeline reliability through monitoring, alerting, and automated recovery mechanisms.</li></ul><p>
</p><p><b>Collaboration &amp; Data Enablement</b></p><ul><li>
Partner with data analysts, data scientists, and business stakeholders to understand data requirements and deliver fit-for-purpose datasets.</li><li>
Develop and maintain data documentation, including data dictionaries, runbooks, and architecture diagrams.</li><li>
Mentor junior team members on best practices in data engineering and cloud-native development.</li></ul><h2 class="jobSectionHeader"><b>
Required Qualifications</b></h2><p><b>
Technical Skills</b></p><div><div><p><br>
</p><ul><li>5+ years of hands-on data engineering experience in a cloud environment.</li></ul></div><div><p><br>
</p><ul><li>Strong proficiency with Databricks (Notebooks, Workflows, Delta Lake, Unity Catalog, SQL Warehouses).</li></ul></div><div><p><br>
</p><ul><li>Deep expertise with AWS services: S3, IAM, VPC, Glue, Redshift, RDS, Lambda, Kinesis, or EventBridge.</li></ul></div><div><p><br>
</p><ul><li>Proficiency in Python for data engineering workloads; SQL expertise required.</li></ul></div><div><p><br>
</p><ul><li>Experience designing and building data lake / lakehouse architectures.</li></ul></div><div><p><br>
</p><p>Familiarity with data transformation tooling such as dbt, Apache Spark, or similar frameworks.</p></div><div><p><br>
</p><ul><li>Knowledge of data modeling concepts (dimensional modeling, data vault, or medallion architecture).</li></ul></div><br>
</div><p>
</p><p><b>Experience &amp; Education</b></p><div><div><p><br>
</p><ul><li>Bachelor's degree in Computer Science, Information Systems, Engineering, or equivalent practical experience.</li></ul></div><div><p><br>
</p><ul><li>Proven track record of delivering end-to-end data platform solutions at scale.</li></ul></div><div><p><br>
</p><ul><li>Experience with infrastructure-as-code tools such as Terraform is a strong plus.</li></ul></div><div><p><br>
</p><ul><li>Databricks Certified Associate or Professional certification preferred.</li></ul></div><div><p><br>
</p><ul><li>AWS Certified Data Analytics or Solutions Architect certification preferred</li></ul></div><br>
</div><p><b>
We are not able to consider candidates who will require sponsorship at any point during employment.</b></p><p>
EEO/Vets/Women/Minorities/Disabled</p><p>
www.bennettig.com</p></div>

Data Platform Engineer

The Glove

<div>
 <p>
  <strong>
   Role : Data Lakehouse Architect
  </strong>
 </p>
 <p>
  <strong>
   Location:
  </strong>
  UAE (Dubai preferred) - On-site
 </p>
 <p>
  <strong>
   Experience:
  </strong>
  11 years - 16 years
 </p>
 <p>
  <strong>
   Notice period :
  </strong>
  immediate to 15 days max.
 </p>
 <p>
  <strong>
   Note: Candidate should be from the Retail or Ecommerce domain only
  </strong>
 </p>
 <p>
  <br/>
 </p>
 <p>
  <strong>
   ROLE OVERVIEW
  </strong>
 </p>
 <p>
  We are looking for a hands-on
  <strong>
   Lead Data Engineer
  </strong>
  with 11+ years experience who has built and operated production-grade data lakehouse platforms.
 </p>
 <p>
  This person will work directly under the Data Lakehouse Architect and own the end-to-end engineering execution across ingestion, transformation, orchestration, governance, and consumption layers. This is a technical delivery role requiring deep engineering skills and the ability to work across all layers of the lakehouse.
 </p>
 <p>
  <br/>
 </p>
 <p>
  <strong>
   KEY RESPONSIBILITIES
  </strong>
 </p>
 <ul>
  <li>
   Build and operate end-to-end ingestion pipelines from 20+ heterogeneous source systems including Oracle Retail, WMS, TMS, Loyalty platforms, and third-party APIs into the Bronze layer
  </li>
  <li>
   Implement CDC pipelines for real-time and near-real-time data capture across relational and NoSQL sources
  </li>
  <li>
   Design and build Silver and Gold transformation layers including data cleansing, enrichment, SCD Type 1 and 2, and complex business rule application
  </li>
  <li>
   Develop and maintain orchestration workflows with automated retry, failure alerting, and SLA tracking
  </li>
  <li>
   Implement data quality checks, validation rules, and reprocessing/backfill capabilities
  </li>
  <li>
   Enforce security policies, PII masking, row-level and column-level access control as defined by the Architect
  </li>
  <li>
   Enable consumption layers for Tableau, direct SQL users, and downstream API integrations
  </li>
  <li>
   Support historical data migration from legacy cloud data warehouse into the lakehouse
  </li>
  <li>
   Maintain pipeline documentation, source-to-target mappings, and data dictionaries
  </li>
  <li>
   Build clean, well-structured data pipelines that support AI/ML feature engineering and model training workflows
  </li>
 </ul>
 <p>
  <br/>
 </p>
 <p>
  <strong>
   REQUIRED SKILLS AND EXPERIENCE
  </strong>
 </p>
 <ul>
  <li>
   12+ years in data engineering with hands-on experience across ingestion, transformation, orchestration, and governance layers in production lakehouse environments
  </li>
  <li>
   Proficient in Python and PySpark for pipeline development and custom connector building
  </li>
  <li>
   Hands-on experience with cloud data services (e.g. S3/Blob Storage, Glue, DMS, Kinesis/Event Hubs, EMR or equivalent)
  </li>
  <li>
   Strong CDC implementation experience using DMS, Debezium, or equivalent across Oracle, RDS, and NoSQL sources
  </li>
  <li>
   Experienced with Delta Lake/Apache Iceberg ACID transactions, merge/upsert operations, and partition management at scale
  </li>
  <li>
   Strong hands-on experience with dbt for SQL-based transformation layers
  </li>
  <li>
   Proficient with orchestration tools such as Apache Airflow (MWAA), Dagster, or Step Functions
  </li>
  <li>
   Experienced with data quality frameworks such as Great Expectations, Deequ, or dbt tests
  </li>
  <li>
   Hands-on with security implementation: IAM policies, PII masking, column-level and row-level access control
  </li>
  <li>
   Strong SQL skills and dimensional modeling (star schema, snowflake schema) for BI consumption layers
  </li>
  <li>
   Retail or e-commerce domain experience (Oracle Retail, Magento, Shopify, WMS, TMS) is a strong advantage
  </li>
  <li>
   Familiarity with AI/ML pipeline requirements including feature store design, data preparation for model training, and vector database integration
  </li>
 </ul>
 <p>
  <br/>
 </p>
 <p>
  <strong>
   PREFERRED / NICE-TO-HAVE PLATFORMS
  </strong>
 </p>
 <ul>
  <li>
   AWS (Lake Formation, Glue, Redshift, EMR, SageMaker)
  </li>
  <li>
   Databricks
  </li>
  <li>
   Snowflake
  </li>
  <li>
   Microsoft Fabric
  </li>
  <li>
   Google BigQuery
  </li>
  <li>
   Dataiku
  </li>
  <li>
   Apache Iceberg / Hudi
  </li>
 </ul>
 <p>
  <br/>
 </p>
 <p>
  <strong>
   EDUCATION
  </strong>
 </p>
 <ul>
  <li>
   Bachelor&rsquo;s or master&rsquo;s degree in computer science, Information Systems, or a related field
  </li>
  <li>
   Cloud Data Analytics or Solutions Architect certification from a major cloud provider (AWS, Azure, or GCP) is a strong advantage
  </li>
  <li>
   Databricks Certified Data Engineer or Snowflake SnowPro certifications are a plus
  </li>
 </ul>
 <p>
  <br/>
 </p>
 <p>
  <strong>
   Interested candidates share their resume at uma.jangra@glovetalent.com !!
  </strong>
 </p>
</div>


Data Lakehouse Architect

QuantumGate

<div>
 <p>
  QuantumGate is dedicated to developing and commercializing cutting-edge post-quantum cryptographic solutions. Our mission is to safeguard enterprise digital environments through innovative protocols and applications that address the evolving challenges of the post-quantum era.
 </p>
 <p>
  <br/>
 </p>
 <p>
  We are seeking a talented and motivated Data Engineer to join our fast-growing ML-team. The ideal candidate has strong experience with distributed data processing, NoSQL databases, and modern data lake technologies. You will play a key role in designing and maintaining scalable data pipelines that power business intelligence, analytics, and machine learning initiatives.
 </p>
 <p>
  <br/>
 </p>
 <p>
  <strong>
   Key Responsibilities
  </strong>
 </p>
 <p>
  <br/>
 </p>
 <ul>
  <li>
   Design, develop, and maintain scalable ETL/ELT pipelines using Apache Spark and/or similar solutions.
  </li>
  <li>
   Manage and optimize large datasets stored in Apache Iceberg tables.
  </li>
  <li>
   Design, build, and optimize data infrastructure.
  </li>
  <li>
   Integrate data from various sources including MongoDB, APIs, and flat files.
  </li>
  <li>
   Optimize data processing workflows for batch and real-time analytics.
  </li>
  <li>
   Implement and enforce data governance, data quality, and security best practices.
  </li>
  <li>
   Troubleshoot and resolve data-related issues in dev/production environments.
  </li>
  <li>
   Work closely with data scientists, analysts, and product teams to understand data needs and deliver efficient solutions.
  </li>
  <li>
   Monitor pipeline performance and troubleshoot data quality issues.
  </li>
 </ul>
 <p>
  <br/>
 </p>
 <p>
  <strong>
   Qualifications
  </strong>
 </p>
 <p>
  <br/>
 </p>
 <ul>
  <li>
   Bachelor's degree in Computer Science, Engineering, or related field.
  </li>
  <li>
   3+ years of experience in data engineering or related roles.
  </li>
  <li>
   Strong hands-on experience with:
  </li>
  <li>
   o Apache Spark (PySpark, Scala, or Java)
  </li>
  <li>
   o MongoDB (data modeling, aggregation framework, replication)
  </li>
  <li>
   o Apache Iceberg or other data lake table formats (Delta Lake, Hudi)
  </li>
  <li>
   Proficient in SQL and at least one programming language (Python, Scala, or Java).
  </li>
  <li>
   Experience with data pipeline orchestration tools (e.g., Airflow).
  </li>
  <li>
   Understanding of data warehousing concepts (star schema, partitioning).
  </li>
  <li>
   Familiarity with cloud platforms such as AWS or Azure.
  </li>
 </ul>
 <p>
  <br/>
 </p>
 <p>
  <strong>
   Nice to have qualifications
  </strong>
 </p>
 <p>
  <br/>
 </p>
 <ul>
  <li>
   Experience with stream processing (e.g., Kafka, Flink).
  </li>
  <li>
   Experience with Kubernetes for containerized data workloads.
  </li>
  <li>
   Familiarity with CI/CD practices and infrastructure as a code (e.g., Terraform)
  </li>
 </ul>
 <p>
  <br/>
 </p>
 <p>
  Join us to drive innovation and shape the future of technology!
 </p>
</div>


Senior Data Engineer

Tata Consultancy Services (TCS)

<div></div><div><div><div><div><div>The Agentic AI Data Engineer is a hands-on role focused on building and maintaining the data pipelines and infrastructure that fuel AI agent systems. Within TCS&rsquo;s AI &amp; Data group (Americas), you will be the builder who turns data architecture plans into reality, ensuring that AI models and agents have continuous access to high-quality, timely data. This client-facing consulting role involves hybrid work from client site as needed for deployment. You&rsquo;ll work across wide array of business functions within Retail. By combining expertise in data ingestion, transformation, and integration with knowledge of AI data needs, you will play a critical part in enabling AI agents to perform reliably and accurately in production.</div><br>
<div><br>
</div><div>What You Would Be Doing:
</div><div>Build Data Ingestion Pipelines: Develop robust pipelines to extract data from various sources (databases, APIs, flat files, streaming sources) relevant to the AI solution.
</div><div>Data Transformation &amp; Processing: Implement transformation and cleaning steps on raw data to make it usable for AI, ensuring efficiency and scalability.
</div><div>Loading Data to Storage/Indices: Set up processes to load processed data into target storage systems that AI agents or models will use.
</div><div>Real-Time Data Feeds: Implement streaming or incremental update pipelines when AI systems require real-time or frequently updated data.
</div><div>Pipeline Automation &amp; Scheduling: Use orchestrators or schedulers to automate the data workflows.
</div><div>Data Integration &amp; API Development: Develop and maintain integration components for real-time data fetching.
</div><div>Collaborate on RAG/Knowledge Base Updates: Work closely with AI Data Architects on implementing RAG updates.
</div><div>Testing and Validation of Data Pipelines: Develop tests and monitoring for your data pipelines.
</div><div>Optimize Pipeline Performance: Profile and optimize data pipelines for speed and resource usage.
</div><div>Documentation and Handover: Document pipeline processes, configurations, and dependencies clearly.
</div><div>Industry-Specific Data Handling: Adapt data engineering to specific domain needs.
</div><div>Collaboration &amp; Agile Implementation: Work as part of an agile product team, collaborating with data architects, AI engineers, and others.
</div><div>Maintain and Evolve Pipelines: Monitor pipelines and handle maintenance post go-live.</div><br>
<div><br>
</div><div>What Skills Are Expected:
</div><div>Programming &amp; Scripting: Strong programming skills, especially in Python, and experience with other languages like SQL.
</div><div>Data Pipeline Development: Practical experience building data pipelines end-to-end.
</div><div>Database and SQL Skills: Proficiency in writing and optimizing SQL queries.
</div><div>Big Data &amp; Distributed Processing: Experience with big data technologies like Apache Spark.
</div><div>Streaming Data Experience: Familiarity with streaming frameworks and tools like Kafka.
</div><div>API Integration and Web Services: Ability to interact with web APIs for data ingestion or extraction.
</div><div>Data Formats and Parsing: Strong understanding of data formats and ability to parse JSON, XML, or custom text formats.
</div><div>DevOps for Data Pipelines: Basic DevOps skills, including using Git for version control and CI/CD pipelines.
</div><div>Problem Solving &amp; Debugging: Strong ability to troubleshoot data issues.
</div><div>Data Quality Focus: Attentiveness to data quality and skills in implementing checks and validating outputs.
</div><div>Collaboration &amp; Commun ication: Good communication skills to work with the team and clients.
</div><div>Time Management &amp; Flexibility: Ability to handle multiple tasks and prioritize effectively.
</div><div>Domain Data Understanding: Aptitude to learn domain context from data.
</div><div>Security &amp; Privacy Business Units: Understanding of handling sensitive data securely in pipelines.
</div><div>Continuous Learning: Willingness to learn new tools or frameworks as needed.</div><br>
<div><br>
</div><div>Key Technology Capabilities:
</div><div>ETL / Data Integration Tools: Experience with tools such as Apache Airflow, Informatica PowerCenter, or cloud-based ones like Azure Data Factory.
</div><div>Big Data Processing: Proficiency in Apache Spark and knowledge of Hadoop HDFS.
</div><div>SQL &amp; Databases: Strong practical SQL skills and familiarity with relational database systems.
</div><div>NoSQL and Other Data Stores: Knowledge of specific systems like MongoDB or Cassandra.
</div><div>Stream Processing: Hands-on usage of Apache Kafka and understanding of consumer group mechanics.
</div><div>Cloud Storage &amp; Compute: Familiarity with cloud storage services like Amazon S3 and cloud compute for ETL.
</div><div>APIs &amp; Web Services: Experience building or using connectors to RESTful APIs.
</div><div>File Formats &amp; Data Serialization: Understanding of various file formats and ability to convert between them.
</div><div>Operating Systems &amp; Scripting: Comfortable with Linux shell and basic shell scripting.
</div><div>Version Control &amp; CI/CD: Using Git for source control and setting up CI pipelines for data projects.
</div><div>Monitoring &amp; Logging Tools: Utilizing monitoring tools for data workflows.
</div><div>Data Visualization/Verification: Basics of tools like Excel or Python&rsquo;s Jupyter notebooks for data sanity checks.
</div><div>Security &amp; Networking: Understanding network configurations for data transfer.
</div><div>Testing Frameworks: Familiarity with PyTest or unittest for writing tests for data transformations.
</div><div>Collaboration Tools: Experience with tools like JIRA and documentation tools.
</div><div>AI/ML Familiarity: Bonus if you understand some AI/ML fundamentals</div><br>
<div><br>
</div><div>Salary Range: 124300-168100 a year
</div><div>#LI-MM6</div><br>
<div><br>
</div></div></div></div><div><div><div><b>Location</b></div>
<div>Seattle, WA</div>
</div><div><div><b>Job Function</b></div>
<div>CONSULTANCY</div>
</div><div><div><b>Role</b></div>
<div>Engineer</div>
</div><div><div><b>Job Id</b></div>
<div>415327</div>
</div><div><div><b>Desired Skills</b></div>
<div>Artificial Intelligence</div>
</div><div><div><b>Salary Range</b></div>
<div>$124,300-$168,100 a year</div></div></div></div><div></div>

Senior Engineer, AI Data Management

Search 8373+ Data pipeline engineer jobs near you this June. New roles added regularly across leading industries. Apply today.