data engineer

Q2Impact

<div>Project Summary: This contract supports the Joint Staff Strategy, Plans, and Policy Directorate (J5), through its Deputy Directorate for Counter Threats and International Cooperation (DDCTIC), in meeting the minimum statutory requirements of Public Law 115-68, the Women, Peace, and Security (WPS) Act of 2017. The contract delivers WPS training and education for the Joint Force, sociocultural analysis, and data collection for annual statutory reporting. The Data Engineer supports the J5 data infrastructure underpinning all of these functions, building and maintaining the data pipelines, repositories, and AI-enabled tools that allow analysts and senior decision-makers across the directorate to access mission-critical information. This position is full-time and is contingent upon award. 
Position Summary: The Data Engineer designs, builds, and maintains the data infrastructure supporting J5 WPS statutory reporting and analytical functions. The role is responsible for identifying and integrating mission-critical data sources from across the DoW and interagency partners, managing curated datasets within the J5 Data Catalog, and developing and maintaining WPS AI agents. The Data Engineer works closely with WPS analysts and Joint Staff leadership to ensure data assets are standardized, documented, and decision-ready. 
General Duties and Responsibilities:<ul><li>
Identify, acquire, and engineer access to mission-critical data sources from across the DoW and interagency partners.</li><li>
Design, build, and maintain automated data pipelines to ingest, clean, and integrate data into a centralized repository.</li><li>
Publish and manage curated datasets as discoverable assets within the J5 Data Catalog, ensuring documentation, standardization, and accessibility for analysts and decision-makers.</li><li>
Develop and maintain WPS AI agents in support of J5 analytical and reporting requirements.</li><li>
Support data collection and collation for NATO reporting requirements and annual WPS statutory reporting.</li><li>
Produce interactive dashboards and data visualizations for senior military leadership, including Flag Officers, General Officers, and SES members.</li><li>
Prepare and present technical materials to senior leadership as required.</li><li>
Contribute to Monthly Status Reports and other deliverables as required by the PWS.</li></ul> 
Qualifications:<ul><li>
Proven, hands-on expertise in data engineering, including building and maintaining secure data pipelines and managing data warehouse solutions in a cloud environment (e.g., AWS, Azure).</li><li>
Expert-level proficiency in data programming languages (e.g., Python, R).</li><li>
Experience developing interactive dashboards for senior leadership using visualization tools (e.g., Tableau, Power BI).</li><li>
Ability to prepare and present materials to Flag Officers, General Officers, or Senior Executive Service (SES) members.</li><li>
Familiarity with DoW data governance frameworks and interagency data-sharing protocols preferred.</li><li>
Experience with AI/ML tool development in a government or national security context preferred.</li><li>
Minimum of three (3) years of experience leading data-centric programs, with demonstrated expertise in designing and implementing enterprise data strategy, architecture, and governance.</li><li>
Minimum of five (5) years of function-related experience at the operational or strategic level, with a demonstrated understanding of DoW organization and operations.</li><li>
Experience includes joint doctrine, strategic planning, the operational planning process, and DoD security cooperation.</li><li>
Proficiency in Microsoft Office suite required.</li><li>
Master's degree or higher in Data Science, Computer Science, Statistics, or a related quantitative field.</li></ul>
Location and Travel: Pentagon / National Capital Region (NCR). Telework considered only on an exception basis. Travel in support of contract requirements as directed by the COR. 
Requirements 
Required Security Clearance: U.S. Citizenship required. Must possess and maintain a final Top Secret (TS) security clearance, fully adjudicated with an indication of &quot;determined eligibility of DCID 6/4&quot; in the Joint Personnel Adjudication System (JPAS).</div>

Data Engineer

Pyx Health

<div><div>Pyx Health is looking for a motivated and technically solid Healthcare Data Engineer to join our growing Data &amp; Analytics team. In this role, you will contribute to building and maintaining our data infrastructure on Azure, work alongside senior engineers to develop reliable data pipelines, and collaborate with analytics teams to engineer data foundations that support our healthcare solutions. You will work with Databricks, Airflow, Python, and SQL to implement and improve data workflows in a collaborative, high-growth environment. Only candidates residing in the USA may apply.

KEY RESPONSIBILITIES
</div>
Pipeline Development &amp; Maintenance
<ul><li> Build and maintain data pipelines for ingesting, transforming, and cleaning healthcare data in the Azure cloud using Databricks, PySpark, and Delta Lake</li></ul>
<ul><li> Implement pipeline logic from defined specifications, with guidance from senior engineers on architectural decisions</li></ul>
<ul><li> Build reusable testing frameworks to ensure pipeline reliability</li></ul>
<ul><li> Monitor pipelines for failures and performance issues, escalating complex problems appropriately</li></ul>

Orchestration &amp; Automation
<ul><li> Develop and maintain Airflow DAGs with appropriate error handling and retry logic</li></ul>
<ul><li> Support deployment and configuration of pipelines via Astronomer on Azure using the Astro CLI</li></ul>
<ul><li> Contribute to improving pipeline reliability and reducing manual intervention</li></ul>

Data Modeling &amp; Storage
<ul><li> Implement data models for efficient storage and retrieval using Delta Lake, including merge/upsert patterns</li></ul>
<ul><li> Design and optimize Delta Lake tables for analytic workloads</li></ul>
<ul><li> Leverage Unity Catalog for data governance, organization, and security</li></ul>
<ul><li> Apply Change Data Capture (CDC) patterns under the direction of senior team members</li></ul>
<ul><li> Write and optimize T-SQL and Databricks SQL scripts, stored procedures, and ETL support</li></ul>

Azure Infrastructure
<ul><li> Work within the Azure ecosystem including ADLS Gen2, Key Vaults, Logic Apps, and Azure DevOps</li></ul>
<ul><li> Follow established security and scalability standards when building data infrastructure</li></ul>
<ul><li> Support infrastructure tasks with guidance from senior engineers on architecture</li></ul>

Data Quality &amp; Monitoring
<ul><li> Ensure scripts and datasets are well-documented to support enterprise data governance</li></ul>
<ul><li> Design, implement and continuously improve automated data quality monitoring, reconciliation, and alerting to ensure reliable downstream reporting</li></ul>
<ul><li> Troubleshoot and resolve pipeline failures, documenting root causes and resolutions</li></ul>

Collaboration &amp; Documentation
<ul><li> Work closely with data scientists, analysts, and business stakeholders to understand reporting requirements, troubleshoot issues, and improve data assets</li></ul>
<ul><li> Participate in code reviews, incorporating feedback to improve code quality</li></ul>
<ul><li> Document pipelines, processes, and implementation decisions clearly and consistently</li></ul>
<ul><li> Stay current with data engineering technologies, particularly in the healthcare space</li></ul><div>

QUALIFICATIONS
</div>
<ul><li> 2&ndash;4 years of experience as a Data Engineer or in a closely related data role</li></ul>
<ul><li> Hands-on experience with Azure cloud services (ADLS, Databricks, or similar)</li></ul>
<ul><li> Working knowledge of SQL for scripting and data modeling including T-SQL, Spark, and Databricks SQL</li></ul>
<ul><li> Ability to contribute to technical projects with moderate oversight</li></ul>
<ul><li> Strong communication skills &mdash; comfortable asking questions and providing updates to cross-functional partners</li></ul>
<ul><li> Familiarity with CI/CD concepts, Git, and version control workflows</li></ul>
<ul><li> Solid problem-solving skills with a systematic approach to debugging</li></ul>
<ul><li> Familiarity with healthcare data standards and regulations (HIPAA, HL7, etc.) is a plus</li></ul>
<ul><li> Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field (or equivalent work experience)</li></ul>
<div>
MUST HAVES
</div>
<ul><li> Databricks SQL (Mid): Delta Lake fundamentals, basic merge/upsert patterns, familiarity with CDC concepts.</li></ul>
<ul><li> Python (Mid): Pipeline logic, data transformation, SQL scripting; some PySpark/Spark DataFrame experience.</li></ul>
<ul><li> Databricks Spark Notebooks (Mid): Notebook-based development, basic cluster usage, Delta table operations.</li></ul>
<ul><li> Airflow Python Development (Mid): Ability to write and maintain DAGs; understanding of error handling and retry patterns.</li></ul>
<ul><li> Airflow Astro Configuration (Foundational): Familiarity with Astronomer or willingness to learn; basic Astro CLI usage.</li></ul>
<ul><li> Azure Ecosystem (Foundational): Working knowledge of ADLS Gen2, Key Vaults, and Azure DevOps.</li></ul>
<ul><li> T-SQL (Foundational&ndash;Mid): Basic stored procedures, SQL Server querying, and data manipulation.</li></ul> 


 NICE TO HAVES
<ul><li> Azure Data Factory (Foundational): Basic familiarity with pipeline authoring and triggers.</li></ul>
<ul><li> Git + Azure DevOps CI/CD (Foundational): Experience with branching, pull requests, peer review processes, and version control best practices as well as deploying production data pipelines using CI/CD workflows.</li></ul></div>

Kforce Inc

<div>
 
 Responsibilities
 
 
 
 Kforce has a client in Boston, MA that is seeking a Data Engineer. Position Overview: We are seeking a Senior Data Engineer to join a core engineering squad within our enterprise technology division. This hands-on role requires deep technical knowledge in enterprise data architecture and a proven history of processing highly sophisticated, large-scale transaction datasets. You will spearhead the creation of robust data pipelines, incorporate advanced automated analytics and document processing tools, and link multi-cloud data environments. This position operates on-site at our regional office on a hybrid schedule. Core Responsibilities:
 
 
 <ul>
 <li>
 Unified Framework Architecture: Oversee the design, construction, and modernization of our centralized data network, establishing repeatable methodologies for information access, governance, and delivery across internal departments
 </li>
 <li>
 Modern Lakehouse Formats: Direct the evolution of our lakehouse storage framework, overseeing ingestion, data delivery, replication, stream management, speed tuning, and standard lifecycle procedures for both reporting and production environments
 </li>
 <li>
 Distributed Computing: Formulate the system processing strategy across stateful, stateless, and clustered computing environments
 </li>
 <li>
 Real-Time Streaming: Architect and promote message-based, real-time messaging patterns for immediate information ingestion and cross-system connection
 </li>
 <li>
 Strategic Planning: Convert business needs into clear technical milestones
 </li>
 <li>
 Cross-Functional Leadership: Direct technical strategy across critical platform domains.
 
 
 </li>
 </ul>
 Engineering Rigor: Standardize operational frameworks for data verification, tracing, compliance, and monitoring
 
 
 <ul>
 <li>
 Technical Mentorship: Guide developing engineers, elevating development standards regarding design validation, code quality, and technical accountability
 
 
 </li>
 </ul>
 
 Requirements
 
 
 
 <ul>
 <li>
 Education: Technical degree in a relevant field like Software Engineering, Computer Science, or equivalent practical experience
 </li>
 <li>
 Experience: 4+ years navigating enterprise database design, clustered computing, or infrastructure orchestration, with clear tenure operating in an influential technical capacity
 </li>
 <li>
 Mindset: An infrastructure-first mindset with a history of delivering high-capacity storage ecosystems focused on developer productivity and data tracking
 </li>
 <li>
 Table Formats: Comprehensive hands-on deployment of modern open-source table formats (such as Iceberg or Delta), including storage partition strategies, metadata oversight, and performance optimization
 </li>
 <li>
 Compute Engines: Deep knowledge of clustered compute engines, workflow orchestration tools, system recovery, and computational tuning
 </li>
 <li>
 Messaging: Proven implementation of asynchronous communication models using modern broker networks and messaging flows
 </li>
 <li>
 Analytics: Experience optimizing data ecosystems for production analytics, including specialized pipeline building and model deployment workflows
 </li>
 <li>
 Languages: Professional fluency in scripting languages (such as Python and SQL); Familiarity with object-oriented languages (like Java or Scala) and enterprise processing tools is a strong plus
 </li>
 <li>
 DevOps Tools: Competency working with cloud-managed containerization tools, infrastructure-as-code automation, and centralized system monitoring software
 </li>
 <li>
 Leadership: Ability to direct high-stakes architectural choices, influence engineering directions, and bridge communication gaps between technical teams and business directors
 </li>
 <li>
 Communication: Excellent presentation skills, with a capability to explain complicated cloud infrastructure concepts to non-technical partners
 
 
 </li>
 </ul>
 The pay range is the lowest to highest compensation we reasonably in good faith believe we would pay at posting for this role. We may ultimately pay more or less than this range. Employee pay is based on factors like relevant education, qualifications, certifications, experience, skills, seniority, location, performance, union contract and business needs. This range may be modified in the future.
 
 
 We offer comprehensive benefits including medical/dental/vision insurance, HSA, FSA, 401(k), and life, disability &amp; ADD insurance to eligible employees. Salaried personnel receive paid time off. Hourly employees are not eligible for paid time off unless required by law. Hourly employees on a Service Contract Act project are eligible for paid sick leave.
 
 
 Note: Pay is not considered compensation until it is earned, vested and determinable. The amount and availability of any compensation remains in Kforce's sole discretion unless and until paid and may be modified in its discretion consistent with the law.
 
 
 This job is not eligible for bonuses, incentives or commissions.
 
 
 Kforce is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, pregnancy, sexual orientation, gender identity, national origin, age, protected veteran status, or disability status.
 
 
 By clicking &ldquo;Apply Today&rdquo; you agree to receive calls, AI-generated calls, text messages or emails from Kforce and its affiliates, and service providers. Note that if you choose to communicate with Kforce via text messaging the frequency may vary, and message and data rates may apply. Carriers are not liable for delayed or undelivered messages. You will always have the right to cease communicating via text by using key words such as STOP.
</div>

Nawy Real Estate

<div>Nawy Proptech is in search of a highly motivated and talented Data Engineer to become a valuable addition to our dynamic team. As a Data Engineer at Nawy, you will play a key role in data management, ensuring data quality, integrity, and modeling. We are seeking an individual with a strong technical background, hands-on experience with data processing technologies, and a passion for building robust and efficient data solutions.<ul><li>
Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and design appropriate solutions.</li><li>
Work closely with IT and infrastructure teams to deploy and maintain data storage and processing systems, including databases, data warehouses, and cloud infrastructure.</li><li>
Design, build, and maintain robust, scalable data pipelines to ingest, process, and transform data from various sources, including internal databases and external APIs.</li><li>
Optimize data pipelines for performance, scalability, and reliability, ensuring timely and accurate delivery of data.</li><li>
Implement data validation and quality checks to identify and rectify any issues in data pipelines, ensuring data integrity.</li><li>
Explore and evaluate new technologies, tools, and frameworks to improve data engineering processes and capabilities.</li><li>
Perform data modeling and schema design to support analytical and reporting requirements.</li><li>
Design and build data marts and business layers to facilitate efficient data access and analysis.</li></ul> 

Requirements 
<ul><li>
Bachelor&rsquo;s degree in Computer Science, Information Systems, or other related technical field or equivalent work.</li><li>
Proven experience as a Data engineer or similar role, with at least 1 years of experience.</li><li>
Advanced SQL knowledge/experience - complex queries and query optimization.</li><li>
Proficiency in programming languages such as Python.</li><li>
Experienced in the development of data warehouses and proficient in data modeling.</li><li>
Familiarity with cloud platforms such as AWS, Azure, or Google Cloud Platform.</li><li>
Experience in building data streams using technologies such as Kafka, Flink, or Spark Streaming is considered a plus.</li><li>
Strong problem-solving skills and attention to detail.</li><li>
The ability and willingness to learn new technologies on the job.</li><li>
Self driven, highly motivated, fast learner, ambitious and creative.</li></ul></div>

VTG Defense

<div>Overview:
<div>
VTG is looking for a Data Engineer in Chantilly, VA</div> 
<div></div>
Responsibilities:
<div>
Seeking a Cloud Data Engineer to design, develop, and optimize scalable cloud-based data platforms that support analytics, AI/ML, and business intelligence initiatives. This role focuses on cloud architecture, data engineering, pipeline development, and modernization efforts while collaborating with cross-functional teams to deliver secure, high-performance data solutions in a mission-driven environment.</div>
Qualifications:
<div>Required Skills and Qualifications:</div><ul><li>
3+ years of experience designing cloud-native data platforms in AWS</li><li>
2+ years of experience designing scalable data lake, warehouse, and analytics solutions</li><li>
2+ years of experience with cloud migration and modernization efforts</li><li>
2+ years of experience with the data pipelines, including data acquisition, data prep, and database architecture</li><li>
Experience with data engineering projects supporting data science, AI/ML implementations, and Business Intelligence dashboards</li><li>
Experience with non-relational and semi-relational databases</li><li>
Experience creating software for retrieving, parsing, and processing structured and unstructured data</li><li>
Experience with team lead functions, creating solutions within a collaborative and cross-functional team environment</li><li>
TS/SCI clearance with a polygraph</li><li>
Bachelor's degree in CS</li></ul><div>
</div><div>
Preferred Skills and Qualifications:</div><ul><li>
Experience with Pyspark and Spark ecosystems such as Databricks and EMR</li><li>
Experience evaluating data analytic or science technologies and platforms</li><li>
Experience going through an A&amp;A process or leading modifications</li><li>
Experience with Cloud Architecture and evaluating and selecting cloud analytics architectures</li><li>
Experience with Data Marts and Data Warehouses and creating, managing, and troubleshooting complex operational data flows</li><li>
Experience with Source code management and integration such as GitHub</li><li>
Experience with designing, developing, deploying, or testing in Amazon Web Services (AWS)</li><li>
Experience with scripting languages such as Python and SQL</li><li>
Experience with APIs development</li><li>
AWS architecture Certifications such as Solution Architect Certification</li></ul></div>

Vista Technology Services

VISTA Technology Services, Inc. (VISTA) is currently seeking a Data Engineer to support a U.S. Intelligence Agency in Springfield, VA.
 
This position is contingent upon contract award.
 
The Data Engineer will provide design, implementation, and management of databases and data pipelines to support operations and will ensure data integrity, availability, and security for all technical security data systems.
 
Responsibilities: 
<ul>
<li>Design and implement database structures for data collection and analysis</li>
<li>Designing, building, and maintaining scalable data pipelines and architectures.</li>
<li>Developing and implementing data transformations and algorithms.</li>
<li>Implement data storage solutions for long-term analysis of technical security data</li>
<li>Create data models to support analytical requirements operations</li>
<li>Ensure data security and compliance with applicable regulations</li>
<li>Coordinate with software development teams to ensure proper data integration</li>
<li>Provide technical leadership for junior data engineers</li>
</ul>
Basic Qualifications: 
<ul>
<li>Bachelor's degree in Computer Science, Data Science, or related field</li>
<li>Minimum of 8 years of experience in data engineering or database administration</li>
<li>Advanced proficiency in programming languages like Python, SQL, Java, and Scala</li>
<li>Expertise in ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) processes</li>
<li>In-depth knowledge of relational and NoSQL databases</li>
<li>Hands-on experience with big data frameworks</li>
<li>Proficiency with cloud computing services</li>
<li>Strong understanding of data modeling and data warehousing</li>
<li>Understanding of data security requirements for classified environments</li>
<li>Active TS/SCI, with a willingness to take a polygraph test</li>
</ul>
Desired Qualifications: 
<ul>
<li>Experience with MLOps and supporting machine learning workflows.</li>
<li>Knowledge of emerging data architecture patterns like Data Mesh or Data Fabric.</li>
<li>Expertise in real-time data streaming and processing technologies.</li>
<li>Advanced certifications in cloud data services (e.g., AWS Certified Data Analytics - Specialty).</li>
<li>Proficiency with Infrastructure as Code (aC) tools like Terraform or CloudFormation.</li>
<li>A strong business acumen and the ability to connect data initiatives to business outcomes.</li>
</ul>
Salary at VISTA is determined by various factors, including but not limited to location, the individual’s particular combination of education, knowledge, skills, competencies, and experience, as well as contract-specific affordability and organizational requirements. The projected compensation range for this position is $150,000.00 to $195,000.00 (annualized USD) based on years of experience. The estimate displayed represents the typical salary range for this position and is just one component of VISTA's total compensation package for employees. We also offer a competitive leave package and a generous benefit plan which includes medical, dental, vision, life and disability insurance, tuition assistance, and an employer matching 401(k) plan.
 
Applicants selected for this position will be subject to a government security investigation and must meet eligibility requirements for access to classified information.
 
For this position, VISTA will consider only applicants with an active DoD security clearance.
 
To learn more about VISTA and to apply for this position, please visit our website at www.vistatsi.com.
 
VISTA is an Equal Opportunity Employer/Disability/Veterans Employer

Optomi

<div>
 
 Optomi, in partnership with a leading electric and water utility services company, is seeking a Data Engineer to join their team! This role is critical to the ongoing success of the project as the team is nearing crucial deadlines. The position requires a strong expertise in Python, SQL, Databricks, and Spark technologies to ensure seamless data processing and ETL job execution using Informatica.
 
 
 
 
 
 
 Job Qualifications:
 
 
 <ul>
 <li>
 Extensive experience with Azure and Google Cloud platforms.
 </li>
 <li>
 Proficiency in Python and SQL for data manipulation and analysis.
 </li>
 <li>
 Hands-on experience with Databricks for data engineering tasks.
 </li>
 <li>
 Familiarity with Informatica for ETL processes.
 </li>
 <li>
 Strong knowledge of Spark and Pyspark.
 </li>
 </ul>
 
 
 
 
 
 Job Responsibilities:
 
 
 <ul>
 <li>
 Develop and maintain ETL processes using Informatica.
 </li>
 <li>
 Optimize data workflows in a multi-cloud environment.
 </li>
 <li>
 Collaborate with team members to ensure project success.
 </li>
 <li>
 Address technical challenges and provide innovative solutions.
 </li>
 </ul>
</div>

Search 10000+ Data engineer jobs near you this August. New roles added regularly across leading industries. Apply today.

10000 data engineer jobs Near Me

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Data Engineer