streaming data engineer

Prodapt Solutions

<div>Overview:
<h3 class="jobSectionHeader">Overview:</h3>
We are seeking a talented and motivated GCP Data Engineer with experience in real-time streaming technologies, specifically Apache Kafka, and proficiency in Java. The ideal candidate will be responsible for designing, developing, and maintaining data pipelines and streaming applications in a cloud environment. 

Responsibilities:
<h3 class="jobSectionHeader">Key Responsibilities:</h3><ul><li>
Data Pipeline Development: Design and implement robust data pipelines using Google Cloud services (BigQuery, Dataflow, Pub/Sub) integrated with Kafka for real-time data processing.</li><li>
Streaming Applications: Develop and maintain real-time streaming applications using Kafka and Java, ensuring high availability and performance.</li><li>
Cloud Infrastructure: Utilize GCP technologies to build scalable and efficient data architectures, leveraging tools such as Cloud Storage, Cloud Functions, and Kubernetes.</li><li>
Monitoring and Optimization: Monitor data flow, performance metrics, and troubleshoot issues in production environments. Optimize data pipeline performance and reliability.</li><li>
Collaboration: Work closely with data scientists, analysts, and other engineers to understand data requirements and translate them into technical specifications.</li><li>
Documentation: Maintain comprehensive documentation of data pipelines, architecture, and processes to ensure knowledge sharing and compliance.</li></ul> 

Requirements:
<h3 class="jobSectionHeader">Required Skills:</h3><ul><li>
Cloud Skills: Proficiency in Google Cloud Platform (GCP), including services like BigQuery, Dataflow, and Pub/Sub.</li><li>
Streaming Technologies: Strong experience with Kafka for real-time data streaming and messaging.</li><li>
Programming Language: Proficient in Java, with a solid understanding of object-oriented programming principles and best practices.</li><li>
Data Management: Familiarity with data modeling, ETL processes, and database management.</li><li>
Development Tools: Experience with version control systems (e.g., Git), CI/CD tools, and Agile methodologies.</li><li>
Problem-Solving: Strong analytical and troubleshooting skills, with the ability to work independently and as part of a team.</li></ul><h3 class="jobSectionHeader"> 
Preferred Qualifications:</h3><ul><li>
Bachelor's degree in Computer Science, Information Technology, or a related field.</li><li>
Experience with other programming languages (Python, Scala) is a plus.</li><li>
Knowledge of additional GCP services or data visualization tools (e.g., Looker, Tableau) is advantageous.</li></ul></div>

GCP Data Engineer (Kafka + Java)

Envision Employment Solutions

Envision Employment Solutions is currently looking for a Data Streaming Engineer for one of our partners, a global leader in consulting, digital transformation, technology, and engineering services.
 
Job Summary: 
As a Data Streaming Engineer, you will be building and maintaining robust, scalable Kubernetes-based event streaming platforms. You will play a pivotal role in ensuring the reliability and efficiency of our production and development environments, enabling smooth deployments and accurate event reporting.
 
This position requires flexibility to work on US times and flexibility to travel abroad when needed.
 
Responsibilities: 
<ul>
<li>Kubernetes Infrastructure Management:</li>
<li>Design, provision, and maintain Kubernetes clusters across various environments (cloud, on-premises, hybrid).</li>
<li>Monitor cluster performance and optimize resource utilization.</li>
<li>Troubleshoot issues and implement solutions to ensure high availability and stability.</li>
<li>Splunk Operations & Engineering:</li>
<li>Manage Splunk infrastructure including indexers, search heads.</li>
<li>Troubleshoot ingestion issues, latency, and indexing delays across distributed environments.</li>
<li>Develop and maintain SPL queries, saved searches, alerts, and dashboards.</li>
<li>Perform capacity planning, performance tuning, and upgrade planning for Splunk components.</li>
<li>Implement data onboarding strategies and field extractions for new log sources.</li>
<li>Cribl Pipeline Management:</li>
<li>Candidate Cribl experience is advantageous but not required.</li>
<li>Design, build, and optimize Cribl pipelines for log routing, transformation, filtering, and enrichment.</li>
<li>Integrate Cribl with various data sources and destinations.</li>
<li>Automate pipeline deployments and configuration using CI/CD and GitOps practices.</li>
<li>Production Support:</li>
<li>Participate in incident response and resolution, ensuring minimal downtime and disruption to services.</li>
<li>Continuously analyze and improve the performance, reliability, and security of production environments.</li>
</ul>
Requirements: 
<ul>
<li>Bachelor's degree in Engineering, Computer Science, Information Technology, or a related field.</li>
<li>Minimum of 5 years of professional experience.</li>
<li>Outstanding English communication, both verbal and non-verbal.</li>
<li>Strong Kubernetes Expertise: Deep understanding of Kubernetes architecture, concepts, and best practices. Hands-on experience with deploying and managing Kubernetes clusters at scale.</li>
<li>Splunk or ELK Familiarity: Experience working with log data platforms such as Splunk and ELK, including organizing, interpreting, and preparing data for analysis. Familiarity with tools and processes used to manage and streamline data flows across systems.</li>
<li>GitLab Familiarity: Understanding of GitLab CI/CD features and capabilities. Ability to write efficient and maintainable pipeline scripts.</li>
<li>Cloud Infrastructure Knowledge: Experience with cloud platforms (AWS, Azure, GCP) and cloud-native technologies.</li>
<li>Troubleshooting and Problem-Solving: Ability to analyze complex problems, diagnose root causes, and implement effective solutions.</li>
<li>Collaboration and Communication: Strong communication and interpersonal skills, able to collaborate effectively with cross-functional teams.</li>
</ul>
Benefits: 
<ul>
<li>Competitive Salary based on experience</li>
<li>Social and medical insurance</li>
<li>Learning, development and career progression</li>
</ul>

Data Streaming Engineer

SALLA

<div>We are seeking a skilled Senior Data Engineer with deep expertise in ClickHouse and streaming data, and a passion for building scalable real-time analytics solutions. In this role, you will design, develop, and optimize our data pipelines and analytics infrastructure, empowering our teams to harness real-time insights that enhance customer experience and drive business growth.
Key Responsibilities<ul><li>Design, implement, maintain and document highly scalable data pipelines for real-time and batch processing.</li><li>Build and optimize data systems to support accurate, low-latency analytics and reporting use cases.</li><li>Develop and maintain solutions for streaming and serverless data processing.</li><li>Collaborate with cross-functional teams to implement and support end-to-end analytics workflows.</li><li>Ensure data quality, reliability, and performance across the platform.</li><li>Monitor, troubleshoot, and optimize data infrastructure to maintain high availability.</li><li>Mentor junior engineers and contribute to the continuous improvement of engineering practices.</li><li>Strong problem-solving skills and the ability to thrive in a fast-paced environment.
</li><li>Excellent communication and teamwork skills.
</li></ul>Requirements<ul><li>
5+ years of experience in Data Engineering or related fields.
</li></ul><ul><li>Strong expertise in ClickHouse (schema design, ingestion optimization, query performance tuning, and cluster management).
</li><li>Proven experience with real-time data processing using Apache Kafka, Flink, or Spark Streaming.
</li><li>Deep understanding of distributed systems architecture with emphasis on scalability, reliability, and fault tolerance.
</li><li>Proficiency in one or more programming languages:
</li></ul>Python: Building data pipelines, automation scripts, and integrations.
Go: Developing high-performance data services or tools.
TypeScript: Contributing to data-related front-end or service-side applications.
Bash/Shell scripting: Writing automation scripts for data operations.
Rust (Good to have): Interest or experience in building memory-safe, high-performance systems.
<ul><li>Hands-on experience with cloud platforms such as AWS, GCP, or Azure.
</li><li>Familiarity with containerization and orchestration tools (Docker, Kubernetes).</li></ul></div>

Senior Data Engineer

Confiz

<div>
 We are seeking a Data Engineer with
 
 3+ years
 
 of hands-on experience and a strong background in real-time and batch data processing, containerization, and cloud-based data orchestration. This role is ideal for someone passionate about building robust, scalable, and efficient data pipelines, and who thrives in agile, collaborative environments.
 
 
 
 Responsibilities
 
 
 
 <ul>
 <li>
 Design, build, and maintain real-time data pipelines using streaming frameworks such as Kafka, Apache Flink, and Spark Structured Streaming.
 </li>
 <li>
 Develop batch processing workflows with Apache Spark (PySpark)
 </li>
 <li>
 Orchestrate and schedule data workflows using orchestration frameworks such as Apache Airflow and Azure Data Factory
 </li>
 <li>
 Containerize applications using Docker, manage deployments with Helm, and run them on Kubernetes
 </li>
 <li>
 Implement modern storage solutions using open formats such as Parquet, Delta Lake, and Apache Iceberg
 </li>
 <li>
 Build high-performance analytics engines using tools like Trino or Presto
 </li>
 <li>
 Collaborate with DevOps to manage infrastructure with Terraform and integrate with CI/CD pipelines via Azure DevOps
 </li>
 <li>
 Ensure data quality and consistency using tools like Great Expectations
 </li>
 <li>
 Write modular, well-tested, and maintainable Python and SQL code
 </li>
 <li>
 Develop an observability layer to monitor and optimize performance across data pipelines
 </li>
 <li>
 Participate in agile ceremonies and contribute to sprint planning and reviews
 
 
 </li>
 </ul>
 
 Required Skills
 
 
 
 <ul>
 <li>
 Advanced Python programming with a strong focus on modular and testable code
 </li>
 <li>
 Strong knowledge of SQL and experience working with large-scale datasets
 </li>
 <li>
 Hands-on experience with at least one major cloud platform (Azure preferred)
 </li>
 <li>
 Solid experience with real-time data processing (Kafka, Flink, or Spark Streaming)
 </li>
 <li>
 Expertise in Apache Spark (PySpark) for batch processing
 </li>
 <li>
 Experience implementing lakehouse architectures and working with columnar storage (e.g., ClickHouse)
 </li>
 <li>
 Proficient in using Azure Data Factory or Apache Airflow for data orchestration
 </li>
 <li>
 Experience in building APIs to expose large datasets
 </li>
 <li>
 Solid experience with Docker, Kubernetes, and Helm
 </li>
 <li>
 Familiarity with data lake open formats such as Parquet, Delta Lake, and Iceberg
 </li>
 <li>
 Basic experience with Terraform for infrastructure provisioning
 </li>
 <li>
 Practical experience with data quality frameworks (e.g., Great Expectations)
 </li>
 <li>
 Comfortable working in agile development teams
 </li>
 <li>
 Proven ability in debugging and performance tuning of streaming and batch data jobs
 </li>
 <li>
 Experience with AI-driven tools (e.g., text-to-SQL) is a plus
 </li>
 <li>
 Bachelor&rsquo;s degree in computer science or related discipline
 
 
 </li>
 </ul>
 We have an amazing team of 700+ individuals working on highly innovative enterprise projects &amp; products. Our customer base includes Fortune 100 retail and CPG companies, leading store chains, fast-growth fintech, and multiple Silicon Valley startups.
 
 
 What makes Confiz stand out is our focus on processes and culture. Confiz is
 
 ISO 9001:2015
 
 (QMS),
 
 ISO 27001:2022
 
 (ISMS),
 
 ISO 20000-1:2018
 
 (ITSM),
 
 ISO 14001:2015
 
 (EMS),
 
 ISO 45001:2018
 
 (OHSMS) Certified. We have a vibrant culture of learning via collaboration and making workplace fun.
 
 
 People who work with us work with cutting-edge technologies while contributing success to the company as well as to themselves.
 
 
 To know more about Confiz Limited, visit: https://www.linkedin.com/company/confiz-pakistan/
</div>

Senior/Principal Data Engineer

<div>We are seeking a Data Engineer with 3+ years of hands-on experience and a strong background in real-time and batch data processing, containerization, and cloud-based data orchestration. This role is ideal for someone passionate about building robust, scalable, and efficient data pipelines, and who thrives in agile, collaborative environments.
Responsibilities
<ul><li>Design, build, and maintain real-time data pipelines using streaming frameworks such as Kafka, Apache Flink, and Spark Structured Streaming.</li>
<li>Develop batch processing workflows with Apache Spark (PySpark)</li>
<li>Orchestrate and schedule data workflows using orchestration frameworks such as Apache Airflow and Azure Data Factory</li>
<li>Containerize applications using Docker, manage deployments with Helm, and run them on Kubernetes</li>
<li>Implement modern storage solutions using open formats such as Parquet, Delta Lake, and Apache Iceberg</li>
<li>Build high-performance analytics engines using tools like Trino or Presto</li>
<li>Collaborate with DevOps to manage infrastructure with Terraform and integrate with CI/CD pipelines via Azure DevOps</li>
<li>Ensure data quality and consistency using tools like Great Expectations</li>
<li>Write modular, well-tested, and maintainable Python and SQL code</li>
<li>Develop an observability layer to monitor and optimize performance across data pipelines</li>
<li>Participate in agile ceremonies and contribute to sprint planning and reviews</li>
</ul>
Required Skills
<ul><li>Advanced Python programming with a strong focus on modular and testable code</li>
<li>Strong knowledge of SQL and experience working with large-scale datasets</li>
<li>Hands-on experience with at least one major cloud platform (Azure preferred)</li>
<li>Solid experience with real-time data processing (Kafka, Flink, or Spark Streaming)</li>
<li>Expertise in Apache Spark (PySpark) for batch processing</li>
<li>Experience implementing lakehouse architectures and working with columnar storage (e.g., ClickHouse)</li>
<li>Proficient in using Azure Data Factory or Apache Airflow for data orchestration</li>
<li>Experience in building APIs to expose large datasets</li>
<li>Solid experience with Docker, Kubernetes, and Helm</li>
<li>Familiarity with data lake open formats such as Parquet, Delta Lake, and Iceberg</li>
<li>Basic experience with Terraform for infrastructure provisioning</li>
<li>Practical experience with data quality frameworks (e.g., Great Expectations)</li>
<li>Comfortable working in agile development teams</li>
<li>Proven ability in debugging and performance tuning of streaming and batch data jobs</li>
<li>Experience with AI-driven tools (e.g., text-to-SQL) is a plus</li>
<li>Bachelor&rsquo;s degree in computer science or related discipline</li>
</ul>
We have an amazing team of 700+ individuals working on highly innovative enterprise projects &amp; products. Our customer base includes Fortune 100 retail and CPG companies, leading store chains, fast-growth fintech, and multiple Silicon Valley startups.
What makes Confiz stand out is our focus on processes and culture. Confiz is ISO 9001:2015 (QMS), ISO 27001:2022 (ISMS), ISO 20000-1:2018 (ITSM), ISO 14001:2015 (EMS), ISO 45001:2018 (OHSMS) Certified. We have a vibrant culture of learning via collaboration and making workplace fun.
People who work with us work with cutting-edge technologies while contributing success to the company as well as to themselves.
To know more about Confiz Limited, visit: https://www.linkedin.com/company/confiz-pakistan/</div>

Tesla

<div>What to Expect<div>
We are seeking a skilled and collaborative Data Engineer to join our team. In this role, you will architect and implement a cutting data platform while leading the development of data pipelines, data warehousing, and reporting infrastructure to support the growth of our energy business including Industrial, Residential, Supercharger, and Solar products. 
 
As a key member of our team, you will design, build, and maintain large-scale data systems to combine data from various data sources, including deployed fleets, internal applications, and other data warehouses. You will develop a state-of-the-art monitoring and alerting layer to ensure system reliability and performance. Your work will contribute to our understanding of key business performance indicators, informing data-driven decisions to optimize our strategies and drive decision making for all our hardware products. 
 
You will build efficient, reusable data components and implement both batch and streaming pipelines that power our systems. Working alongside ML engineers, you'll support feature engineering efforts and create the data infrastructure needed for successful model deployment. Your work will enable critical analytics to track our energy products throughout their lifecycle&mdash;from creation to deployment, usage, maintenance, and replacement. Collaborating with teams across Industrial Energy, Residential Energy, and Solar Product and Service Engineering, you'll help optimize the performance and reliability of our diverse energy product fleet. 
 
This role requires expertise in streaming architectures, real-time data processing, and the ability to build modular, reusable components that can scale with our growing data needs. You will be at the forefront of implementing modern data engineering practices that enable rapid development and deployment of data solutions across the organization.</div> 

What You'll Do<div><ul><li>
Design and implement robust, scalable solutions for processing terabyte-scale data</li><li>
Develop and maintain high-performance data pipelines using Apache Spark, Kafka, and other modern data technologies</li><li>
Build and optimize batch and streaming data processing systems</li><li>
Develop aggregate and summary tables for various engineering teams that span across multiple product lines and geographies</li><li>
Design and develop new systems and tools to enable cross-functional teams to consume and understand data faster</li><li>
Implement and maintain CI/CD pipelines for data applications</li><li>
Research and incorporate emerging data infrastructures, tools, and technologies</li></ul></div> 
What You'll Bring<div><ul><li>
Strong data engineering background with expertise in at least two programming languages (Python, Scala, Java, or Rust)</li><li>
Expert-level experience with Apache Spark (PySpark, Spark SQL, Spark Streaming)</li><li>
Experience with modern data lake technologies (Delta Lake, Apache Iceberg, Apache Hudi)</li><li>
Experience with containerization and orchestration (Docker, Kubernetes)</li><li>
Proficiency in SQL and familiarity with database internals and performance concepts</li><li>
Demonstrated experience working with terabyte-scale data processing</li></ul></div> 
Compensation and Benefits<div>
Benefits</div><div>
Along with competitive pay, as a full-time Tesla employee, you are eligible for the following benefits at day 1 of hire:
<ul><li>Aetna PPO and HSA plans &gt; 2 medical plan options with $0 payroll deduction</li>
<li>Family-building, fertility, adoption and surrogacy benefits</li>
<li>Dental (including orthodontic coverage) and vision plans, both have options with a $0 paycheck contribution</li>
<li>Company Paid (Health Savings Account) HSA Contribution when enrolled in the High Deductible Aetna medical plan with HSA</li>
<li>Healthcare and Dependent Care Flexible Spending Accounts (FSA)</li>
<li>401(k) with employer match, Employee Stock Purchase Plans, and other financial benefits</li>
<li>Company paid Basic Life, AD&amp;D, short-term and long-term disability insurance</li>
<li>Employee Assistance Program</li>
<li>Sick and Vacation time (Flex time for salary positions), and Paid Holidays</li>
<li>Back-up childcare and parenting support resources</li>
<li>Voluntary benefits to include: critical illness, hospital indemnity, accident insurance, theft &amp; legal services, and pet insurance</li>
<li>Weight Loss and Tobacco Cessation Programs</li>
<li>Tesla Babies program</li>
<li>Commuter benefits</li>
<li>Employee discounts and perks program</li>
</ul></div><div>Expected Compensation</div><div>
$80,000 - $186,000/annual salary + cash and stock awards + benefits
Pay offered may vary depending on multiple individualized factors, including market location, job-related knowledge, skills, and experience. The total compensation package for this position may also include other elements dependent on the position offered. Details of participation in these benefit plans will be provided if an employee receives an offer of employment.</div> 
Tesla is an Equal Opportunity / Affirmative Action employer committed to diversity in the workplace. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, age, national origin, disability, protected veteran status, gender identity or any other factor protected by applicable federal, state or local laws. 
 
Tesla is also committed to working with and providing reasonable accommodations to individuals with disabilities. Please let your recruiter know if you need an accommodation at any point during the interview process.</div>

Sr. Big Data Engineer, Energy Service Engineering

Huspy

The Main Event: What You ll Drive, Build, and Own
Design and Scale Data Pipelines: Build and maintain reliable batch and streaming pipelines that connect our applications, events, and external sources to Snowflake, ensuring data flows seamlessly across our platform.
Streaming Data at Scale: Work with Kafka, Schema Registry, and event-driven architectures to power real-time data products and ensure downstream systems get trustworthy, well-structured events.
Pipeline Orchestration: Use Airflow to orchestrate complex workflows, automate dependencies, and ensure data availability for analytics and business processes.
Data Infrastructure &amp; Cloud Services: Leverage AWS tools (ECR, ECS, S3, etc.) to build scalable, containerized data services and optimize storage, compute, and costs.
Data Quality &amp; Observability: Implement testing frameworks, monitoring, and alerting for data pipelines to ensure reliability and trust at scale.
Infrastructure as Code: Manage data infrastructure with tools like Terraform, Helmfile, or CloudFormation, automating deployments and ensuring reproducibility.
Optimize for Scale and Performance: Continuously improve the throughput, latency, and cost efficiency of pipelines, ensuring the platform scales with the business.
Modeling &amp; Governance: Apply strong data modeling fundamentals (e.g., Kimball, Data Vault) and enforce governance around PII, compliance, and lifecycle management.
Collaboration for Impact: Partner with analysts, business stakeholders, and product teams to translate requirements into well-designed pipelines and models that unlock business value.
Automation &amp; CI/CD: Implement testing, linting, and automated deployments for data pipelines to ensure high-quality, production-ready code.

Data Engineer

Search 10000+ Streaming data engineer jobs near you this November. New roles added regularly across leading industries. Apply today.

10000 streaming data engineer jobs NEAR_ME

GCP Data Engineer (Kafka + Java)

Data Streaming Engineer

Senior Data Engineer

Senior/Principal Data Engineer

Senior/Principal Data Engineer

Sr. Big Data Engineer, Energy Service Engineering

Data Engineer