streaming data engineer

Virtusa Lifespaces India Pvt Ltd

<div><div><div><div><div>5&ndash;8 years of experience designing and building data pipelines using 
Apache Spark, Databricks or equivalent bigdata frameworks. 
<li>Handson expertise with streaming and messaging systems such as</li> 
Apache Kafka (publish subscribe architecture), Confluent Cloud, 
RabbitMQ or Azure Event Hub. Experience creating producers, 
consumers and topics and integrating them into downstream 
processing. 
<li>Deep understanding of relational databases and CDC. Proficiency</li> 
in SQL Server, Oracle or other RDBMSs; experience capturing 
change events using Debezium or native CDC tools and 
transforming them for downstream consumption. 
<li>Proficiency in programming languages such as Python, Scala or</li> 
Java and solid knowledge of SQL for data manipulation and 
transformation. 
<li>Cloud platform expertise. Experience with Azure or AWS services for</li> 
data storage, compute and orchestration (e.g., ADLS, S3, Azure 
Data Factory, AWS Glue, Airflow, DBX, DLT). 
<li>Data modelling and warehousing. Knowledge of data Lakehouse</li> 
architectures, Delta Lake, partitioning strategies and performance 
optimisation. 
<li>Version control and DevOps. Familiarity with Git and CI/CD</li> 
pipelines; ability to automate deployment and manage 
infrastructure as code. 
<li>Strong problem solving and communication skills. Ability to work</li> 
with cross functional teams and articulate complex technical 
concepts to nontechnical stakeholders.</div>
</div> 
</div></div><div>About Virtusa
Teamwork, quality of life, professional and personal development: values that Virtusa is proud to embody. When you join us, you join a team of 27,000 people globally that cares about your growth &mdash; one that seeks to provide you with exciting projects, opportunities and work with state of the art technologies throughout your career with us.
Great minds, great potential: it all comes together at Virtusa. We value collaboration and the team environment of our company, and seek to provide great minds with a dynamic place to nurture new ideas and foster excellence.
Virtusa was founded on principles of equal opportunity for all, and so does not discriminate on the basis of race, religion, color, sex, gender identity, sexual orientation, age, non-disqualifying physical or mental disability, national origin, veteran status or any other basis covered by appropriate law. All employment is decided on the basis of qualifications, merit, and business need.</div></div>

Data Engineer

Photon

<div><div><div>Job Description: Database Lead
Experience
5&ndash;8 years
Location
Offshore
Role Overview
We are looking for a Database Lead with strong GCP, database, and ETL experience to support a cloud-native data platform. The role involves database design, data modeling, schema evolution, data migrations, ETL pipelines, and data streaming across MySQL, PostgreSQL, and Google Cloud databases (Spanner, Cloud SQL, AlloyDB).
The candidate will act as a technical lead, collaborating with onshore architects and leading offshore DB/ETL resources as additional team members are onboarded.
Key Responsibilities
<ul><li>Lead data modeling, schema design, and database architecture across MySQL, PostgreSQL, Spanner, Cloud SQL, and AlloyDB.
</li><li>Own data migrations (one-time migrations, incremental/delta loads, and ongoing data synchronization).
</li><li>Design and manage config-driven ETL / ELT pipelines, with table-level configuration files and minimal transformations.
</li><li>Support on-demand onboarding of new tables and manage schema changes / evolution.
</li><li>Design and support data streams for batch and near-real-time ingestion.
</li><li>Work across teams on event-driven architectures, ensuring reliable data publishing and downstream consumption.
</li><li>Enable CI/CD deployments for database changes, ETL jobs, dataflows, and configurations.
</li><li>Monitor and troubleshoot ETL pipelines, data syncs, and streaming jobs; support reprocessing and recovery.
</li><li>Prepare and maintain playbooks, runbooks, and operational documentation.
</li><li>Lead and mentor offshore DB/ETL engineers, review designs, guide implementation, and ensure delivery quality.
</li></ul>Mandatory Skills &amp; Experience
<ul><li>5&ndash;8 years of experience in database and data engineering roles.
</li><li>Proven expertise in data modeling, schema design, and data migrations.
</li><li>Strong hands-on experience with MySQL and PostgreSQL.
</li><li>Experience with Google Cloud databases: Spanner, Cloud SQL, AlloyDB.
</li><li>Experience with ETL / ELT pipelines, data integration, and data streaming.
</li><li>Strong understanding of schema evolution, delta processing, and data synchronization.
</li><li>Advanced SQL skills.
</li><li>Experience working on GCP, including services like Pub/Sub, Dataflow, BigQuery, and Cloud Storage.
</li><li>Experience with CI/CD pipelines for data and database deployments.
</li><li>Ability to technically lead offshore teams and work with onshore stakeholders.
</li></ul>Nice to Have
<ul><li>Exposure to other databases / technologies such as AS/400, MongoDB, Cassandra, and GraphQL.
</li><li>Experience in enterprise data modernization or legacy-to-cloud programs.
</li><li>Professional Cloud Certifications (Google / Microsoft / AWS), including:
<ul><li>Google Professional Data Engineer
</li><li>Google Cloud Architect
</li><li>Google Cloud Engineer</li></ul></li></ul></div></div></div>

Database Lead | Technical Lead | India

WALBRYDGE TECHNOLOGIES PVT LTD

Immediate Hiring &ndash; Walbrydge TechnologiesRole OverviewWalbrydge Technologies is hiring Data Engineers to design and build the core data infrastructure that powers real-time analytics, operational intelligence, and machine learning pipelines across our enterprise ecosystem.This role is critical to enabling high-quality, scalable, and reliable data platforms that support real-time dashboards, analytics, and future AI-driven capabilities.Position DetailsRole: Data Engineer &ndash; Platform &amp; Analytics Infrastructure Experience: 5&ndash;8 Years Positions: 3 Location: Hyderabad, India(Offshore) Employment Type: Full-Time Start Date: 1-2 WeeksInterested candidates may send their resume to: info@walbrydge.comKey ResponsibilitiesData Platform &amp; Pipeline Engineering<ul><li>Design and implement end-to-end data pipelines supporting both batch and real-time streaming workloads.</li><li>Build and maintain ETL / ELT workflows using modern data engineering tools.</li><li>Architect and manage data lakes, data warehouses, and real-time event pipelines.</li></ul>Data Ingestion &amp; Processing<ul><li>Develop scalable ingestion pipelines from distributed microservices, APIs, and event streams.</li><li>Implement real-time data processing using Kafka, Spark Streaming, Flink, or equivalent technologies.</li><li>Optimize pipelines for latency, throughput, and fault tolerance.</li></ul>Data Quality, Governance &amp; Optimization<ul><li>Implement data quality checks, validation rules, lineage tracking, and governance standards.</li><li>Design efficient data models, partitioning strategies, and schema evolution.</li><li>Optimize storage and compute for performance, scalability, and cost efficiency.</li></ul>Cross-Functional Collaboration<ul><li>Work closely with ML Engineers, Data Scientists, Backend Engineers, and DevOps teams.</li><li>Support analytics, reporting, and ML feature pipelines with clean, reliable datasets.</li><li>Ensure security, access control, and compliance across all data systems.</li></ul>Required Skills &amp; ExperienceCore Technical Skills<ul><li>5&ndash;8 years of professional experience in Data Engineering.</li><li>Strong proficiency in Python and SQL.</li><li>Hands-on experience with ETL/ELT orchestration tools such as Airflow, dbt, or similar.</li><li>Experience building streaming and event-driven pipelines using Kafka, Kinesis, Spark Streaming, or Flink.</li></ul>Data Storage &amp; Analytics Platforms<ul><li>Strong experience with data warehouses such as:</li><li>Snowflake, Redshift, BigQuery, Azure Synapse, or Databricks</li><li>Hands-on experience with data lakes:</li><li>AWS S3, Azure Data Lake (ADLS), or Google Cloud Storage (GCS)</li></ul>Platform &amp; Architecture Knowledge<ul><li>Strong understanding of data modeling, schema design, partitioning, and indexing.</li><li>Experience with API-based ingestion and real-time event streaming.</li><li>Familiarity with CI/CD pipelines, containerization, and cloud-native architectures.</li></ul>Job Types: Full-time, PermanentBenefits:<ul><li>Flexible schedule</li><li>Food provided</li><li>Health insurance</li><li>Leave encashment</li><li>Paid sick time</li><li>Paid time off</li><li>Provident Fund</li><li>Work from home</li></ul>Work Location: In person

Data Engineer – Platform & Analytics Infrastructure

Valtech Group

<div>Why Valtech? We&rsquo;re advisors, visionaries, creative and techies. We embrace all things digital. We talk to each other. We have fun. We love our clients. We&rsquo;re looking ahead &bull; We are global</div>
 <div>Join our innovative, fast-growing data team at the forefront of cloud data architecture on Microsoft Azure. We're building scalable, secure, and modern data platforms using cutting-edge Azure services and Databricks unified analytics platform. If you're passionate about creating high-performance data infrastructure and solving complex big data challenges in a cloud-native environment, this is the perfect opportunity for you.
As a Senior Data Engineer specializing in Azure and Databricks, you will architect and implement enterprise-grade cloud-native data solutions on the Microsoft Azure ecosystem. This is a hands-on engineering role with significant architectural influence, where you'll work extensively with Azure Data Factory, Databricks, Delta Lake, and other modern data tools to create efficient, maintainable, and scalable data pipelines using medallion architecture and lakehouse patterns.
What you do 
 
4&ndash;7 years of hands-on experience in data engineering with strong focus on cloud data platforms. Proven experience with big data technologies and distributed computing
Technical Expertise 
Azure Mastery: Deep expertise in Azure data services including Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage, and Azure SQL Database 
Databricks Proficiency: Advanced proficiency in Databricks including cluster management, notebook development, and Delta Lake 
Big Data Processing: Strong experience with Apache Spark (PySpark, Scala), distributed computing concepts, and performance optimization 
Programming: Proficiency in Python, Scala, and SQL for data processing and transformation 
 
Data Architecture 
Experience with lakehouse architecture, medallion architecture, and modern data warehouse design patterns 
Streaming Technologies: Knowledge of real-time data processing using Azure Event Hubs, Kafka, and Spark Streaming 
Cloud Platforms: Deep understanding of Microsoft Azure ecosystem and native data services
Data Management &amp; DevOps 
Data Governance: Understanding of data governance frameworks, Unity Catalog, and data quality practices 
DevOps: Experience with CI/CD practices, Git-based workflows, and Infrastructure as Code 
Orchestration: Knowledge of data orchestration tools like Azure Data Factory, Airflow, or similar platforms 
Security: Understanding of Azure security best practices, RBAC, and data encryption
Advanced Technologies 
Experience with Azure Machine Learning and MLOps practices 
Knowledge of containerization (Docker, Kubernetes) and Azure Container Instances 
Experience with Power BI for data visualization and reporting 
Understanding of data mesh architecture and domain-driven design principles 
Experience with Azure DevOps for CI/CD pipeline management 
Familiarity with monitoring tools like Azure Monitor, Application Insights, and Databricks monitoring
Specialized Skills 
Experience with real-time analytics and complex event processing 
Knowledge of graph databases and Azure Cosmos DB 
Understanding of data lake optimization techniques and performance tuning 
Experience with multi-cloud or hybrid cloud architectures
Technology Stack 
Core Azure Services 
Data Storage: Azure Data Lake Storage Gen2, Azure Blob Storage, Azure SQL Database 
Data Processing: Databricks, Azure Synapse Analytics, Azure Data Factory 
Streaming: Azure Event Hubs, Azure Stream Analytics, Kafka 
Analytics: Databricks SQL, Azure Analysis Services, Power BI 
Machine Learning: Azure Machine Learning, MLflow, Databricks ML
Big Data &amp; Processing 
Compute: Apache Spark (PySpark, Scala), Databricks Runtime 
Data Formats: Delta Lake, Parquet, JSON, Avro 
Architecture: Medallion Architecture, Lakehouse, Data Mesh 
Orchestration: Azure Data Factory, Airflow, Azure Logic Apps
Development &amp; DevOps 
Programming: Python, Scala, SQL, PowerShell 
Version Control: Git, Azure DevOps, GitHub 
Infrastructure: ARM Templates, Terraform, Azure Resource Manager 
Monitoring: Azure Monitor, Application Insights, Databricks monitoring
Security &amp; Governance 
Security: Azure Active Directory, Key Vault, RBAC 
Governance: Unity Catalog, Azure Purview, Data Lineage 
Compliance: GDPR, data encryption, access controls
Certifications 
Microsoft Azure Data Engineer Associate (DP-203) 
Databricks Certified Associate Developer for Apache Spark 
Databricks Certified Professional Data Engineer 
Azure Solutions Architect or Azure Data Fundamentals certifications
What we ask
Data Architecture &amp; Engineering 
Design and implement enterprise-scale, robust data solutions using Azure and Databricks 
Architect scalable ELT/ETL pipelines using Azure Data Factory, Azure Synapse Analytics, and Databricks 
Build automated data ingestion processes from various sources including streaming and batch data 
Develop and maintain data transformation workflows using PySpark, Scala, and SQL on Databricks 
 
Data Platform &amp; Optimization 
Implement lakehouse architecture using Delta Lake and Databricks Delta tables 
Design and optimize Databricks clusters for performance, cost management, and resource utilization 
Apply medallion architecture (Bronze, Silver, Gold) for data processing and transformation 
Optimize Spark jobs and queries for maximum performance and cost efficiency 
 
Big Data &amp; Analytics 
Build streaming data pipelines using Azure Event Hubs, Kafka, and Databricks Structured Streaming 
Implement machine learning pipelines using MLflow and Databricks ML capabilities 
Design and maintain data models for analytical workloads and real-time processing 
Work with large-scale datasets and implement partitioning strategies for optimal performance 
 
Quality &amp; Governance 
Develop comprehensive data quality tests and monitoring using Great Expectations and custom frameworks 
Implement data governance, lineage, and security policies within Azure and Databricks 
Create and maintain comprehensive data documentation and lineage tracking using Unity Catalog 
Build data validation and testing frameworks for proactive data monitoring 
 
DevOps &amp; Automation 
Build and maintain CI/CD pipelines for Databricks notebooks and data workflows 
Develop custom Python/Scala scripts for data processing, manipulation, and automation tasks 
Implement Infrastructure as Code (IaC) using ARM templates or Terraform for Azure resources 
Work with orchestration tools for pipeline scheduling and dependency management 
 
Collaboration &amp; Analytics 
Collaborate with data scientists, analysts, and ML engineers to support advanced analytics use cases 
Enable self-service analytics capabilities using Databricks SQL and Azure Analytics services 
Work closely with stakeholders to understand data requirements and deliver scalable solutions
What we offer
You&rsquo;ll join an international network of data professionals within our organisation. We support continuous development through our dedicated Academy. If you're looking to push the boundaries of innovation and creativity in a culture that values freedom and responsibility, we encourage you to apply.
At Valtech, we&rsquo;re here to engineer experiences that work and reach every single person. To do this, we are proactive about creating workplaces that work for every person at Valtech. Our goal is to create an equitable workplace which gives people from all backgrounds the support they need to thrive, grow and meet their goals (whatever they may be). You can find out more about what we&rsquo;re doing to create a Valtech for everyone here.<div>
Please do not worry if you do not meet all of the criteria or if you have some gaps in your CV. We&rsquo;d love to hear from you and see if you&rsquo;re our next member of the Valtech team!</div></div>

Azure & Databricks - Senior Data Engineer

BMC Solutions

Carpediem Studio is Hiring!Position: PCR Engineer (Audio, Video, Lighting, Broadcast &amp; Live Streaming)We&rsquo;re looking for an experienced PCR Engineer to manage and operate AVL systems, broadcast setups, and live streaming operations in a fast-paced production environment.Key Responsibilities:<ul><li>Operate and manage Production Control Room (PCR) systems</li><li>Set up and operate audio, video, and lighting equipment</li><li>Handle live streaming &amp; broadcasting on digital and satellite platforms</li><li>Coordinate camera switching, audio mixing &amp; lighting control during live events</li><li>Monitor signal flow, AV quality, and system performance</li><li>Troubleshoot issues in real time during live productions</li><li>Manage encoders, decoders, switchers, mixers &amp; control systems</li><li>Coordinate with producers, directors &amp; technical teams</li><li>Conduct post-event checks and reporting</li></ul>Requirements:<ul><li>Diploma / Bachelor&rsquo;s in Media Technology, Broadcast Engineering, or related field</li><li>2&ndash;3 years experience in PCR / Broadcast / Live Production</li></ul>Job Type: Full-timePay: Rs70,000.00 - Rs90,000.00 per monthWork Location: In person

PCR Engineer (Audio, Video, Lighting, Broadcast & Live Streaming)

Systems Limited

<div>We are looking for an experienced ETL PySpark Data Engineer to design, build, and optimize scalable data pipelines.
<h4 class="jobSectionHeader">
Responsibilities</h4>
<ul>
<li>Design, develop, and maintain ETL pipelines using PySpark
</li>
<li>Process large-scale structured and semi-structured data efficiently
</li>
<li>Build batch and (optional) streaming data pipelines
</li>
<li>Perform data ingestion, transformation, and loading from multiple sources
</li>
<li>Optimize Spark jobs for performance and scalability
</li>
<li>Work with cloud data platforms (Azure / AWS / GCP)
</li>
<li>Ensure data quality, reliability, and governance
</li>
<li>Collaborate with data analysts, architects, and stakeholders
</li>
</ul><h4 class="jobSectionHeader">Requirement:</h4>
<ul>
<li>Strong hands-on experience with PySpark
</li>
<li>Solid understanding of ETL concepts &amp; data warehousing
</li>
<li>Strong SQL skills
</li>
<li>Experience with Spark SQL, DataFrames, and RDDs
</li>
<li>Familiarity with HDFS, Hive, Delta Lake
</li>
<li>Experience with cloud services (Azure Data Factory, Databricks, AWS Glue, etc.)
</li>
<li>Knowledge of Linux &amp; Git</li></ul></div>

Senior Consultant Data Analytics

JioStar

<div><div>Job Summary: As a Software Development Engineer II, you&rsquo;ll build a world-class platform for our customers by solving complex technical problems and building systems which run at scale. You will play a critical role in developing code, and solve problems independently while shaping our next-gen solutions that drive our company's mission forward. We seek candidates with a passion for crafting elegant code and building innovative software solutions.</div><div></div><div> 
About the team: Join our Player Team to build and optimize video playback for millions of users on our Android platform. If you&rsquo;re passionate about ExoPlayer, streaming tech, and creating a smooth user experience</div>
<h3 class="jobSectionHeader">Key responsibilities:</h3><ul><li>
Build &amp; maintain our Android video player using ExoPlayer.</li><li>
Optimize adaptive streaming, DRM, offline playback &amp; error handling.</li><li>
Work with cross-platform teams on media APIs, analytics, and QoE improvements.</li><li>
Debugging, performance tuning &amp; QA mindset.</li><li>
Optimize playback start time (latency), minimize buffering, and manage memory usage to prevent ANRs (Application Not Responding).</li><li>
Leverage MediaCodec APIs to ensure efficient hardware-accelerated decoding and reduce battery consumption.</li><li>
Resolve device-specific playback issues across different OEMs (Samsung, Pixel, OnePlus, etc.) and chipset architectures.</li></ul><h3 class="jobSectionHeader">
Skills and attributes for success:</h3><ul><li>
Strong knowledge of ExoPlayer/Android Media APIs</li><li>
Experience with HLS/DASH, DRM (Widevine), ABR streaming</li><li>
Exposure to live streaming, CMCD, QoE, or VR/360 playbackGood to have OTT/Streaming experience</li><li>
2-4 years of experience in Kotlin (preferred), with a deep understanding of the Android SDK and LifeCycles.</li><li>
Good experience with ExoPlayer / Media3.
</li><li>Knowledge of PlayerView, MediaSource, and TrackSelector.</li><li>
Deep knowledge of DASH, HLS, and Low-Latency HLS.</li><li>
Experience implementing Widevine (Modular) and managing secure key exchanges.</li><li>
Proficiency in using SurfaceView vs. TextureView for video rendering and handling aspect ratio transformations.</li></ul><h3 class="jobSectionHeader">
Preferred education and experience:</h3><ul><li>
Bachelor&rsquo;s or Master&rsquo;s degree in Computer Science or a related field.</li><li>
2-4 years of Android dev (Kotlin/Java).</li></ul><div>
About Us</div><div>
Perched firmly at the nucleus of spellbinding content and innovative technology, JioStar is a leading global media &amp; entertainment company that is reimagining the way audiences consume entertainment and sports. Its television network and streaming service together reach more than 750 million viewers every week, igniting the dreams and aspirations of hundreds of million people across geographies.</div><div></div><div> 
 JioStar is an equal opportunity employer. The company values diversity and its mission is to create a workplace where everyone can bring their authentic selves to work. The company ensures that the work environment is free from any discrimination against persons with disabilities, gender, gender identity and any other characteristics or status that is legally protected.</div>

If you would like more information about how your data is processed, please contact us.</div>

Software Development Engineer II (Android)

Search 10000+ Streaming data engineer jobs near you this January. New roles added regularly across leading industries. Apply today.