data scientist

associate data architect - data lake/data streaming Jobs in Karachi

associate data architect - data lake/data streaming

data analyst

ibm datastage administrator Jobs in Karachi

ibm datastage administrator

ai data scientist

associate data architect data lake data streaming

Confiz

<div>We are seeking a Data Engineer with 8+ years of hands-on experience and a strong background in real-time and batch data processing, containerization, and cloud-based data orchestration. This role is ideal for someone passionate about building robust, scalable, and efficient data pipelines, and who thrives in agile, collaborative environments.
Key Responsibilities
<ul><li>Design, build, and maintain real-time data pipelines using streaming frameworks such as Kafka, Apache Flink, and Spark Structured Streaming.</li>
<li>Develop batch processing workflows with Apache Spark (PySpark)</li>
<li>Orchestrate and schedule data workflows using orchestration frameworks such as Apache Airflow and Azure Data Factory</li>
<li>Containerize applications using Docker, manage deployments with Helm, and run them on Kubernetes</li>
<li>Implement modern storage solutions using open formats such as Parquet, Delta Lake, and Apache Iceberg</li>
<li>Build high-performance analytics engines using tools like Trino or Presto</li>
<li>Collaborate with DevOps to manage infrastructure with Terraform and integrate with CI/CD pipelines via Azure DevOps</li>
<li>Ensure data quality and consistency using tools like Great Expectations</li>
<li>Write modular, well-tested, and maintainable Python and SQL code</li>
<li>Develop an observability layer to monitor and optimize performance across data pipelines</li>
<li>Lead and mentor a team of data engineers</li>
<li>Participate in agile ceremonies and contribute to sprint planning and reviews</li>
</ul>
Required Skills &amp; Experience
<ul><li>Advanced Python programming with a strong focus on modular and testable code</li>
<li>Strong knowledge of SQL and experience working with large-scale datasets</li>
<li>Hands-on experience with at least one major cloud platform (Azure preferred)</li>
<li>Solid experience with real-time data processing (Kafka, Flink, or Spark Streaming)</li>
<li>Expertise in Apache Spark (PySpark) for batch processing</li>
<li>Experience implementing lakehouse architectures and working with columnar storage (e.g., ClickHouse)</li>
<li>Proficient in using Azure Data Factory or Apache Airflow for data orchestration</li>
<li>Experience in building APIs to expose large datasets</li>
<li>Solid experience with Docker, Kubernetes, and Helm</li>
<li>Familiarity with data lake open formats such as Parquet, Delta Lake, and Iceberg</li>
<li>Basic experience with Terraform for infrastructure provisioning</li>
<li>Practical experience with data quality frameworks (e.g., Great Expectations)</li>
<li>Comfortable working in agile development teams</li>
<li>Proven ability in debugging and performance tuning of streaming and batch data jobs</li>
<li>Experience with AI-driven tools (e.g., text-to-SQL) is a plus</li>
<li>Bachelor&rsquo;s degree in computer science or related discipline</li>
</ul>
We have an amazing team of 700+ individuals working on highly innovative enterprise projects &amp; products. Our customer base includes Fortune 100 retail and CPG companies, leading store chains, fast-growth fintech, and multiple Silicon Valley startups.
What makes Confiz stand out is our focus on processes and culture. Confiz is ISO 9001:2015 (QMS), ISO 27001:2022 (ISMS), ISO 20000-1:2018 (ITSM), ISO 14001:2015 (EMS), ISO 45001:2018 (OHSMS) Certified. We have a vibrant culture of learning via collaboration and making workplace fun.
People who work with us work with cutting-edge technologies while contributing success to the company as well as to themselves.
To know more about Confiz Limited, visit: https://www.linkedin.com/company/confiz-pakistan/</div>

Associate Data Architect - Data Lake/Data Streaming

Datamatics Global Services Ltd

<div>Job Title: Lead Confluent Kafka Engineer/Architect 
Location: Remote (Pak) / Hybrid (Riyadh, KSA) 
Employment Type: Full-Time / Contract-to-Hire 
Experience Level: 10+ years (Senior/Lead) 
 
Role Summary 
As a Lead Confluent Kafka Engineer, you'll architect, design, setup, install, implement and optimize high-throughput data streaming solutions using Confluent Platform and Apache Kafka. You'll lead a team of engineers in delivering production-grade pipelines, ensuring scalability, reliability, and security. This role involves hands-on development, mentoring, and collaborating with data architects, DevOps, and stakeholders to implement event-driven architectures. You'll champion best practices in real-time data processing, from proof-of-concepts to enterprise deployments, including full lifecycle management from installation to optimization. 
 
Key Responsibilities<ul><li>
Architecture &amp; Design: Lead the design of scalable Kafka clusters and Confluent-based ecosystems (e.g., Kafka Streams, ksqlDB, Schema Registry, Connect) for on-prem, hybrid, and multi-cloud (GCP) environments.</li><li>
Implementation &amp; Development: Build and maintain real-time data pipelines, integrations, and microservices using Kafka producers/consumers; integrate with tools like Flink, Spark, or ML frameworks for advanced analytics.</li><li>
Installation &amp; Setup: Oversee the end-to-end installation and initial configuration of Confluent Platform and Apache Kafka clusters, including:<ul><li>
Deploying Confluent Enterprise/Community editions on Kubernetes (via Helm/Operator), bare-metal servers, or managed cloud services (e.g., Confluent Cloud, GCP).</li><li>
Configuring brokers, ZooKeeper/KRaft mode, topics, partitions, replication factors, and security settings (e.g., SSL/TLS, SASL, ACLs) using Ansible, Terraform, or Confluent CLI.</li><li>
Setting up auxiliary components like Schema Registry, Kafka Connect clusters, and monitoring agents (e.g., JMX exporters) with automated scripts for reproducible environments.</li><li>
Performing initial health checks, load testing (e.g., with Kafka's performance tools), and integration with existing infrastructure (e.g., VPC peering, load balancers).</li></ul></li><li>
Operations &amp; Maintenance: Oversee monitoring, troubleshooting, performance tuning, and lifecycle management (upgrades, patching) of Kafka/Confluent instances; implement DevSecOps practices for CI/CD pipelines.</li><li>
Team Leadership: Mentor junior engineers, conduct code reviews, and drive technical proofs-of-concept (POCs); gather requirements and define standards for Kafka as a managed service (e.g., access controls, documentation).</li><li>
Optimization &amp; Innovation: Ensure high availability (&gt;99.99%), fault tolerance, and cost-efficiency; explore emerging features like Kafka Tiered Storage or Confluent Cloud integrations for AI workloads.</li><li>
Collaboration &amp; Delivery: Partner with cross-functional teams (data engineers, architects, product owners) to align streaming solutions with business goals; provide thought leadership on event-driven patterns.</li><li>
Security &amp; Compliance: Implement RBAC, encryption, and auditing; conduct root-cause analysis for incidents and ensure GDPR/HIPAA compliance in data flows.</li></ul>
Required Qualifications &amp; Skills<ul><li>
Bachelor's/Master's in Computer Science, Engineering, or related; certifications like Confluent Developer/Administrator a plus.</li><li>
10+ years in software engineering; 5+ years hands-on with Apache Kafka &amp; Confluent Platform (Cloud/Enterprise editions).</li><li>
Proficiency in Java/Scala/Python (8/11+); Kafka Streams/Connect/ksqlDB; Schema Registry; REST/gRPC APIs.</li><li>
Event-driven/microservices design; data pipeline optimization; handling high-volume streams (TB/day scale).</li><li>
Expertise in containerization (Docker/Kubernetes); CI/CD (Jenkins/GitHub Actions); Terraform/Ansible for IaC.</li><li>
Multi-cloud experience (AWS, GCP, Azure); monitoring tools (Prometheus, Grafana, Confluent Control Center).</li><li>
Experience with streaming integrations (e.g., Flink, Spark Streaming for CDC).
</li><li>Contributions to open-source Kafka projects or publications on streaming architectures.
</li><li>Knowledge of AI/ML data pipelines (e.g., Kafka + TensorFlow/PyTorch).
</li><li>Familiarity with observability tools and security (OAuth, Kerberos).
</li><li>Strong problem-solving, communication, and leadership; experience leading POCs and cross-team projects.</li><li>
Agile/Scrum leadership in fast-paced environments.</li><li>
Experience in client facing roles and leading teams.</li></ul>
eSAVL5n77F</div>

Lead Confluent Kafka Engineer / Architect

Systems Limited

<div>We are seeking a highly skilled Senior Data Engineer to design, build, and optimize scalable data pipelines and data platforms. The ideal candidate should have strong experience with cloud data engineering, ELT/ETL frameworks, and modern data technologies, enabling the business to make data-driven decisions efficiently.<h3 class="jobSectionHeader"> 
</h3>
<h3 class="jobSectionHeader">Responsibilities</h3>
<ul>
<li>Design, develop, and maintain scalable data pipelines for batch and real-time processing.
</li>
<li>Build and optimize data lake, data warehouse, and data mart architectures.
</li>
<li>Develop and enhance ETL/ELT workflows using best practices.
</li>
<li>Work with large datasets, ensuring data quality, integrity, and governance.
</li>
<li>Collaborate with Data Scientists, Analysts, and Product teams to understand data needs.
</li>
<li>Implement data modeling (Star, Snowflake, 3NF) for analytics and reporting.
</li>
<li>Optimize database performance, query tuning, and data storage solutions.
</li>
<li>Ensure data security, access control, and compliance standards.
</li>
<li>Lead technical design discussions and mentor junior data engineers.
</li>
<li>Work with CI/CD pipelines, version control, and automation tools.</li> 
</ul>
<h3 class="jobSectionHeader">Requirements:</h3>
<ul>
<li>Strong experience with cloud platforms: AWS / Azure / GCP.
</li>
<li>Hands-on experience in SQL, Python, and distributed data processing.
</li>
<li>Expertise in tools like Spark, Databricks, Hadoop, Hive, Airflow, Kafka or similar.
</li>
<li>Experience with data warehouses such as Snowflake, Redshift, BigQuery, or Synapse.
</li>
<li>Strong understanding of ETL/ELT, data modeling, and data architecture principles.
</li>
<li>Experience building real-time streaming pipelines (Kafka, Kinesis, EventHub).
</li>
<li>Proficiency with CI/CD, Git, DevOps workflows.
</li>
<li>Knowledge of data governance, security, and metadata management.</li></ul></div>

Managing Consultant Data Engineering

<div>We are looking for a skilled Data Engineer with hands-on experience in building, optimizing, and maintaining large-scale data pipelines on the Cloudera Hadoop platform. The candidate should have strong expertise in distributed systems, ETL development, data ingestion, and cluster operations. 


Responsibilities
<ul>
<li>Design, build, and maintain scalable data pipelines using Hadoop ecosystem tools (HDFS, Hive, Impala, Spark, Kafka, Sqoop, Oozie).
</li>
<li>Manage and optimize Cloudera clusters, including configuration, upgrades, performance tuning, and security.
</li>
<li>Develop and optimize Spark (PySpark/Scala) applications for batch and real-time processing.
</li>
<li>Configure and maintain Cloudera Manager, YARN, and Hive Metastore.
</li>
<li>Build data ingestion processes from various sources such as RDBMS, APIs, streaming systems, and cloud storage.
</li>
<li>Work closely with Data Architects to implement data models, governance, quality, and lineage.
</li>
<li>Implement best practices for data security, Kerberos, Ranger, and Sentry.
</li>
<li>Monitor, troubleshoot, and resolve production issues, ensuring high availability.
</li>
<li>Collaborate with analytics, BI, and product teams for data delivery and automation.</li> 
</ul>
<h3 class="jobSectionHeader">Requirements:</h3>
<ul>
<li>Strong experience with Cloudera CDH/CDP distributions.
</li>
<li>Hands-on expertise with Hadoop tools: 
HDFS, Hive, Impala, Spark, Kafka, Sqoop, Oozie, Zookeeper.
</li>
<li>Solid programming skills in Python / Scala / Java.
</li>
<li>Strong SQL experience and ability to optimize complex queries.
</li>
<li>Experience with ETL pipelines, data warehousing concepts, and data lake architectures.
</li>
<li>Knowledge of Linux, shell scripting, and DevOps tools (Git, Jenkins).
</li>
<li>Experience working with cloud environments (AWS/GCP/Azure) is a plus.
</li>
<li>Good understanding of distributed computing and big data performance optimization.</li></ul></div>

Senior Consultant Data Analytics

Softech Worldwide

<div>
 We're seeking passionate, driven, and self-motivated Technical Account Manager to join our
 
 
 Pre/Post-Sales team. If you're eager to grow in a dynamic environment and be part of a company that&rsquo;s at the forefront of AI innovation, this is your chance to hit the ground running.
 
 
 VIDIZMO is a USA-based technology company headquartered in Tysons, Virginia, and a Microsoft Solutions Partner in Data &amp; AI, Infrastructure, and Digital &amp; App Innovation. Offering a AI-Powered Intelligence Hub, we empower Fortune 500 companies, large enterprises, governments, and the public sector to securely manage, analyze, and govern their data with complete control and compliance.
 
 
 Our Multimodal AI Data Intelligence Platform leverages Large Language Models (LLMs) and RAG (Retrieval-Augmented Generation) to deliver powerful capabilities such as auto-tagging, redaction, content summarization, OCR, translation, subtitle creation, object detection and tracking, content search, sentiment and emotion analysis, topic extraction, document classification, and facial attribute detection.
 
 
 
 Key Responsibilities
 
 
 
 Job Description:
 
 
 <ul>
 <li>
 Develop an in-depth understanding of VIDIZMO Enterprise Video, Digital Media &amp; Digital Evidence Management Solutions, its target industries, markets, and buyer persona
 </li>
 <li>
 Creating and deliver technical product and solution demonstrations, presentations, and network designs to our prospects/customers
 </li>
 <li>
 Deliver technical pre-sales consultations, and product capability assessments with potential customers, document customer requirements &amp; proposing a winning solution
 </li>
 <li>
 Provide support in responding to technical proposals for RFIs/RFPs, creating and maintaining technical documents aimed towards relaying VIDIZMO solutions
 </li>
 <li>
 Provide technical support to Business Development &amp; Sales team
 </li>
 <li>
 Delivering product and solution training to channel partners, integrators &amp; end-users
 </li>
 <li>
 Develop &amp; maintain a broad knowledge of live &amp; on-demand streaming video solutions, streaming video formats &amp; other streaming video platform providers (competitors)
 </li>
 <li>
 Develop an understanding of unicast &amp; multicast networks, streaming technologies, video content caching, streaming protocols such as RTSP, RTMP, HLS, Microsoft Smooth Streaming, IIS, firewall configurations required to streaming video, video formats, encoding, variable bit rate, adaptive bit rate, video playback in browsers, HTML5 playback, VIDIZMO iOS &amp; Android apps, Azure, AWS Media &amp; Cognitive Services and Azure/AWS Marketplaces
 </li>
 <li>
 Deliver Proof of Concepts (POC), Pilots to customers, channel partners, integrators, and end-users
 </li>
 <li>
 Articulating competitive differentiators and value propositions at both technical and business levels
 
 
 </li>
 </ul>
 
 Essential Skills
 
 
 
 <ul>
 <li>
 Strong communication skills, both written and oral
 </li>
 <li>
 Strong basic technical understanding of Networks, Systems, Software &amp; Cloud
 </li>
 <li>
 Ability to Deploy VIDIZMO Solution in various topologies and designs
 </li>
 <li>
 Good documentation skills for designing solution by using illustrations, diagrams and written documentation
 </li>
 <li>
 Must be detailed oriented and a problem solver
 </li>
 <li>
 Demonstrate proactivity &amp; ownership
 </li>
 <li>
 Cold calling experience
 
 
 </li>
 </ul>
 
 Preferred Skills
 
 
 
 <ul>
 <li>
 Strong understanding of various technologies used in VIDIZMO such as Video, Streaming, On-demand, CDN, Cloud, Cognitive services, Artificial Intelligence, Networks and Business applications
 </li>
 <li>
 Has good understanding of the Software Development lifecycle, Analysis, Quality Assurance, Product &amp; Project Management
 </li>
 <li>
 Azure or AWS Certifications
 </li>
 <li>
 Hands-on experience with Deploying Enterprise IT/Technology Solution &amp; Products
 </li>
 <li>
 Hands on experience with Azure or AWS, both IaaS and PaaS
 
 
 </li>
 </ul>
 
 Required Qualifications
 
 
 
 <ul>
 <li>
 BS/BSc or MS/MSc in Computer Science
 </li>
 <li>
 2-3 years of experience is required
 </li>
 <li>
 Fluent in English communication, written &amp; verbal
 </li>
 <li>
 Authorized to Work Legally in the mentioned countries
 
 
 </li>
 </ul>
 Multiple Locations: Pakistan, India, UAE, Australia, Canada &amp; USA
 
 
 Worldwide Office Hours : 1) US Eastern (UTC -5) , 2) USA PST ( UTC -8) , 3) Central European Standard ( UTC +1 ) , 4) Japan Standard time ( UTC +9)
 
 
 Benefits: Medical, Health/Life Insurance, Annual Paid Leaves, Commissions, Bonus (benefits vary based on location)
</div>

Technical Account Manager

10Pearls

<div>About 10Pearls
10Pearls is an end-to-end digital technology services partner helping businesses utilize technology as a competitive advantage. We help our customers digitalize their existing business, build innovative new products, and augment their existing teams with high-performance team members.
Our broad expertise in product management, user experience/design, cloud architecture, software development, data insights and intelligence, cyber security, emerging tech, and quality assurance ensures that we are delivering solutions that address business needs.
10Pearls is proud to have diverse clientele, including large enterprises, SMBs, and high-growth startups. We work with clients across industries including healthcare/life sciences, education, energy, communications/media, financial services, and hi-tech. Our many long-term, successful partnerships are built upon trust, integrity, and successful delivery and execution.
Role Overview
We are seeking a highly skilled Backend Engineer with strong expertise in real-time media streaming, low-latency communication, and distributed backend systems using C++ or Golang. The ideal candidate must have deep hands-on experience with streaming protocols, real-time communication frameworks, and high-performance backend engineering. This role involves designing and building backend microservices that power real-time audio/video communication, streaming pipelines, and low-latency data flows across distributed systems.
Responsibilities<ul><li>
Design, develop, and maintain high-performance backend services using Go or C++.</li><li>
Build and optimize real-time microservices that support low-latency communication across a distributed architecture.</li><li>
Develop and enhance WebSocket, REST, and real-time APIs used by client-facing applications.</li><li>
Work extensively with media streaming technologies, including RTP, SRTP, WebRTC, and related protocols.</li><li>
Implement and optimize WebRTC SFUs (Selective Forwarding Units) for scalable audio/video streaming.</li><li>
Integrate and optimize media workflows using tools and libraries such as FFmpeg and GStreamer.</li><li>
Work with databases such as MySQL, Redis, MongoDB, and design efficient data models for real-time workloads.</li><li>
Ensure backend services meet high standards of security, reliability, and performance using modern DevOps and SRE practices.</li><li>
Collaborate closely with Frontend, DevOps, and Product teams to deliver end-to-end features.</li><li>
Write clean, well-documented, and maintainable code following best engineering practices.</li><li>
Diagnose performance bottlenecks and optimize backend systems for scalability and speed.</li></ul>
Requirements<ul><li>
5+ years of hands-on backend development experience using Go and/or C++.</li><li>
Strong expertise in real-time media streaming, including one or more of the following:</li><li>
RTP / SRTP</li><li>
WebRTC (MUST-HAVE)</li><li>
WebRTC SFU implementation</li><li>
Deep understanding of video/audio processing, with experience using FFmpeg, GStreamer, or similar frameworks.</li><li>
Understanding of networking concepts, including TCP, UDP, ICE, STUN/TURN, and WebSockets.</li><li>
Strong understanding of concurrency, asynchronous programming, and performance tuning.</li><li>
Experience building and consuming RESTful and WebSocket APIs.</li><li>
Hands-on experience with message queues such as Kafka, NATS, or RabbitMQ.</li><li>
Experience working with cloud platforms (AWS, GCP, Azure).</li><li>
Strong problem-solving skills and the ability to work on low-latency, high-throughput systems.</li><li>
Proficient in C++ or Go for media streaming</li><li>
Familiarity with monitoring, logging, and tracing tools (e.g., Prometheus, Grafana, OpenTelemetry) is a plus.</li><li>
A good understanding of containerization and orchestration using Docker and Kubernetes is preferred.</li><li>
Experience working with real-time communication platforms is an added advantage.</li></ul>
pVmxbZwqt8</div>

Senior/Staff Software Consultant - Golang/C++

Explore 871+ Associate data architect data lake data streaming in Karachi this December. Find roles across top companies and apply now to boost your career.

871 associate data architect data lake data streaming jobs IN Karachi

Associate Data Architect - Data Lake/Data Streaming

Associate Data Architect - Data Lake/Data Streaming

Lead Confluent Kafka Engineer / Architect

Managing Consultant Data Engineering

Senior Consultant Data Analytics

Technical Account Manager

Senior/Staff Software Consultant - Golang/C++