data scientist

data engineer

data analyst

aws data engineer

ai data scientist

associate data architect data lake data streaming

Confiz

<div><p>We are seeking a Data Engineer with 8+ years of hands-on experience and a strong background in real-time and batch data processing, containerization, and cloud-based data orchestration. This role is ideal for someone passionate about building robust, scalable, and efficient data pipelines, and who thrives in agile, collaborative environments.</p>
<p><b>Key Responsibilities</b></p>
<ul><li>Design, build, and maintain real-time data pipelines using streaming frameworks such as Kafka, Apache Flink, and Spark Structured Streaming.</li>
<li>Develop batch processing workflows with Apache Spark (PySpark)</li>
<li>Orchestrate and schedule data workflows using orchestration frameworks such as Apache Airflow and Azure Data Factory</li>
<li>Containerize applications using Docker, manage deployments with Helm, and run them on Kubernetes</li>
<li>Implement modern storage solutions using open formats such as Parquet, Delta Lake, and Apache Iceberg</li>
<li>Build high-performance analytics engines using tools like Trino or Presto</li>
<li>Collaborate with DevOps to manage infrastructure with Terraform and integrate with CI/CD pipelines via Azure DevOps</li>
<li>Ensure data quality and consistency using tools like Great Expectations</li>
<li>Write modular, well-tested, and maintainable Python and SQL code</li>
<li>Develop an observability layer to monitor and optimize performance across data pipelines</li>
<li>Lead and mentor a team of data engineers</li>
<li>Participate in agile ceremonies and contribute to sprint planning and reviews</li>
</ul>
<p><b>Required Skills &amp; Experience</b></p>
<ul><li>Advanced Python programming with a strong focus on modular and testable code</li>
<li>Strong knowledge of SQL and experience working with large-scale datasets</li>
<li>Hands-on experience with at least one major cloud platform (Azure preferred)</li>
<li>Solid experience with real-time data processing (Kafka, Flink, or Spark Streaming)</li>
<li>Expertise in Apache Spark (PySpark) for batch processing</li>
<li>Experience implementing lakehouse architectures and working with columnar storage (e.g., ClickHouse)</li>
<li>Proficient in using Azure Data Factory or Apache Airflow for data orchestration</li>
<li>Experience in building APIs to expose large datasets</li>
<li>Solid experience with Docker, Kubernetes, and Helm</li>
<li>Familiarity with data lake open formats such as Parquet, Delta Lake, and Iceberg</li>
<li>Basic experience with Terraform for infrastructure provisioning</li>
<li>Practical experience with data quality frameworks (e.g., Great Expectations)</li>
<li>Comfortable working in agile development teams</li>
<li>Proven ability in debugging and performance tuning of streaming and batch data jobs</li>
<li>Experience with AI-driven tools (e.g., text-to-SQL) is a plus</li>
<li>Bachelor&rsquo;s degree in computer science or related discipline</li>
</ul>
<p>We have an amazing team of 700+ individuals working on highly innovative enterprise projects &amp; products. Our customer base includes Fortune 100 retail and CPG companies, leading store chains, fast-growth fintech, and multiple Silicon Valley startups.</p>
<p>What makes Confiz stand out is our focus on processes and culture. Confiz is <b>ISO 9001:2015</b> (QMS), <b>ISO 27001:2022</b> (ISMS), <b>ISO 20000-1:2018</b> (ITSM), <b>ISO 14001:2015</b> (EMS), <b>ISO 45001:2018 </b>(OHSMS) Certified. We have a vibrant culture of learning via collaboration and making workplace fun.</p>
<p>People who work with us work with cutting-edge technologies while contributing success to the company as well as to themselves.</p>
<p>To know more about Confiz Limited, visit: https://www.linkedin.com/company/confiz-pakistan/</p></div><p></p>

Associate Data Architect - Data Lake/Data Streaming

<div>
 We are seeking a Data Engineer with 8+ years of hands-on experience and a strong background in real-time and batch data processing, containerization, and cloud-based data orchestration. This role is ideal for someone passionate about building robust, scalable, and efficient data pipelines, and who thrives in agile, collaborative environments.
 <br/>
 <br/>
 <strong>
  Key Responsibilities
  <br/>
  <br/>
 </strong>
 <ul>
  <li>
   Design, build, and maintain real-time data pipelines using streaming frameworks such as Kafka, Apache Flink, and Spark Structured Streaming.
  </li>
  <li>
   Develop batch processing workflows with Apache Spark (PySpark)
  </li>
  <li>
   Orchestrate and schedule data workflows using orchestration frameworks such as Apache Airflow and Azure Data Factory
  </li>
  <li>
   Containerize applications using Docker, manage deployments with Helm, and run them on Kubernetes
  </li>
  <li>
   Implement modern storage solutions using open formats such as Parquet, Delta Lake, and Apache Iceberg
  </li>
  <li>
   Build high-performance analytics engines using tools like Trino or Presto
  </li>
  <li>
   Collaborate with DevOps to manage infrastructure with Terraform and integrate with CI/CD pipelines via Azure DevOps
  </li>
  <li>
   Ensure data quality and consistency using tools like Great Expectations
  </li>
  <li>
   Write modular, well-tested, and maintainable Python and SQL code
  </li>
  <li>
   Develop an observability layer to monitor and optimize performance across data pipelines
  </li>
  <li>
   Lead and mentor a team of data engineers
  </li>
  <li>
   Participate in agile ceremonies and contribute to sprint planning and reviews
   <br/>
   <br/>
  </li>
 </ul>
 <strong>
  Required Skills &amp; Experience
  <br/>
  <br/>
 </strong>
 <ul>
  <li>
   Advanced Python programming with a strong focus on modular and testable code
  </li>
  <li>
   Strong knowledge of SQL and experience working with large-scale datasets
  </li>
  <li>
   Hands-on experience with at least one major cloud platform (Azure preferred)
  </li>
  <li>
   Solid experience with real-time data processing (Kafka, Flink, or Spark Streaming)
  </li>
  <li>
   Expertise in Apache Spark (PySpark) for batch processing
  </li>
  <li>
   Experience implementing lakehouse architectures and working with columnar storage (e.g., ClickHouse)
  </li>
  <li>
   Proficient in using Azure Data Factory or Apache Airflow for data orchestration
  </li>
  <li>
   Experience in building APIs to expose large datasets
  </li>
  <li>
   Solid experience with Docker, Kubernetes, and Helm
  </li>
  <li>
   Familiarity with data lake open formats such as Parquet, Delta Lake, and Iceberg
  </li>
  <li>
   Basic experience with Terraform for infrastructure provisioning
  </li>
  <li>
   Practical experience with data quality frameworks (e.g., Great Expectations)
  </li>
  <li>
   Comfortable working in agile development teams
  </li>
  <li>
   Proven ability in debugging and performance tuning of streaming and batch data jobs
  </li>
  <li>
   Experience with AI-driven tools (e.g., text-to-SQL) is a plus
  </li>
  <li>
   Bachelor&rsquo;s degree in computer science or related discipline
   <br/>
   <br/>
  </li>
 </ul>
 We have an amazing team of 700+ individuals working on highly innovative enterprise projects &amp; products. Our customer base includes Fortune 100 retail and CPG companies, leading store chains, fast-growth fintech, and multiple Silicon Valley startups.
 <br/>
 <br/>
 What makes Confiz stand out is our focus on processes and culture. Confiz is
 <strong>
  ISO 9001:2015
 </strong>
 (QMS),
 <strong>
  ISO 27001:2022
 </strong>
 (ISMS),
 <strong>
  ISO 20000-1:2018
 </strong>
 (ITSM),
 <strong>
  ISO 14001:2015
 </strong>
 (EMS),
 <strong>
  ISO 45001:2018
 </strong>
 (OHSMS) Certified. We have a vibrant culture of learning via collaboration and making workplace fun.
 <br/>
 <br/>
 People who work with us work with cutting-edge technologies while contributing success to the company as well as to themselves.
 <br/>
 <br/>
 To know more about Confiz Limited, visit: https://www.linkedin.com/company/confiz-pakistan/
</div>


TechBiz Global GmbH

<div><p>At TechBiz Global, we are providing recruitment service to our TOP clients from our portfolio. We are currently seeking an
</p><p><b>Data Architect</b> to join one of our <b>clients</b>' teams. If you're looking for an exciting opportunity to grow in a innovative environment, this could be the perfect fit for you.</p><p><b>
Role Summary:</b></p><p>
The Data Architect will design and govern the complete data ecosystem for the ValueX platform &mdash; including data ingestion, processing, modeling, storage, orchestration, and governance.
</p><p>They will define the <b>data architecture blueprint</b> supporting the system&rsquo;s core modules.</p><p><b>
1. Customer Segmentation
</b></p><p><b>2. Business Decisioning &amp; Offer Assignment Engine
</b></p><p><b>3. Real-Time Offer Orchestration Module (ROOM)
</b></p><p><b>4. ML/AI Model Management &amp; Simulation Engine</b>
</p><p>The architect will ensure scalability to handle 50 million+ customer profiles, real-time event streams, and ML-driven decisioning &mdash; with a focus on performance, cost optimization, and maintainability.</p><p></p><p><b>
Key Responsibilities:
</b></p><p><b>Data Architecture &amp; Design</b></p><ul><li><p>
Define the end-to-end data architecture across batch, streaming, and real-time layers.</p></li><li><p>
Design Customer 360 and Offer 360 data models, including feature store and historical layers.
</p></li><li><p>Establish logical, physical, and semantic data models to enable segmentation, scoring, and orchestration.
</p></li><li><p>Define data contracts for Kafka topics, API payloads, and Adobe/CDP integrations.
</p></li><li><p>Set up data versioning and lineage frameworks to track data provenance</p></li></ul><p></p><p><b>
Data Ingestion &amp; Integration</b></p><ul><li><p>
Architect data ingestion pipelines from multiple telco sources: OCS, CRM, Kenan, Billing, DWH, Adobe AEP, Pricefx, and external APIs.
</p></li><li><p>Define patterns for real-time event ingestion (recharge, offer purchase, balance check).
</p></li><li><p>Standardize data access through APIs or data products for downstream modules.
</p></li><li><p>Design connectors for cloud storage (e.g., S3, Delta Lake) and integration middleware (e.g., n8n, DecisionRules.io, KNIME).</p></li></ul><p></p><p><b>
Data Management &amp; Governance</b>
</p><ul><li><p>Define and enforce data quality, lineage, catalog, and access policies.
</p></li><li><p>Establish metadata management frameworks (e.g., DataHub, Collibra, Amundsen).
</p></li><li><p>Set up data validation and DQ frameworks (Great Expectations, Deequ).
</p></li><li><p>Govern data partitioning, schema evolution, retention, and archiving strategies.
</p></li><li><p>Ensure compliance with data privacy and regulatory standards (e.g., PDPA, GDPR, local telecom data policies).</p></li></ul><p></p><p><b>
Scalability, Cost &amp; Performance</b>
</p><ul><li><p>Design for high performance and cost-efficient scalability (25M 50M 75M customers).
</p></li><li><p>Optimize compute/storage balance across environments (Dev, UAT, Prod).
</p></li><li><p>Define data lakehouse optimization strategies (Z-Order, Delta caching, compaction).
</p></li><li><p>Monitor and manage query performance, cluster sizing, and job orchestration costs.</p></li></ul><p></p><p><b>
Collaboration &amp; Governance</b>
</p><ul><li><p>Work closely with Data Engineers, Data Scientists, and Application Developers to ensure architectural alignment.
</p></li><li><p>Lead architecture review boards and maintain data design documentation (ERD, flow diagrams, schema registry).
</p></li><li><p>Serve as technical liaison between business stakeholders, data teams, and platform vendors (Databricks, Adobe, Pricefx).
</p></li><li><p>Provide best practices and design patterns for model deployment, retraining, and data lifecycle management.</p></li></ul><ul><li><p>
10+ years in Data Architecture and Data Platform Design</p></li><li><p>
Data Architecture-Data lakehouse design, data modeling (dimensional, normalized, semantic), schema management.</p></li><li><p>
ETL/ELT &amp; Orchestration-Databricks, Snowflake, dbt, Airflow, AWS Glue, Azure Data Factory.</p></li><li><p>
Streaming &amp; Real-Time-Apache Kafka, Spark Streaming, Kinesis, Flink</p></li><li><p>
Data Modeling-Customer 360, Offer 360, Transaction 360, Feature Store</p></li><li><p>
Cloud Platforms-AWS (S3, Glue, Lambda, EMR), Azure (ADF, Synapse), or GCP (BigQuery, Dataflow)</p></li><li><p>
Storage &amp; Compute-Delta Lake, Parquet, Iceberg, Snowflake</p></li><li><p>
Data Quality &amp; Governance-Great Expectations, Deequ, DataHub, Collibra</p></li><li><p>
Programming &amp; Scripting-Python, SQL, PySpark, YAML</p></li><li><p>
API &amp; Integration Design-REST, GraphQL, Kafka Connect, JSON schema</p></li><li><p>
Security &amp; Compliance-IAM, encryption (KMS), access control, masking, PDPA compliance
</p></li></ul><p></p><p><b>Preferred (Nice-to-Have)</b></p><ul><li><p>
Telecom industry experience (recharge, balance, offer, churn, usage data).
</p></li><li><p>Experience integrating with Adobe Experience Platform (AEP) or Pricefx.
</p></li><li><p>Knowledge of DecisionRules.io, KNIME, or n8n for workflow orchestration.
</p></li><li><p>Familiarity with AI/ML pipelines and MLOps frameworks (MLflow, SageMaker).
</p></li><li><p>Exposure to knowledge graphs (Neo4j, GraphFrames) for segmentation and recommendation.
</p></li><p></p><li><p><b>Educational Background Bachelor&rsquo;s or Master&rsquo;s degree in Computer Science, Data Engineering, or related fields.
</b></p></li><li><p><b>Certifications in AWS/Azure Data Architect, Databricks Certified Data Engineer, or Snowflake Architect are preferred.</b></p></li></ul></div>

Data Architect – Dynamic Pricing & Decision Intelligence Platform

Tata Consultancy Services (TCS)

<div></div><div><div><div><p><div><div><div><div><p>The Data Architect will play a vital role in fostering a data-driven culture and enhancing data capabilities</p>
&#65279;&#65279;<ul><li>Experience within data and information management, including data modelling, metadata-management, data classification, data privacy and data security.</li>
&#65279;&#65279;<li>Experience within data platform design, implementation, and management, including technologies for data acquisition, storage and processing.</li>
&#65279;&#65279;<li>Experience within or foundational knowledge of modern data architecture frameworks and strategies, including but not limited to data fabric, data mesh, and data lakehouse.</li>
&#65279;&#65279;<li>Experience or fundamental knowledge about data engineering practices and technologies, including streaming and batch processing, data modelling for analytics, data lake/warehouse/lakehouse design, data quality/observability and DataOps.</li>
&#65279;&#65279;<li>Experience within or fundamental knowledge about cloud architecture, software engineering practices, identity and access management and API management.</li>
&#65279;&#65279;<li>Foundational knowledge of enterprise architecture principles and frameworks.</li>
&#65279;&#65279;<li>Strong communication and facilitation skills with the ability to elicit requirements from various stakeholders across the organization</li>
</ul></div></div><br>
</div></div></p></div></div><div><div><div><b>Location</b></div>
<div>Hyderabad</div>
</div><div><div><b>Job Function</b></div>
<div>TECHNOLOGY</div>
</div><div><div><b>Role</b></div>
<div>Analyst</div>
</div><div><div><b>Job Id</b></div>
<div>384431</div>
</div><div><div><b>Desired Skills</b></div>
<div>Azure | SQL</div>
</div></div><p>Desired Candidate Profile
</p><div><div><b>Qualifications</b> : BACHELOR OF ENGINEERING</div></div></div><div></div>

Data Architect

VALUEMOMENTUM Inc

<div><div><b>Job Title:</b> Databricks Engineers (SSE/Lead/Architect)
</div><div><b>Job ID:</b> POS-16819</div>
<div><b>Primary Skill:</b> AWS Services</div>
<div><b>Location:</b> Hyderabad
</div><div><b>Experience:</b> 10.00</div><br>
<br>
<div><p><b>Key Skills:</b>Databricks &amp; PySpark<br>
<b>Mode of Work:</b> Work From Office</p><p><b>
Experience</b><b>: </b>4-16 Years</p><p></p><p><b><br>
About the job:</b><br>
Join ValueMomentum&rsquo;s Data Engineering team to design, build and scale Azure Databricks lakehouse solutions for P&amp;C insurance customers. This hiring drive covers three experience bands &mdash; SSE, Tech Lead and Architect &mdash; so apply to the level that matches your background. Across levels, you&rsquo;ll own end-to-end data platform delivery: authoring ETL/ELT pipelines (ADF &amp; Databricks), developing PySpark transformations, enabling Delta Lake patterns, implementing governance, and delivering production-grade CI/CD and cost-optimised architectures.</p><p></p><p><b><br>
Know your team:</b></p><p>
At ValueMomentum Technology Solution Centers, we are a team of passionate software engineering teams who thrive on tackling complex business challenges with innovative solutions catering to the P&amp;C insurance value chain. We achieve this through a strong software engineering foundation and by continuously refining our processes, methodologies, tools, agile delivery teams, and core engineering archetypes. Our core expertise lies in six key areas: <b>Platforms, Infra/Cloud, Application, Data, Core, and Quality Assurance</b>.</p><p></p><p><br>
Join our team that invests in your growth. Our Infinity Program empowers you to build your career with role-specific skill development, leveraging immersive learning platforms. You'll have the opportunity to work with some of the best minds serving insurance customers in the US, UK and Canadian markets.</p><p></p><p><b><br>
Responsibilities:</b></p><ul><li>
Design, develop and operate data pipelines using Azure Databricks, ADF and ADLS.</li><li>
Implement transformations in SQL, Python &amp; PySpark; build testable, modular code.</li><li>
Collaborate with product, BI, infra and QA teams to deliver data models and analytics-ready datasets.</li><li>
Integrate pipelines into CI/CD, monitor jobs, troubleshoot performance and ensure data quality.</li><li>
Follow Agile practices and contribute to sprint planning, design reviews and documentation.</li></ul><p></p><p><b><br>
Requirements:</b></p><ul><li>
Hands-on Databricks &amp; PySpark experience.</li><li>
Strong SQL and Python skills.</li><li>
Experience with Azure Data Factory, ADLS (Data Lake) and Azure DevOps/Git pipelines.</li><li>
Working knowledge of Delta Lake concepts, CI/CD for data workloads and Agile delivery.</li><li>
Good debugging, performance tuning and data modelling capabilities.</li></ul><p></p><p><b><br>
Good To Have:
</b><br>
Candidates with these additional skills on their resumes have an added advantage when their profiles are being assessed. Some of them are as follows:</p><ul><li>
Experience with Auto Loader, Delta Live Tables (DLT), Unity Catalogue, and Starburst.</li><li>
Migration experience from Hadoop/SSIS/Informatica/Vertica to Databricks.</li><li>
Azure Synapse, Delta Lake optimisations, streaming patterns.</li><li>
Insurance / P&amp;C domain experience and pre-sales/solutioning exposure.</li></ul><br>
</div><br>
<div><div><b>About the Company:</b><br>
<br>
ValueMomentum is a leading solutions provider for the global property and casualty insurance industry. The company helps insurers stay ahead with sustained growth and high performance, enhancing stakeholder value and fostering resilient societies. Having served over 100 insurers, ValueMomentum is one of the largest services providers exclusively focused on the insurance industry.<br>
</div><br>
<div><b>Benefits:</b><br>
<br>
We at ValueMomentum offer you a congenial environment to work and grow in the company of experienced professionals. Some benefits that are available to you are:
<ul><li>Competitive compensation package.</li>
<li><i>Career Advancement: </i>Individual Career Development, coaching and mentoring programs for professional and leadership skill development.</li>
<li>Comprehensive training and certification programs.</li>
<li><i>Performance Management: </i>Goal Setting, continuous feedback and year-end appraisal. Reward &amp; recognition for the extraordinary performers.</li></ul></div></div></div>

<div>
 We are seeking a Data Engineer with 5+ years of hands-on experience and a strong background in real-time and batch data processing, containerization, and cloud-based data orchestration. This role is ideal for someone passionate about building robust, scalable, and efficient data pipelines, and who thrives in agile, collaborative environments.
 <br/>
 <br/>
 <strong>
  Key Responsibilities
  <br/>
  <br/>
 </strong>
 <ul>
  <li>
   Design, build, and maintain real-time data pipelines using streaming frameworks such as Kafka, Apache Flink, and Spark Structured Streaming.
  </li>
  <li>
   Develop batch processing workflows with Apache Spark (PySpark)
  </li>
  <li>
   Orchestrate and schedule data workflows using orchestration frameworks such as Apache Airflow and Azure Data Factory
  </li>
  <li>
   Containerize applications using Docker, manage deployments with Helm, and run them on Kubernetes
  </li>
  <li>
   Implement modern storage solutions using open formats such as Parquet, Delta Lake, and Apache Iceberg
  </li>
  <li>
   Build high-performance analytics engines using tools like Trino or Presto
  </li>
  <li>
   Collaborate with DevOps to manage infrastructure with Terraform and integrate with CI/CD pipelines via Azure DevOps
  </li>
  <li>
   Ensure data quality and consistency using tools like Great Expectations
  </li>
  <li>
   Write modular, well-tested, and maintainable Python and SQL code
  </li>
  <li>
   Develop an observability layer to monitor and optimize performance across data pipelines
  </li>
  <li>
   Participate in agile ceremonies and contribute to sprint planning and reviews
   <br/>
   <br/>
  </li>
 </ul>
 <strong>
  Required Skills &amp; Experience
  <br/>
  <br/>
 </strong>
 <ul>
  <li>
   Advanced Python programming with a strong focus on modular and testable code
  </li>
  <li>
   Strong knowledge of SQL and experience working with large-scale datasets
  </li>
  <li>
   Hands-on experience with at least one major cloud platform (Azure preferred)
  </li>
  <li>
   Solid experience with real-time data processing (Kafka, Flink, or Spark Streaming)
  </li>
  <li>
   Expertise in Apache Spark (PySpark) for batch processing
  </li>
  <li>
   Experience implementing lakehouse architectures and working with columnar storage (e.g., ClickHouse)
  </li>
  <li>
   Proficient in using Azure Data Factory or Apache Airflow for data orchestration
  </li>
  <li>
   Experience in building APIs to expose large datasets
  </li>
  <li>
   Solid experience with Docker, Kubernetes, and Helm
  </li>
  <li>
   Familiarity with data lake open formats such as Parquet, Delta Lake, and Iceberg
  </li>
  <li>
   Basic experience with Terraform for infrastructure provisioning
  </li>
  <li>
   Practical experience with data quality frameworks (e.g., Great Expectations)
  </li>
  <li>
   Comfortable working in agile development teams
  </li>
  <li>
   Proven ability in debugging and performance tuning of streaming and batch data jobs
  </li>
  <li>
   Experience with AI-driven tools (e.g., text-to-SQL) is a plus
  </li>
  <li>
   Bachelor&rsquo;s degree in computer science or related discipline
   <br/>
   <br/>
  </li>
 </ul>
 We have an amazing team of 700+ individuals working on highly innovative enterprise projects &amp; products. Our customer base includes Fortune 100 retail and CPG companies, leading store chains, fast-growth fintech, and multiple Silicon Valley startups.
 <br/>
 <br/>
 What makes Confiz stand out is our focus on processes and culture. Confiz is
 <strong>
  ISO 9001:2015
 </strong>
 (QMS),
 <strong>
  ISO 27001:2022
 </strong>
 (ISMS),
 <strong>
  ISO 20000-1:2018
 </strong>
 (ITSM),
 <strong>
  ISO 14001:2015
 </strong>
 (EMS),
 <strong>
  ISO 45001:2018
 </strong>
 (OHSMS) Certified. We have a vibrant culture of learning via collaboration and making workplace fun.
 <br/>
 <br/>
 People who work with us work with cutting-edge technologies while contributing success to the company as well as to themselves.
 <br/>
 <br/>
 To know more about Confiz Limited, visit: https://www.linkedin.com/company/confiz-pakistan/
</div>


Senior/Principal Data Engineer

Explore 3020+ Associate data architect data lake data streaming in Pakistan this December. Find roles across top companies and apply now to boost your career.

3020 associate data architect data lake data streaming jobs IN Pakistan

Associate Data Architect - Data Lake/Data Streaming

Associate Data Architect - Data Lake/Data Streaming

Associate Data Architect - Data Lake/Data Streaming

Data Architect – Dynamic Pricing & Decision Intelligence Platform

Data Architect

Data Architect

Senior/Principal Data Engineer