big data platform architect

N Consulting Limited

LocationBloomsbury Square, London, United Kingdom# Technology Architect - Data Engineering at N Consulting LtdLocationBloomsbury Square, London, United KingdomSalary£400 - £500 /dayJob TypeContractDate PostedSeptember 17th, 2025Apply Now We're Hiring: Technology Architect - Data Engineering (Hybrid - London) Contract: 6 months Work Mode: Hybrid (12 days WFO/month)We are looking for an experienced Technology Architect with deep expertise in PySpark, ADF, and Databricks to lead and design data engineering solutions for our client. What You'll Do: Lead technical design using Medallion architecture and Azure ServicesCreate conceptual diagrams, source-to-target mapping, and dimensional data modelsGuide the data engineering team on development with ADF, Databricks, Python, JSON, and YAMLImplement performance optimization strategies during data transformationDeploy code artifacts using GitHub Workflows/CI-CD pipelines Collaborate effectively with team members and stakeholdersOptional: Work with Log Analytics and KQL queries Must-Have Skills: 10+ years of experience in Data Engineering Hands-on experience with PySpark, ADF, Databricks, SQL Strong understanding of dimensional modeling, normalization, schema design, and data harmonization Experience with Erwin and data modeling toolsExcellent communication, problem-solving, and client-facing skills Why Join Us: Work in a dynamic, collaborative environment Hands-on experience with cutting-edge Azure data engineering technologies Hybrid setup providing flexibility while staying connected to the team<img src="https://www.jobg8.com/Tracking.aspx?8I79n2dBKkRvNJmK9ygcKi%2f0hzunNcynz" width="0" height="0" />

Technology Architect - Data Engineering

Microsoft

<div>
 Are you insatiably curious, deeply passionate about the realm of databases and analytics, and ready to tackle complex challenges in a dynamic environment in the era of AI? If so, we invite you to join our team as a Cloud &amp; AI Solution Engineer in Data Platform for
 <strong>
  Enterprise customers
 </strong>
 at Microsoft. Here, you'll be at the forefront of innovation, working on cutting-edge projects that leverage the latest technologies to drive meaningful impact. Join us and be part of a team that thrives on collaboration, creativity, and continuous learning.
 <br/>
 <br/>
 Databases &amp; Analytics is a growth opportunity for Microsoft Azure, as well as its partners and customers. It includes a rich portfolio of products including IaaS and PaaS services on the Azure Platform in the age of AI. These technologies empower customers to build, deploy, and manage database and analytics applications in a cloud-native way.
 <br/>
 <br/>
 As an
 <strong>
  Data Platform
 </strong>
 <strong>
  Solution Engineer
 </strong>
 , you will play a pivotal role in helping enterprises unlock the full potential of Microsoft&rsquo;s cloud database and analytics stack across every stage of deployment. You&rsquo;ll collaborate closely with engineering leaders and platform teams to accelerate the Fabric Data Platform, including Azure Databases and Analytics, through hands-on engagements like Proof of Concepts, hackathons, and architecture workshops. This opportunity will allow you to accelerate your career growth, develop deep business acumen, hone your technical skills, and become adept at solution design and deployment. You&rsquo;ll guide customers through secure, scalable solution design, influence technical decisions, and accelerate database and analytics migration into their deployment workflows. In summary, you&rsquo;ll help customers modernize their data platform and realize the full value of Microsoft&rsquo;s platform, all while enjoying flexible work opportunities.
 <br/>
 <br/>
 As a trusted technical advisor, you&rsquo;ll guide customers through secure, scalable solution design, influence technical decisions, and accelerate database and analytics migration into their deployment workflows. In summary, you&rsquo;ll help customers modernize their data platform and realize the full value of Microsoft&rsquo;s platform.
 <br/>
 <br/>
 Microsoft&rsquo;s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.
 <br/>
 <br/>
 <strong>
  Responsibilities
  <br/>
  <br/>
 </strong>
 <ul>
  <li>
   Drive technical sales with decision makers using demos and PoCs to influence solution design and enable production deployments.
  </li>
  <li>
   Lead hands-on engagements&mdash;hackathons and architecture workshops&mdash;to accelerate adoption of Microsoft&rsquo;s cloud platforms.
  </li>
  <li>
   Build trusted relationships with platform leads, co-designing secure, scalable architectures and solutions
  </li>
  <li>
   Resolve technical blockers and objections, collaborating with engineering to share insights and improve products.
  </li>
  <li>
   Maintain deep expertise in Analytics Portfolio: Microsoft Fabric (OneLake, DW, real-time intelligence, BI, Copilot), Azure Databricks, Purview Data Governance and Azure Databases: SQL DB, Cosmos DB, PostgreSQL.
  </li>
  <li>
   Maintain and grow expertise in on-prem EDW (Teradata, Netezza, Exadata), Hadoop &amp; BI solutions.
  </li>
  <li>
   Represent Microsoft through thought leadership in cloud Database &amp; Analytics communities and customer forums
   <br/>
   <br/>
  </li>
 </ul>
 <strong>
  Qualifications
  <br/>
  <br/>
 </strong>
 <strong>
  Required Qualifications (RQs):
  <br/>
  <br/>
 </strong>
 <ul>
  <li>
   5+ years technical pre-sales or technical consulting experience
  </li>
  <ul>
   <li>
    OR Bachelor's Degree in Computer Science, Information Technology, or related field AND 4+ years technical pre-sales or technical consulting experience
   </li>
   <li>
    OR Master's Degree in Computer Science, Information Technology, or related field AND 3+ year(s) technical pre-sales or technical consulting experience
   </li>
   <li>
    OR equivalent experience
   </li>
  </ul>
  <li>
   Expert on Azure Databases (SQL DB, Cosmos DB, PostgreSQL) from migration &amp; modernize and creating new AI apps.
  </li>
  <li>
   Expert on Azure Analytics (Fabric, Azure Databricks, Purview) and competitors (BigQuery, Redshift, Snowflake) in data warehouse, data lake, big data, analytics, real-time intelligent, and reporting using integrated Data Security &amp; Governance.
  </li>
  <li>
   Proven ability to lead technical engagements (e.g., hackathons, PoCs, MVPs) that drive production-scale outcomes.
   <br/>
  </li>
 </ul>
 <strong>
  Preferred Qualifications (PQs)
  <br/>
  <br/>
 </strong>
 <ul>
  <li>
   6+ years technical pre-sales, technical consulting, or technology delivery, or related experience
  </li>
  <ul>
   <li>
    OR equivalent experience
   </li>
  </ul>
  <li>
   4+ years experience with cloud and hybrid, or on premises infrastructure, architecture designs, migrations, industry standards, and/or technology management
  </li>
  <li>
   Proficient on data warehouse &amp; big data migration including on-prem appliance (Teradata, Netezza, Oracle), Hadoop (Cloudera, Hortonworks) and Azure Synapse Gen2.
   <br/>
  </li>
 </ul>
 Microsoft is an equal opportunity employer. Consistent with applicable law, all qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.
</div>


Cloud & AI Solution Engineer - Data Platform

Marc Ellis

Role: AVP, Cloud Data Engineer <p>Location: Dubai</p><p>Job Type: 6 months contract ( Extendable)</p> Qualification <ul><li>Bachelor's degree in computer science, Computer Engineering, IT, or related fields</li><li>12+ years of overall experience with at least 5 years of leading teams delivering data lakes, data warehouses, data lakehouse</li></ul> Minimum Experience <ul><li>Data Engineering skills:<ul><li>Experience in deploying Lakehouse on Azure/AWS</li><li>Proficiency in at least one of the programming languages viz. Python, Scala</li><li>Proficiency in Data Integration tool - Informatica DEI or Big Data Management (BDM)</li><li>Minimum 2 years' experience on developing and deploying workloads on Apache Spark</li><li>SQL skills are must</li><li>Must have Cloud DevOps skills to be able to build and maintain the infrastructure as a code</li><li>At least 5 years of experience in Cloud, preferably Azure</li></ul></li><li>Communication skills to be able to explain the technical scenarios to senior IT and business stakeholders.</li></ul><p>Experience in agile way of working.</p><img src="https://www.jobg8.com/Tracking.aspx?2BLAx2Vk0CaZP5gjEFxIqqR3%2bFfDMQmTg" width="0" height="0" />

AVP Data Lake Engineering

Caterpillar

<div><p><b>Career Area:</b></p>
Technology, Digital and Data<p><b>
Job Description:</b></p><p><b>
Your Work Shapes the World at Caterpillar Inc.
</b></p><p>When you join Caterpillar, you're joining a global team who cares not just about the work we do &ndash; but also about each other. We are the makers, problem solvers, and future world builders who are creating stronger, more sustainable communities. We don't just talk about progress and innovation here &ndash; we make it happen, with our customers, where we work and live. Together, we are building a better world, so we can all enjoy living in it.</p><p>
We are seeking a <b>highly skilled and visionary Software Engineering Manager</b> to lead a team of engineers in building Caterpillar&rsquo;s next-generation <b>Digital Manufacturing Data Platform</b>. This platform is central to our Future of Manufacturing initiative&mdash;designed to unify and operationalize data across design, engineering, production, and supply chain operations.</p><p>
The ideal candidate will possess deep expertise in <b>Big Data, Data Warehousing, real-time data movement</b>, and <b>Snowflake-based architectures</b>. You will architect and deliver scalable, secure, and intelligent data platforms that enable advanced analytics, AI, and digital twin capabilities across global manufacturing ecosystems.</p><div></div><p><b>
Key Responsibilities</b></p><p><b>
Team Leadership &amp; Management</b></p><ul><li>
Lead, mentor, and manage a team of data engineers and platform developers.</li><li>
Foster a culture of technical excellence, collaboration, and continuous learning.</li><li>
Drive Agile practices and ensure timely delivery of high-quality solutions.</li></ul><p><b>
Technical Strategy &amp; Architecture</b></p><ul><li>
Architect and oversee the development of scalable, secure, and resilient data platforms.</li><li>
Design and implement near real-time data movement and streaming architectures using tools like Kafka, Spark, and cloud-native services.</li><li>
Establish best practices in data modeling, ETL/ELT, data governance, and metadata management.</li></ul><p><b>
Data Engineering &amp; Snowflake Expertise</b></p><ul><li>
Lead the development of robust data pipelines for ingestion, transformation, and delivery using Snowflake, dbt, and cloud-native tools.</li><li>
Optimize data storage, retrieval, and processing for performance, reliability, and cost-efficiency.</li><li>
Implement data quality frameworks, lineage tracking, and schema evolution strategies.</li></ul><p><b>
Big Data &amp; Data Warehousing</b></p><ul><li>
Build and maintain large-scale data lakes and data warehouses for structured and unstructured data.</li><li>
Design scalable data architectures to support manufacturing analytics, predictive maintenance, and supply chain optimization.</li></ul><p><b>
Cloud &amp; Platform Engineering</b></p><ul><li>
Leverage Azure and AWS services for data ingestion, transformation, and analytics.</li><li>
Deploy software using CI/CD tools (Azure DevOps preferred, Jenkins, AWS CloudFormation).</li><li>
Ensure platform scalability, security, and operational readiness across global deployments.</li></ul><p><b>
AI &amp; Advanced Analytics Enablement</b></p><ul><li>
Collaborate with Data Science and AI teams to operationalize ML models and analytics workflows.</li><li>
Promote integration of AI capabilities into data engineering pipelines (e.g., GenAI, MCP, ATA).</li><li>
Support real-time analytics and edge AI use cases in manufacturing environments.</li></ul><p><b>
Stakeholder Engagement</b></p><ul><li>
Partner with product managers, manufacturing SMEs, and business leaders to understand requirements and deliver impactful data solutions.</li><li>
Communicate technical concepts to non-technical audiences and influence strategic decisions.</li></ul><div></div><p><b>
Must-Have Skills</b></p><ul><li>
Proven experience in Big Data processing and Data Warehousing.</li><li>
Expertise in building end-to-end near real-time data pipelines for OLTP &amp; OLAP.</li><li>
Strong architecture exposure for building robust, scalable Data Platforms.</li><li>
Deep expertise in Snowflake, SQL, NoSQL, and distributed data systems.</li><li>
Experience with data transformation tools (dbt, Apache Spark, Azure Data Factory).</li><li>
Strong analytical skills and solid knowledge of computer science fundamentals.</li><li>
Deep exposure to Azure and AWS cloud platforms.</li><li>
Good understanding of AI concepts and latest developments (Gen AI, MCP, ATA, etc.).</li></ul><div></div><p><b>
Nice-to-Have Skills</b></p><ul><li>
Knowledge of the NVIDIA ecosystem and its applications in data and AI.</li><li>
Experience building production-ready AI solutions and integrating with MLOps workflows.</li><li>
Familiarity with modern data visualization and BI tools (e.g., Power BI, Tableau, Looker).</li></ul><div></div><p><b>
Qualifications</b></p><ul><li>
Bachelor&rsquo;s or Master&rsquo;s degree in Computer Science, Engineering, or related field.</li><li>
15+ years of experience in data engineering, with at least 5+ years in a leadership role.</li><li>
Demonstrated success in managing engineering teams and delivering complex data solutions.</li><li>
Excellent communication, leadership, and stakeholder management skills.</li></ul>
Relocation is available for this position.<p><b>
Posting Dates:</b></p>
October 22, 2025 - October 30, 2025<p>
Caterpillar is an Equal Opportunity Employer. Qualified applicants of any age are encouraged to apply</p><p>
Not ready to apply? Join our Talent Community.</p></div>

Senior Manager Software Engineering-Data Engineering

eMumba

<p></p><div><div><div><div><div><div><div><div><div>We are looking for Platform Software Engineer having 4 to 8 years of experience who can perform embedded system software design and implementation for leading high-performance data networking products as a member of a highly capable and dynamic engineering team.</div></div></div>
</div><h2 class="jobSectionHeader"><b>Key Responsibilities
</b></h2><div><div><div><ul><li>Architect/Design and Develop embedded platform software for high end networking solutions.</li>
<li>Work closely with HW Development team to enable rapid platform bring up.</li>
<li>Participate in HW Design reviews, articulate system software requirements.</li>
<li>Demonstrate cross functional collaboration with Product Marketing, SQA and other development teams to ensure embedded platform software meets all performance, scalability, interop and security requirements.</li>
</ul></div></div></div><h2 class="jobSectionHeader"><b>Skills, Requirements and Expertise
</b></h2><div><div><div><ul><li>Minimum of B.S./B.E./B.Tech in CS/EE/ECE.</li>
<li>Minimum of 4 to 8 years of experience developing complex networking solutions.</li>
<li>Excellent understanding of Linux Internals, Device Driver development, BIOS/BMC Operations.</li>
<li>Excellent knowledge of switching ASICs (Broadcom XGS/DNX).</li>
<li>Expert programming skills in C/C++; GoLang experience is a plus.</li>
<li>Ability to thrive in fast paced agile development environment.</li>
<li>Familiarity with Secure coding practices is a big plus.</li>
<li>Solid understanding of network security (Encryption, Authentication, DDOS mitigation etc) is a big plus.</li>
<li>Strong debugging and troubleshooting skills.</li>
<li>Must have strong written, oral communication and interpersonal skills.</li>
<li>Understanding of virtualization technologies (Virtual Machines, Containers) is a plus.</li>
</ul></div></div></div><h2 class="jobSectionHeader"><b>About Emumba
</b></h2><div><div>We specialize in delivering innovative solutions and exceptional services to meet the diverse needs of our clients. With a strong commitment to quality and customer satisfaction, we strive to exceed expectations and drive success in every project we undertake.</div></div><p></p><div><br>
</div></div></div></div></div><p></p><div><div><div><div><div><div><div>
</div></div><div><div><p>Department</p>
<p><b>Backend</b></p>
<p>Employment Type</p>
<p><b>Full Time</b></p>
<p>Location</p>
<p><b>Islamabad, Pak</b></p>
<p>Workplace type</p>
<p><b>Hybrid</b></p></div></div></div></div></div></div></div></div><p></p>

Platform Software Engineer

Pennant Technologies

<div><div><b>Job Title:</b> Senior Big Data Engineer</div><div><b>
Department:</b> Data &amp; Analytics</div><div><b>
Employment Type:</b> Full&#8209;time</div><div><b>
Role Overview</b></div><div>
We are seeking an experienced Big Data Engineer to join our growing Data &amp; Analytics team. In this role, you&rsquo;ll design, build, and optimize our modern data platform&mdash;enabling scalable pipelines, reliable data lakes, and robust ML operations. You&rsquo;ll work closely with data scientists, analysts, and stakeholders to turn raw data into actionable insights.</div><div><b>
Key Responsibilities</b></div><ul><li><b>
Data Platform Development &amp; Maintenance</b></li><li>
Design, deploy, and maintain ETL/ELT pipelines using Apache Airflow and dbt/meltano.</li><li>
Architect and manage data lakes on HDFS and object storage (e.g., S3/HDFS).</li><li>
Implement table formats and governance with Apache Iceberg and Unity Catalog/Apache ranger/Atlas.</li><li><b>
Big Data Processing</b></li><li>
Develop distributed data processing jobs in Apache Spark (Scala/Java/Python).</li><li>
Optimize Spark applications for performance, cost, and scalability.</li><li>
Good working knowledge in Databases (Postgres, Mongo, SQL, Sql server etc)</li><li><b>
Collaboration with Data science teams</b></li><li>
Integrate MLflow for experiment tracking, model registry, and seamless deployment.</li><li>
Collaborate with data science teams to operationalize models in production.</li><li><b>
Containerization &amp; Orchestration</b></li><li>
Containerize data services and workflows using Docker/kubenetes.</li><li>
Orchestrate deployments on Kubernetes clusters (EKS/GKE/AKS).</li><li><b>
Center of Excellence &amp; Best Practices</b></li><li>
Contribute to and govern the dbt Center of Excellence: define standards, code reviews, and documentation.</li><li>
Advocate and enforce data engineering best practices: CI/CD, testing, monitoring, and observability.</li><li><b>
Cross&#8209;Functional Collaboration</b></li><li>
Partner with Analytics, Product, and Engineering teams to understand requirements and deliver solutions.</li><li>
Mentor junior engineers and share knowledge through code reviews and technical talks.</li></ul><div><b>
Required Skills &amp; Experience</b></div><ul><li>
5+ years of professional experience in Big Data engineering or similar roles.</li><li><b>
Airflow:</b> Hands&#8209;on experience authoring DAGs, configuring operators/sensors, and managing scheduler/executor backends.</li><li><b>
Apache Spark:</b> Proficient in building and tuning large&#8209;scale ETL jobs (Scala/Python).</li><li><b>
Apache Iceberg &amp; Unity Catalog:</b> Familiarity with table formats, partitioning strategies, and catalog management.</li><li><b>
Data Lake &amp; HDFS:</b> Solid understanding of storage architectures, file formats (Parquet/ORC), and data partitioning.</li><li><b>
MLflow:</b> Working knowledge of experiment tracking, model registry, and deployment workflows.</li><li><b>
Kubernetes:</b> Skilled in deploying, scaling, and debugging containerized applications and stateful workloads.</li><li><b>
dbt COE:</b> Experience with dbt for data modeling, CI/CD integration, and establishing a CoE for standards.</li><li>
Strong programming skills in Python and/or Scala.</li><li>
Experience with version control (Git), CI/CD pipelines (Jenkins/GitHub Actions), and monitoring tools (Prometheus, Grafana).</li></ul><div><b>
Nice&#8209;to&#8209;Have</b></div><ul><li>
Familiarity with cloud platforms (AWS, GCP, or Azure) and managed services (EMR, Dataproc, EKS, GKE).</li><li>
Knowledge of additional data catalogs (e.g., AWS Glue, Hive Metastore).</li><li>
Background in event&#8209;driven architectures (Kafka, Pulsar).</li><li>
Prior involvement in data governance or compliance initiatives.</li></ul><div><b>
Education &amp; Certifications</b></div><ul><li>
Bachelor&rsquo;s or Master&rsquo;s degree in Computer Science, Engineering, or related field.</li><li>
Relevant certifications (e.g., Any cloud certification AWS/GCP/Azure/Databricks Certified Big Data &ndash; Specialty) are a plus.</li></ul><div><b>
Soft Skills</b></div><ul><li>
Excellent problem&#8209;solving and analytical abilities.</li><li>
Strong written and verbal communication; able to explain technical concepts to non&#8209;technical stakeholders.</li><li>
Proactive mindset with a passion for learning and driving innovation.</li><li>
Collaborative team player who thrives in agile, fast&#8209;paced environments.</li></ul><div><b>
Join us</b> and help shape the future of our data platform&mdash;enabling smarter decisions, faster insights, and transformative business outcomes!</div></div><div></div>

Big Data Engineer

Oracle

<div><div><div>We're seeking a Senior Data Integration Engineer to design, develop, and manage enterprise-grade data integration solutions. The ideal candidate will have experience with ETL/ELT processes, API-driven integrations, and enterprise data platforms, with strong technical skills and collaboration abilities.
</div><div>Please apply, If you have the passion and standard methodologies work in an environment where challenges are a norm, where individual brilliance is valued and goes hand in hand with team performance, Where being proactive is how we do things !!!</div></div></div>
<br><div></div><div><div>Successful applicants should have/demonstrate:
</div><p>Key responsibilities
</p><ul><li>Architect, design, and optimize scalable big data solutions for batch and real-time processing.
</li><li>Develop and maintain ETL/ELT pipelines to ingest, transform, and synchronize data from diverse sources.
</li><li>Integrate data from cloud applications, on-prem systems, APIs, and streaming workspaces into centralized data repositories.
</li><li>Implement and manage <b>data lakes </b>and <b>data warehouses </b>solutions on cloud infrastructure.
</li><li>Ensure <b>data consistency, quality, and compliance </b>with governance and security standards.
</li><li>Collaborate with data architects, data engineers, and business stakeholders to align integration solutions with organizational needs.
</li></ul><p>Core qualifications
</p><ul><li>Proficiency in Python, Java, or Scala for big data processing.
</li><li><b>Big Data Frameworks: </b>Strong expertise in Apache Spark, Hadoop, Hive, Flink, or Kafka.
</li><li>Hands-on experience with data modeling, data lakes (Delta Lake, Iceberg, Hudi), and data warehouses (Snowflake, Redshift, BigQuery).
</li><li><b>ETL/ELT Development: Expertise with tools like Informatica, Talend, SSIS, Apache NiFi, dbt, or custom Python-based frameworks.
</b></li><li><b>APIs &amp; Integration: Strong hands-on experience with REST, SOAP, GraphQL APIs, and integration platforms (MuleSoft, Dell Boomi, SnapLogic).
</b></li><li><b>Data Pipelines: Proficiency in batch and real-time integration (Kafka, AWS Kinesis/ Azure Event Hub/ GCP Pub/Sub).
</b></li><li><b>Databases: Deep knowledge of SQL (Oracle, PostgreSQL, SQL Server) and NoSQL (MongoDB, Cassandra, DynamoDB) systems.
</b></li></ul><p>Preferred experience
</p><ul><li>Expertise with at least one major cloud platform (AWS, Azure, GCP).
</li><li>Experience with data services such as AWS EMR/Glue, GCP Dataflow/Dataproc, or Azure Data Factory.
</li><li>Familiarity with containerization (Docker) and orchestration (Kubernetes).
</li><li>Knowledge of CI/CD pipelines for data engineering.
</li><li>Experience with OCI and Oracle Database (including JSON/REST, sharding) and/or Oracle microservices tooling.</li></ul></div><div></div>

Senior Big Data Engineer

Search 10000+ Big data platform architect jobs near you this October. New roles added regularly across leading industries. Apply today.