big data engineer

Caregenix Solutions

<p>We're Hiring: <b>ETL Engineer</b> (Remote &ndash; Full Time)</p><p>Company: CareGenix Solutions</p><p>Location: Remote</p><p>Experience: <b>3+ Years</b></p><p>Type: Full-Time</p><p>Caregenix Solutions, a fast-growing healthcare tech company, is seeking a skilled ETL Engineer to design and maintain robust data pipelines that power real-time insights and analytics.</p><p>Key Responsibilities:</p><p>Build and manage scalable ETL/ELT pipelines using AWS services (Glue, Lambda, S3, EMR, Redshift, etc.).</p><p>Design and optimize data models and databases (Redshift, Athena, RDS).Ingest data from diverse sources and ensure quality through validation checks.</p><p>Collaborate with teams to deliver reliable, insight-driven data solutions.Monitor pipeline performance and troubleshoot as needed.</p><p>Follow best practices for data security and governance.Requirements6+ years in Data Engineering or similar roles.</p><p>Strong in SQL and Python or Scala.Hands-on experience with AWS data tools: Glue, Redshift, EMR, Athena, Kinesis, etc.</p><p>Experience with workflow/orchestration tools (Airflow, Step Functions).Solid knowledge of data warehousing, performance tuning, and big data.</p><p>Familiarity with CI/CD, Git, and IaC tools (Terraform, CloudFormation).Bachelor&rsquo;s in Computer Science, Engineering, or related field.</p><p>To Apply Send your resume to careers@caregenix.solutions</p><p>Subject Line: ETL Engineer</p><p>Be part of a team transforming healthcare through data. We&rsquo;d love to hear from you</p><p>Job Type: Full-time</p><p>Pay: Up to Rs400,000.00 per month</p><p>Application Question(s):</p><ul><li>what is your expected pay?</li></ul><p>Experience:</p><ul><li>Data/ ETL : 2 years (Required)</li></ul><p>Work Location: Remote</p>

Data Engineer

Tata Consultancy Services (TCS)

<div></div><div><div><div><div><div>Must Have Technical/Functional Skills
</div><div>8+ years of skilled Databricks Developer to design, develop, and optimize data pipelines and analytics solutions using Databricks on cloud platforms (Azure/AWS/GCP). The ideal candidate will have strong experience in Spark, Python/Scala, and SQL, along with a solid understanding of big data processing and data engineering best practices
</div><div>Hands-on experience with Databricks and Apache Spark.
</div><div>Strong programming skills in Python and/or Scala.
</div><div>Proficiency in SQL for data manipulation and analysis.
</div><div>Experience with Delta Lake and Lakehouse architecture.
</div><div>Familiarity with cloud platforms (Azure Data Lake, AWS S3, or GCP).
</div><div>Knowledge of CI/CD pipelines and DevOps practices.
</div><div>Understanding of data modeling, data governance, and security.
</div><div>Experience with performance tuning for big data jobs.</div><div><br>
</div>
<div></div><br>
<div>Salary Range $110000-$135,000 year
</div>
<div></div><div>TCS Employee Benefits Summary:
</div><div>Discretionary Annual Incentive.
</div><div>Comprehensive Medical Coverage: Medical &amp; Health, Dental &amp; Vision, Disability Planning &amp; Insurance, Pet Insurance Plans.
</div><div>Family Support: Maternal &amp; Parental Leaves.
</div><div>Insurance Options: Auto &amp; Home Insurance, Identity Theft Protection.
</div><div>Convenience &amp; Professional Growth: Commuter Benefits &amp; Certification &amp; amp; Training Reimbursement.
</div><div>Time Off: Vacation, Time Off, Sick Leave &amp; Holidays.
</div><div>Legal &amp; Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.
</div>
<div></div><div>#LI-SP1
</div></div></div></div><div><div><div><b>Location</b></div>
<div>Pleasanton, CA</div>
</div><div><div><b>Job Function</b></div>
<div>TECHNOLOGY</div>
</div><div><div><b>Role</b></div>
<div>Engineer</div>
</div><div><div><b>Job Id</b></div>
<div>398064</div>
</div><div><div><b>Desired Skills</b></div>
<div>Python</div>
</div><div><div><b>Salary Range</b></div>
<div>$110,000-$135,000 a year</div>
</div></div><p>Desired Candidate Profile
</p><div><div><b>Qualifications</b> : BACHELOR OF COMPUTER SCIENCE</div></div></div><div></div>

Databricks engineer

Binary Brix 

<p><b>Data Engineer (Remote)</b></p><p><b>Experience:</b> 4&ndash;5 years | <b>Job Type:</b> Full-Time | <b>Location:</b> Remote</p><p>We are looking for an experienced <b>Data Engineer</b> to design, build, and maintain scalable data pipelines and architectures for our next-gen data-driven SaaS platform. This role is fully remote and ideal for someone passionate about <b>big data, cloud platforms, and analytics</b>.</p><p><b>Key Responsibilities</b></p><ul><li>Build, maintain, and optimize <b>ETL/ELT pipelines</b> for large-scale data processing.</li><li>Work with <b>structured &amp; unstructured data</b> from multiple sources.</li><li>Develop and manage <b>data warehouses</b> and <b>data lakes</b>.</li><li>Ensure <b>data quality, governance, and security</b>.</li><li>Collaborate with <b>data scientists, analysts, and software teams</b> to deliver insights.</li><li>Monitor performance and troubleshoot <b>data pipelines</b>.</li><li>Integrate solutions with <b>cloud platforms</b> (AWS, Azure, GCP).</li></ul><p><b>Required Skills &amp; Experience</b></p><ul><li>4&ndash;5 years in <b>data engineering</b> or related roles.</li><li>Strong <b>Python, SQL, and scripting</b> skills.</li><li>Experience with <b>ETL/ELT tools</b> (Spark, Airflow, Databricks).</li><li>Hands-on with <b>cloud data platforms</b> (AWS, Azure, GCP).</li><li>Familiarity with <b>data warehouses</b> (Snowflake, Redshift, BigQuery).</li><li>Knowledge of <b>streaming platforms</b> like Kafka is a plus.</li></ul><p><b>Nice to Have</b></p><ul><li>DBT or other <b>data transformation frameworks</b>.</li><li>CI/CD for <b>data workflows</b>.</li><li>Exposure to <b>ML pipelines</b>.</li></ul><p>Job Type: Full-time</p><p>Pay: Rs200,000.00 - Rs300,000.00 per month</p><p>Work Location: Remote</p>

VTG Defense

<div>Overview:
<div>
We are looking for a Software Engineer / Data Engineer to join our team.</div>
Responsibilities:
<ul><li><div>
Design, build, and operate large-scale Big Data systems, including persistence, partitioning, indexing, and search capabilities.</div></li><li><div>
Develop and maintain Java-based applications and APIs.</div></li><li><div>
Architect and implement cloud-native solutions using AWS or comparable cloud platforms.</div></li><li><div>
Design and optimize high-performance graph databases containing tens of billions of edges.</div></li><li><div>
Develop graph traversal capabilities using Apache TinkerPop, Gremlin, JanusGraph, or similar technologies.</div></li><li><div>
Build and maintain NoSQL and relational database solutions supporting complex Big Data applications.</div></li><li><div>
Design partition and sort key strategies to ensure consistent system performance.</div></li><li><div>
Develop aggregation operations to de-duplicate records across continuous data feeds.</div></li><li><div>
Build and operate serverless data processing pipelines using AWS Lambda, Step Functions, and PySpark.</div></li><li><div>
Design and operate large-scale geospatial indexing solutions using GeoMESA.</div></li><li><div>
Develop and maintain Kubernetes-based containerized environments.</div></li><li><div>
Implement DevSecOps and agile development practices in production environments.</div></li><li><div>
Maintain configuration management using Git-based repositories.</div></li><li><div>
Facilitate technical discussions across cross-functional teams to develop mission-aligned implementation strategies.</div></li><li><div>
Ensure compliance with federal security, regulatory, and accreditation requirements.</div></li><li><div>
Implement data security and governance controls including LDAP integration, encryption, and auditing.</div></li></ul>
Qualifications:
<ul><li>Active TS/SCI with Polygraph required.
</li><li>Bachelor&rsquo;s degree in Geospatial Intelligence, Geography, Remote Sensing, Intelligence Studies, Engineering, or related field, or equivalent experience</li><li>
Demonstrated experience with Java development.</li><li>
Experience designing and operating Big Data systems.</li><li>
Experience developing and maintaining APIs.</li><li>
Experience designing cloud-native architectures using AWS or similar cloud platforms.</li><li>
Experience building and optimizing large-scale graph databases using technologies such as Cassandra, DynamoDB, Neo4j, or JanusGraph.</li><li>
Experience developing graph traversal capabilities using Apache TinkerPop and Gremlin.</li><li>
Experience designing and operating NoSQL solutions.</li><li>
Experience in data modeling, partition sharding, stream processing, and metrics gathering.</li><li>
Experience developing high-performance data processing pipelines.</li><li>
Experience with Kubernetes, Docker, and container orchestration.</li><li>
Experience with Apache NiFi.</li><li>
Experience implementing DevSecOps and agile methodologies.</li><li>
Experience with data security controls including encryption and centralized access management (LDAP).</li><li>
Experience working with structured, semi-structured, and unstructured data formats (CSV, JSON, AVRO, Parquet, Protocol Buffers, etc.).</li><li>
Experience with relational and NoSQL databases including PostgreSQL, MariaDB, MongoDB, Cassandra, ELK, MinIO, and AWS S3.</li><li>
Experience working in Linux environments such as CentOS or Rocky Linux.</li><li>
Experience with Python and related libraries.</li><li>
Experience supporting large collaboration and development environments</li></ul></div>

Software Engineer

HRCONSOL

<p>We&rsquo;re Hiring: Azure Data Engineer</p><p><b>Client:</b> Leading USA-Based Software House<br/><b>Location:</b> Karachi (On-Site)<br/><b>Employment Type:</b> Full-Time<br/><b>Work Arrangement:</b> On-Site<br/><b>Work Hours:</b> Ability to overlap up to <b>6 hours with US CST</b></p><p>Role Overview</p><p>HRCONSOL is hiring an experienced <b>Azure Data Engineer</b> for a leading US-based software organization&rsquo;s Karachi Head Office. This role is ideal for a self-driven professional with strong expertise in <b>Azure and Databricks</b>, capable of owning end-to-end data engineering solutions and collaborating with global teams.</p><p>Key Responsibilities</p><ul><li>Design, develop, and implement advanced analytics solutions using <b>Azure &amp; Databricks</b></li><li>Perform data migration and transformation on Databricks</li><li>Build and maintain scalable data pipelines and API integrations</li><li>Collaborate with data engineers, data scientists, and business stakeholders</li><li>Optimize and tune performance of Azure-based and Databricks solutions</li><li>Monitor and manage Databricks environments for reliability and efficiency</li><li>Lead architecture and development of complex data integration projects</li><li>Conduct code reviews and enforce best practices</li><li>Independently own and manage tasks with timely, high-quality delivery</li><li>Work effectively with cross-functional and remote US-based teams</li></ul><p>Qualifications</p><ul><li>Bachelor&rsquo;s or Master&rsquo;s degree in <b>Computer Science, IT, or related field</b></li><li><b>6+ years</b> of hands-on experience in <b>Azure &amp; Databricks architecture, migration, and development</b></li><li>Strong experience in data pipelines, architectures, and data optimization</li><li>Proficiency in <b>Python, Scala, or Java</b></li><li>Experience with <b>Spark, Hadoop</b>, or other big data technologies</li><li>Strong SQL and relational database knowledge</li><li>Experience with cloud infrastructure monitoring and management tools</li><li>Excellent analytical, troubleshooting, and communication skills</li><li>Ability to work independently and overlap with <b>US CST hours (up to 6 hours)</b></li></ul><p>Preferred</p><ul><li>Azure and/or Databricks certifications</li></ul><p>Technical Skill Set</p><ul><li>Azure Data Factory (ADF)</li><li>Databricks</li><li>SQL &amp; NoSQL Databases</li><li>Data Modeling &amp; Data Warehousing</li><li>ETL Tools &amp; Processes</li><li>Azure Synapse Analytics</li><li>Python and/or Scala</li><li>Power BI or other data visualization tools</li></ul><p>How to Apply</p><p>Interested candidates can apply by sending their <b>updated CV</b> to:<br/><b>jobs@hrconsol.com</b><br/><i>Mention &ldquo;Azure Data Engineer &ndash; Karachi&rdquo; in the subject line.</i></p><p>Job Type: Full-time</p><p>Pay: Rs300,000.00 - Rs600,000.00 per month</p><p>Work Location: In person</p>

Azure Data Engineer

<div>
 <strong>
  Introduction
  <br/>
  <br/>
 </strong>
 In this role, you'll work in one of our IBM Consulting Client Innovation Centers (Delivery Centers), where we deliver deep technical and industry expertise to a wide range of public and private sector clients around the world. Our delivery centers offer our clients locally based skills and technical expertise to drive innovation and adoption of new technology.
 <br/>
 <br/>
 <strong>
  Your Role And Responsibilities
  <br/>
  <br/>
 </strong>
 <ul>
  <li>
   As an Data Engineer at IBM you will harness the power of data to unveil captivating stories and intricate patterns. You'll contribute to data gathering, storage, and both batch and real-time processing.
  </li>
  <li>
   Collaborating closely with diverse teams, you'll play an important role in deciding the most suitable data management systems and identifying the crucial data required for insightful analysis. As a Data Engineer, you'll tackle obstacles related to database integration and untangle complex, unstructured data sets.
  </li>
  <li>
   In this role, your responsibilities may include:
  </li>
  <li>
   Implementing and validating predictive models as well as creating and maintain statistical models with a focus on big data, incorporating a variety of statistical and machine learning techniques
  </li>
  <li>
   Designing and implementing various enterprise search applications such as Elasticsearch and Splunk for client requirements
  </li>
  <li>
   Work in an Agile, collaborative environment, partnering with other scientists, engineers, consultants and database administrators of all backgrounds and disciplines to bring analytical rigor and statistical methods to the challenges of predicting behaviour&rsquo;s.
   <br/>
   <br/>
  </li>
 </ul>
 <strong>
  Required Technical And Professional Expertise
  <br/>
  <br/>
 </strong>
 <ul>
  <li>
   Proof of Concept (POC) Development: Develop POCs to validate and showcase the feasibility and effectiveness of the proposed AI solutions.
  </li>
  <li>
   Help in showcasing the ability of Gen AI code assistant to refactor/rewrite and document code from one language to another
  </li>
  <li>
   Document solution architectures, design decisions, implementation details, and lessons learned.
  </li>
  <li>
   Stay up to date with the latest trends and advancements in AI, foundation models, and large language models.
  </li>
  <li>
   Evaluate emerging technologies, tools, and frameworks to assess their potential impact on solution design and implementation
   <br/>
   <br/>
  </li>
 </ul>
 <strong>
  Preferred Technical And Professional Experience
  <br/>
  <br/>
 </strong>
 <ul>
  <li>
   Experience and working knowledge in COBOL &amp; JAVA would be preferred
  </li>
  <li>
   Having experience in Code generation, code matching &amp; code translation leveraging LLM capabilities would be a Big plus
  </li>
  <li>
   Demonstrate a growth mindset to understand clients' business processes and challenges
  </li>
 </ul>
</div>


Data Scientist-Advanced Analytics

Confiz

<div><p>We are looking for a Principal Data Engineer, responsible for leading data engineering teams. He/she will provide guidance, mentorship, and technical expertise to team members, fostering a collaborative and innovative work environment. Collaboration is key for a Principal Data Engineer as he/she will work closely with stakeholders to define project requirements. Therefore, a commitment to collaborative problem solving, sophisticated design, and the creation of quality products is essential.</p>
<p><b>Responsibilities:</b></p>
<ul><li>Collaborate with Big Data Solution Architects to design, prototype, implement, and optimize data ingestion pipelines so that data is shared effectively across various business systems.</li>
<li>Build &amp; maintain ETL/ELT Ingestion pipelines and analytics solutions using cloud technologies.</li>
<li>Ensure the design, code and procedural aspects of solution are production ready, in terms of operational, security and compliance standards.</li>
<li>Implement monitoring and logging solutions to ensure data pipelines run smoothly and issues are promptly identified.</li>
<li>Ensure all data processing complies with security policies and data protection regulations.</li>
<li>Takes the lead in ensuring that data is accurate, consistent, and reliable.</li>
<li>Participate in day-to-day project and production delivery status meetings, and provide technical support for faster resolution of issues.</li>
<li>Work with Stakeholders and Understand their Requirements</li>
</ul>
<p><b>Qualifications</b></p>
<ul><li>Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field (or equivalent work experience).</li>
<li>5+ years of overall experience in data engineering &amp; data warehousing domain focusing on building large scale data pipelines</li>
<li>3+ years' experience in Big Data Distributed systems such as Microsoft Fabric, Databricks, AWS EMR</li>
<li>Strong experience in developing data pipelines using PySpark and ETL tools like ADF, Talend</li>
<li>Proficiency in Python &amp; SQL</li>
<li>Solid understanding of data modelling, ETL processes, and best practices in data warehousing domain</li>
<li>Strong problems solving and critical thinking skills</li>
<li>Experience of writing effective and maintainable unit and integration tests for ingestion pipelines.</li>
<li>Working knowledge of CI/CD pipelines for deployments.</li>
<li>Working knowledge of using code version control repositories like Git.</li>
<li>Knowledge of designing, developing, and managing Kafka-based data pipelines</li>
<li>Knowledge of security best practices including encryption of sensitive data, encryption at rest and encryption in flight.</li>
</ul>
<p>We have an amazing team of 700+ individuals working on highly innovative enterprise projects &amp; products. Our customer base includes Fortune 100 retail and CPG companies, leading store chains, fast-growth fintech, and multiple Silicon Valley startups.</p>
<p>What makes Confiz stand out is our focus on processes and culture. Confiz is <b>ISO 9001:2015</b> (QMS), <b>ISO 27001:2022</b> (ISMS), <b>ISO 20000-1:2018</b> (ITSM), <b>ISO 14001:2015</b> (EMS), <b>ISO 45001:2018 </b>(OHSMS) Certified. We have a vibrant culture of learning via collaboration and making workplace fun.</p>
<p>People who work with us work with cutting-edge technologies while contributing success to the company as well as to themselves.</p>
<p>To know more about Confiz Limited, visit: https://www.linkedin.com/company/confiz-pakistan/</p></div><p></p>

Principal Data Engineer

Search 9485+ Big data engineer jobs near you this February. New roles added regularly across leading industries. Apply today.

9485 big data engineer jobs NEAR_ME

Data Engineer

Databricks engineer

Data Engineer

Software Engineer

Azure Data Engineer

Data Scientist-Advanced Analytics

Principal Data Engineer