big data engineer

Mern Plus Technologies

<p>Big Data Engineering</p><p>remote</p><p>Good Communication needed</p><p>Must have skills - SQL PySpark, Databricks, AWS, Python</p><p>5-10 yrs</p><p>Job Description:</p><p>&quot;1) Experience in AWS Databricks Data pipelines Development - DLT , Streaming &amp; Data ingestion patterns , Unity Catalog migration</p><p>2) Independently develop AWS data extraction and integration code modules for one-time historical, incremental data flows from Databricks delta lake using Pyspark, Python ,SQL, AWS Data bricks, S3.</p><p>3) Develop scalable data ingestion pipelines as needed</p><p>4) Implement advanced transformation logics, schedule, orchestrate, and validate pipelines</p><p>5) Perform data validation, data reconciliation</p><p>6) Deployment of Components</p><p>7) Perform unit testing and document test results.&quot;</p><p>Primary Skills: AWS Databricks,SQL, Python Secondary Skills: Pyspark, AWS Cloudwarch, AWS S3, Streaming</p><p>Job Types: Full-time, Permanent, Contractual / Temporary, Freelance<br/>Contract length: 12 months</p><p>Pay: &#8377;4,000,000.00 - &#8377;6,000,000.00 per year</p><p>Work Location: Remote</p>

Big Data Engineer

Apple

<div>Apple is where individual imaginations gather together, committing to the values that lead to great work. Every new product we build, service we create, or Apple Store experience we deliver is the result of us making each other&rsquo;s ideas stronger. That happens because every one of us shares a belief that we can make something wonderful and share it with the world, changing lives for the better. It&rsquo;s the diversity of our people and their thinking that inspires the innovation that runs through everything we do. When we bring everybody in, we can do the best work of our lives. Here, you&rsquo;ll do more than join something - you&rsquo;ll add something.<br>
<br>
Apple Pay brought mobile payment to millions of customers, and it&rsquo;s just the beginning. We are looking for engineers who enjoy both hands-on technical work and designing thoughtful, scalable services for Wallet and Apple Pay. Our team&rsquo;s vision is to be the engine of intelligent transformation, leveraging a unified, reliable data platform to build and deploy innovative and solutions that drive significant business impact and enable data-driven decision-making throughout the organization.<br>
<br>
We are seeking pragmatic AI Big Data Engineers to join our dynamic team to build and optimize data and analytics solutions as well as perform ML enablement and participate in generative AI initiatives to craft the future of Wallet and Apple Pay. You will collaborate with cross-functional teams across different time zones and deliver impactful and scalable data architectures.<br>
<br>
<b>Description</b><br>
<ul><li>Instrument APIs, user journey and interaction flows to systematically collect behavioral, transactional, and operational data, enabling robust analytics and insightful reporting</li></ul><br>
<ul><li>Build robust data architectures for Wallet, Payments &amp; Commerce products.
</li></ul><br>
<ul><li>Build robust data architectures supporting large-scale Wallet Payments &amp; Commerce (WPC) applications.
</li></ul><br>
<ul><li>Optimize ETL workflows to enhance data processing efficiency and reliability.
</li></ul><br>
<ul><li>Develop tools and frameworks to optimize data processing performance.
</li></ul><br>
<ul><li>Ensure data quality and integrity across all data systems and platforms.
</li></ul><br>
<ul><li>Collaborate closely with diverse set of partners to gather requirements, prioritize use cases, and ensure high-quality data products delivery.</li></ul><br>
<ul><li>Integrate data pipelines into the broader ML Operations (MLOps) process. This includes automating the data flow for feature engineering, model retraining, performance monitoring models in production, drift detection and ensuring scalability.</li></ul><br>
<ul><li>Construct and maintain data pipelines for Gen AI/RAG solutions, including processes for data extraction, chunking, embedding, and grounding to prepare data for models and perform continuous quality and performance measurement.</li></ul><br>
<b>
Preferred Qualifications</b><br>
Ability to create technical Specs, instrumentation specs and posses the ability to understand APIs, MSDs etc..
<br>
<br>
Expertise in building and refining large-scale data pipelines, as well as developing tools and frameworks for data platforms.<br>
<br>
Hands-on experience with big data technologies such as distributed querying(Trino), real-time analytics(OLAP), near-real-time data processing (NRT), and decentralized data architecture (Apache Mesh).
<br>
<br>
Familiarity with data governance, security protocols, and compliance in financial data systems.<br>
<br>
Experience enabling ML pipelines including automating the data flow for feature engineering, model retraining, performance monitoring models in production, drift detection and ensuring scalability.<br>
<br>
Familiarity with GenAI concepts like Retrieval-Augmented Generation (RAG), Large Language Models (LLMs), prompt engineering, vector embeddings, and LLM fine-tuning<br>
<br>
Works independently with minimal oversight, actively builds relationships, and contributes to a positive team environment.<br>
<br>
Demonstrates sound judgment, applies technical principles to complex projects, evaluates solutions, and proposes new ideas and process improvements.
<br>
<br>
Seeks new opportunities for growth, demonstrates a thorough understanding of technical concepts, exercises independence in problem-solving, and delivers impactful results at the team level.<br>
<br>
Familiarity with Fintech, Wallet domain, digital commerce etc..<br>
<br>
<b>Minimum Qualifications</b><br>
Bachelor&rsquo;s or Master&rsquo;s degree in Computer Science or a related technical field or equivalent experience<br>
<br>
4+ years of experience in designing, developing and deploying data engineering for analytics or ML &amp; AI pipelines.<br>
<br>
Strong proficiency in SQL, Scala, Python, or Java, with hands-on experience in data pipeline tools (e.g., Apache Spark, Kafka, Airflow), CI/CD practices, and version control.<br>
<br>
Familiarity with cloud platforms (AWS, Azure, GCP) and data management and analytics tools like Snowflake, Databricks and Tableau.<br>
<br>
Strong understanding of data warehousing, data modeling (dimensional/star schemas), and metric standardization.
<br>
<br>
Strong problem-solving skills and the ability to work in an agile environment.&quot;,&quot;internalDetails&quot;:null,&quot;eeoContent&quot;:null</div>

AI Big Data Engineer

INFLUX SERVICE

<p><b>Big Data Python Engineer</b></p><p><b>Job Summary:</b><br/>We are looking for a skilled <b>Big Data Python Engineer</b> who can develop, optimize, and manage large-scale data processing solutions. The ideal candidate should have strong experience in Python, Big Data technologies, and data pipelines.</p><p><b>Key Responsibilities:</b></p><ul><li>Design, develop, and maintain scalable <b>data pipelines</b> using Python and Big Data frameworks.</li><li>Work with <b>Hadoop, Hive, Spark</b>, and distributed data systems to process large datasets.</li><li>Optimize ETL workflows for performance, reliability, and scalability.</li><li>Integrate data from multiple sources and ensure high data quality.</li><li>Collaborate with data scientists, analysts, and engineering teams to meet business requirements.</li><li>Troubleshoot data issues and perform data validation.</li><li>Write clean, efficient, and well-documented Python code.</li><li>Implement automation for repetitive data processing tasks.</li><li>Monitor and improve system performance and data pipeline efficiency.</li><li>Ensure data security, compliance, and best practices.</li></ul><p><b>Required Skills:</b></p><ul><li>Strong proficiency in <b>Python</b> and object-oriented programming (OOPs).</li><li>1-3 Hands-on experience with <b>Hadoop, Hive, Spark, HDFS</b>.</li><li>Experience with ETL pipelines and data warehousing concepts.</li><li>Good understanding of SQL and NoSQL databases.</li><li>Knowledge of APIs, REST, and data integration workflows.</li><li>Ability to work with large, complex datasets.</li></ul><p>Job Types: Full-time, Permanent</p><p>Education:</p><ul><li>Bachelor's (Preferred)</li></ul><p>Experience:</p><ul><li>Python: 1 year (Preferred)</li></ul><p>Location:</p><ul><li>Mohali, Punjab (Preferred)</li></ul><p>Work Location: In person</p>

Big Data Python Engineer

Tata Consultancy Services (TCS)

<div></div><div><div><div><div>TCS Inviting Applications for Hyderabad Walk-In<br>
<br>
Role : AWS Data Engineer<br>
<br>
Exp : 4- 10 yrs<br>
<br>
Location : Hyderabad Walk-in<br>
<p>An impressive understanding of Management Information/Business Intelligence/Analytics</p>
<ul><li>Earlier experience of delivering Data orientated projects (data integration/ETL/Data Warehouse/Analytics)</li></ul>
<ul><li>An understanding of Data Warehousing and Big Data architecture principles.</li></ul>
<ul><li>Experience in building a data platform in the Microsoft Azure cloud</li></ul><br>
</div></div></div><div><div><div><b>Location</b></div>
<div>Hyderabad</div>
</div><div><div><b>Job Function</b></div>
<div>TECHNOLOGY</div>
</div><div><div><b>Role</b></div>
<div>Engineer</div>
</div><div><div><b>Job Id</b></div>
<div>384001</div>
</div><div><div><b>Desired Skills</b></div>
<div>AWS | Big Data</div>
</div></div><p>Desired Candidate Profile
</p><div><div><b>Qualifications</b> : BACHELOR OF COMPUTER APPLICATION, BACHELOR OF COMPUTER SCIENCE, BACHELOR OF ENGINEERING</div></div></div><div></div>

AWS Data Engineer

<div></div><div><div><div><div><ul><li> Designing and implementing highly performant data ingestion pipelines from multiple sources using Apache Spark and Databricks</li></ul>
<ul><li> Highly proficient with Python and PL/SQL</li></ul>
<ul><li> Experience in building ETL / data warehouse transformation processes</li></ul>
<ul><li> Strong experience with Big Data, Data analysis with excellent analytical and problem-solving abilities</li></ul>
<ul><li> Experience in Big Data components such as Spark SQL, Data frames, HIVE DB etc. implemented using Data Bricks would be preferred.</li></ul>
<ul><li> Hands on experience on Cloud based environments - AWS preferred</li></ul>
<ul><li> Hands on experience on AWS S3, Lambda, Redshift, Redshift Spectrum, Athena</li></ul>
<ul><li> Excellent understanding of the full SDLC lifecycle including both waterfall, agile and DevOps methodologies</li></ul>
<ul><li> Desire to work on a fast-paced, agile team</li></ul>
<ul><li> Excellent written and verbal communication skills</li></ul>
</div></div></div><div><div><div><b>Location</b></div>
<div>Hyderabad</div>
</div><div><div><b>Job Function</b></div>
<div>TECHNOLOGY</div>
</div><div><div><b>Role</b></div>
<div>Engineer</div>
</div><div><div><b>Job Id</b></div>
<div>383355</div>
</div><div><div><b>Desired Skills</b></div>
<div>AWS | BigData | PL/SQL | SQL</div>
</div></div><p>Desired Candidate Profile
</p><div><div><b>Qualifications</b> : BACHELOR OF ENGINEERING</div></div></div><div></div>

Data Engineer

ZecData Technology

<p><b>Job Description: Data Engineer</b></p><p><b>Position:</b> Data Engineer<br/><b>Location:</b> Indore<br/><b>Experience:</b> 0-7<br/><b>Employment Type:</b> Full-Time</p><p><b>Job Summary:</b></p><p>We are looking for a skilled Data Engineer to design, develop, and maintain data pipelines, ensuring efficient data processing, storage, and retrieval for business analytics and decision-making. The ideal candidate will have strong experience in working with large-scale data systems, ETL processes, and cloud-based platforms.</p><p><b>Key Responsibilities:</b></p><ul><li><b>Data Pipeline Development:</b></li><li>Design, build, and maintain scalable, reliable, and efficient data pipelines to process and integrate data from various sources.</li><li>Ensure the timely and accurate flow of data across systems.</li><li><b>Data Architecture:</b></li><li>Develop and optimize data models and architectures for data warehouses, data lakes, and databases.</li><li>Collaborate with data scientists, analysts, and other stakeholders to ensure data accessibility.</li><li><b>ETL/ELT Processes:</b></li><li>Design and implement ETL/ELT workflows to extract, transform, and load data from diverse sources into target systems.</li><li>Monitor and troubleshoot ETL processes to maintain data integrity.</li><li><b>Data Governance:</b></li><li>Implement data quality checks, validation processes, and monitoring to ensure accuracy and consistency.</li><li>Ensure compliance with data privacy regulations and organizational policies.</li><li><b>Cloud &amp; Big Data Platforms:</b></li><li>Work with cloud-based tools (AWS, Azure, GCP) and big data technologies (Hadoop, Spark) for data storage and processing.</li><li>Optimize performance and cost-effectiveness of cloud-based data solutions.</li><li><b>Collaboration:</b></li><li>Partner with software engineers, data analysts, and business teams to identify and fulfill data requirements.</li><li>Provide support for ad-hoc data requests and troubleshoot data issues.</li></ul><p><b>Key Skills and Qualifications:</b></p><ul><li><b>Technical Skills:</b></li><li>Proficiency in programming languages such as Python, Java, or Scala.</li><li>Strong knowledge of SQL and database systems (e.g., PostgreSQL, MySQL, MongoDB).</li><li>Hands-on experience with ETL tools (e.g., Apache NiFi, Talend, Informatica).</li><li>Familiarity with big data technologies like Hadoop, Spark, and Kafka.</li><li>Experience with cloud platforms (AWS, GCP, Azure) and related services (e.g., S3, Redshift, BigQuery).</li><li><b>Analytical Skills:</b></li><li>Solid understanding of data modeling and database design principles.</li><li>Ability to analyze and interpret large data sets to solve business problems.</li><li><b>Other Skills:</b></li><li>Strong problem-solving and debugging skills.</li><li>Excellent communication and teamwork abilities.</li><li>A keen eye for detail and commitment to delivering high-quality work.</li></ul><p>Job Types: Full-time, Permanent, Fresher</p><p>Pay: &#8377;25,000.00 - &#8377;50,000.00 per month</p><p>Benefits:</p><ul><li>Cell phone reimbursement</li><li>Paid sick time</li><li>Provident Fund</li></ul><p>Experience:</p><ul><li>total work: 1 year (Preferred)</li></ul><p>Work Location: In person</p>

SB Infowaves

<p>We are seeking a skilled <b>Data Engineer (Good exposure in Power BI)</b> to design, build, and optimize scalable data pipelines and infrastructure. The ideal candidate will work closely with data analysts, data scientists, and business stakeholders to ensure data is reliable, accessible, and efficient for analytics and decision-making.</p><p><b>Key Responsibilities</b></p><ul><li>Design, develop, and maintain robust <b>ETL/ELT pipelines</b> for structured and unstructured data.</li><li>Build and manage <b>data warehouses, data lakes, and real-time data platforms</b>.</li><li>Ensure <b>data quality, consistency, and security</b> across all data systems.</li><li>Optimize database performance and implement best practices for <b>scalability and reliability</b>.</li><li>Collaborate with data analysts, BI developers, and data scientists to understand data needs.</li><li>Automate workflows, monitoring, and error handling in data pipelines.</li><li>Work with <b>cloud platforms</b> (AWS, Azure, GCP) and modern data tools to deliver end-to-end solutions.</li><li>Implement governance standards and contribute to data documentation.</li></ul><p><b>Required Skills &amp; Qualifications</b></p><ul><li>Bachelor&rsquo;s/Master&rsquo;s degree in <b>Computer Science, Information Systems, or related field</b>.</li><li>Proven experience as a <b>Data Engineer / ETL Developer / Database Engineer</b>.</li><li>Strong knowledge of <b>SQL, Python, and/or Scala/Java</b>.</li><li>Hands-on experience with <b>data integration tools</b> (e.g., Apache Spark, Kafka, Airflow, Talend, Informatica).</li><li>Proficiency in <b>cloud data services</b>: AWS Redshift, Azure Synapse, Google BigQuery, or Snowflake.</li><li>Solid understanding of <b>data modeling, warehousing concepts, and big data technologies</b> (Hadoop, Hive, etc.).</li><li>Experience with <b>CI/CD pipelines, Git, Docker, Kubernetes</b> is a plus.</li><li>Excellent problem-solving, analytical, and communication skills.</li></ul><p>Job Type: Full-time</p><p>Pay: &#8377;600,000.00 - &#8377;900,000.00 per year</p><p>Experience:</p><ul><li>Data Engineer: 3 years (Preferred)</li></ul><p>Work Location: In person</p>

Search 10000+ Big data engineer jobs near you this December. New roles added regularly across leading industries. Apply today.

10000 big data engineer jobs NEAR_ME

Big Data Engineer

AI Big Data Engineer

Big Data Python Engineer

AWS Data Engineer

Data Engineer

Data Engineer

Data Engineer