big data developer

Brix

Our projects involve building advanced tools, automations, and AI integrations to process and analyze massive datasets (100M+ records). We are looking for a highly skilled Full Stack Developer with strong PostgreSQL, MySQL, and big data experience to join our remote team and help deliver game-changing solutions across multiple businesses.Key Responsibilities:<ul><li>Manage &amp; optimize PostgreSQL &amp; MySQL databases.</li><li>Work with big data (100M+ records) efficiently.</li><li>Consolidate multiple database servers.</li><li>Build custom reports, dashboards, pipelines.</li><li>Link databases with AI for natural queries.</li><li>Deploy &amp; debug custom-built applications.</li><li>Build automations in n8n &amp; similar tools.</li><li>Turn concepts into fully functional products.</li></ul>Required Skills (Max 40 chars each):<ul><li>Full stack development (Node/Python/PHP)</li><li>PostgreSQL database management</li><li>MySQL database management</li><li>Big data query optimization</li><li>Database architecture &amp; indexing</li><li>Linux server administration</li><li>API integration (AI tools)</li><li>n8n / Zapier / Make automation</li><li>Problem solving &amp; independent work</li><li>English communication skills</li></ul>Bonus Skills (Max 40 chars each):<ul><li>Data visualization (Tableau/Power BI)</li><li>Distributed DB / Data warehouse tools</li><li>DevOps (Docker, CI/CD)</li><li>Call center dialer platforms</li></ul>What We Offer:<ul><li>Competitive salary based on skills &amp; experience.</li><li>Fully remote position.</li><li>Long-term career growth opportunities.</li><li>Work on cutting-edge AI, automation, big data.</li></ul>Job Types: Full-time, Part-timePay: Rs300,000.00 - Rs550,000.00 per monthWork Location: Remote

Full Stack Developer – PostgreSQL / Big Data / AI & Automation

NTT DATA

<div>Req ID: 344790
 
 
NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. 
 
We are currently seeking a Data Engineer - DBT to join our team in Bangalore, Karn&#257;taka (IN-KA), India (IN). 
 
&quot;Job Duties: We are seeking an Engineer to join our DBI organization within Data Scientist &amp; Data Engineering team. 
Your role 
As a Data Engineer, your main responsibilities will involve: 
<ul><li>strong experience working in projects in DBT.</li><li>The project is a migration of a SAS code to DBT code, so it's important that the candidate has experience in this type of projects.</li><li>Analyze data models and derive logical conclusions.</li><li>Processes modeling.</li><li>Hands on development and monitoring of the Azure cloud Platform and various associated components for data ingestion, transformation and processing.</li><li>Effectively diagnose, isolate, and resolve complex problems pertaining to data infrastructure, including performance tuning and optimization.</li><li>Designs and writes programs according to functional and non-functional requirements.</li><li>Develops and maintains technical documentation.</li><li>Follows established configuration/change control processes.</li></ul> 
Minimum Skills Required: As a Big Data Developer / Data Engineer, your skills and experience will ideally include: 
<ul><li>University level education or equivalent</li><li>2+ years of experience working as a Data Engineer</li><li>Knowledge of design patterns</li><li>Ability to write complex SQL queries</li><li>A good understanding of distributed computation</li><li>DBT</li><li>GIT</li><li>Experience with Azure cloud.</li><li>Experience with Databricks.</li></ul>
 
Nice to have: 
<ul><li>Python</li><li>1+ Experience in functional programming (preferably Scala - big plus)</li><li>1+ years of experience in Apache Spark 2.0 (big plus)</li><li>Experience in R programming</li><li>Experience with Hadoop technologies (i.e. Sqoop, Hive, Flume etc.)</li><li>Continuous Integration / Continuous Deployment (Azure DevOps)</li><li>Experience in data exploration using notebooks (Databricks notebooks).</li><li>Experience in automation of regular tasks(e.g. data refreshes) using bash and python scripting.&quot;</li></ul> 

About NTT DATA 
 
NTT DATA is a $30+ billion business and technology services leader, serving 75% of the Fortune Global 100. We are committed to accelerating client success and positively impacting society through responsible innovation. We are one of the world's leading AI and digital infrastructure providers, with unmatched capabilities in enterprise-scale AI, cloud, security, connectivity, data centers and application services. Our consulting and industry solutions help organizations and society move confidently and sustainably into the digital future. As a Global Top Employer, we have experts in more than 50 countries. We also offer clients access to a robust ecosystem of innovation centers as well as established and start-up partners. NTT DATA is part of NTT Group, which invests over $3 billion each year in R&amp;D.</div>

Data Engineer - DBT

Icogz Pvt Ltd

Job description:About Us: At icogz Technologies Pvt Ltd, we are pioneers in harnessing the power of generative AI and proprietary algorithms to transform the complexity of Big Data into clear, actionable insights. Our unique approach recognizes the challenges posed by fragmented data silos, which stem from the diverse perspectives of various stakeholders within a company.By utilizing our advanced proprietary algorithms and streamlined data processing flows, we effectively integrate and analyze vast, disparate datasets. This process delivers a unified, cohesive view of your business operations, enabling real-time analysis and insightful decision-making.Our solutions at icogz Technologies empower businesses to swiftly adapt and capitalize on strategic opportunities. By turning big data into a navigable landscape of insights, we help enhance operational agility, improve performance, and drive profitability.Role Overview: We&rsquo;re seeking a seasoned Python Developer to architect and deliver high-performance, scalable backend solutions. You&rsquo;ll be involved in the development of robust APIs, optimize data processing workflows, and ensure code quality while adhering to secure, efficient, and SaaS-ready practices. Your expertise in Django, FastAPI, PostgreSQL, Document DB, and big data will drive our platform&rsquo;s evolution.Key Responsibilities:<ul><li>Design, develop, and maintain RESTful APIs using Django/DRF and FastAPI for big data analytics and LLM based applications.</li><li>Architect, Develop and Maintain multi-tenant SaaS solutions with security-first coding practices (e.g., OWASP and AVS2 standards).</li><li>Optimize PostgreSQL performance for large datasets, including query tuning, indexing, and partitioning.</li><li>Manage and Optimize Document DB operations: Design, implement, and maintain databases in Document DB (e.g., MongoDB, Couchbase) to ensure efficient data storage, retrieval, and scalability. Optimize database queries, indexes, and ensure high availability and disaster recovery strategies are in place.</li><li>Process and analyze big data efficiently using Pandas, PySpark, or similar libraries.</li><li>Integrate third-party APIs (payment gateways, AI/ML services, marketplaces) with fault tolerance and monitoring.</li><li>Implement caching strategies (Redis, Memcached) to reduce latency and improve scalability.</li><li>Write async-friendly code and leverage concurrency for performance-critical tasks.</li><li>Develop dynamic, reusable code components to support rapid iteration in marketplace or SaaS environments.</li><li>Ensure compliance with secured coding practices (authentication, encryption, vulnerability mitigation).</li><li>Collaborate with data scientists/ML engineers to deploy and optimize AI/ML models in production.</li><li>Stay updated with AI-driven tools (e.g., GitHub Copilot) to enhance development efficiency.</li></ul>Requirements:<ul><li>2-5 years of Python development experience, with expert-level proficiency in Django and FastAPI.</li><li>Strong understanding of REST API design, authentication (OAuth2, JWT), and versioning.</li><li>Hands-on experience with PostgreSQL, including query optimization for large datasets.</li><li>Expertise in Document DB systems (e.g., MongoDB, Couchbase), including schema design, query optimization, and storage management.</li><li>Proficient in Pandas, NumPy, or similar libraries for data transformation/analytics.</li><li>Familiarity with async frameworks (e.g., asyncio, Celery) and event-driven architectures.</li><li>Experience building multi-tenant SaaS systems (isolated databases, tenant-aware middleware).</li><li>Knowledge of big data tools (Spark, Dask, pyArrow) and efficient file processing (Parquet, CSV).</li><li>Expertise in caching mechanisms.</li><li>Proven track record of integrating third-party APIs (social media platforms, payment systems, cloud services).</li><li>Adaptability to AI tools.</li><li>Strong problem-solving skills and a commitment to writing clean, maintainable code.</li></ul>Nice-to-Have:<ul><li>Experience with Django Channels for WebSocket/real-time applications.</li><li>Familiarity with cloud platforms (AWS, Azure) and containerization (Docker, Kubernetes).</li><li>Exposure to marketplace ecosystems (e.g., pricing engines, vendor APIs).</li><li>Contributions to open-source projects or tech community involvement.</li><li>Monitoring tools like Grafana, Prometheus.</li><li>Awareness of AI/ML concepts (model deployment, data pipelines) and adaptability to AI tools.</li><li>Exposure to analytical Databases like ClickHouse, DuckDB.</li></ul>Why Join Us?<ul><li>Opportunities to work on AI/ML-driven projects.</li><li>Continuous learning.</li><li>Collaborative culture with a focus on innovation.</li></ul>Job Types: Full-time, PermanentBenefits:<ul><li>Paid time off</li><li>Provident Fund</li></ul>Application Question(s):<ul><li>How many years of experience do you have with Fast API?</li><li>How many years of experience do you have with PostgreSQL?</li></ul>Experience:<ul><li>Python: 3 years (Required)</li></ul>Location:<ul><li>Mumbai, Maharashtra (Required)</li></ul>Work Location: In person

Backend Developer

Prodapt Solutions

<div>Overview:<div></div> 
We are looking for a highly skilled Big Data Engineer with extensive experience in Spark and Scala to join our team. The ideal candidate will play a crucial role in designing, developing, and optimizing large-scale data processing systems. You will work closely with data scientists, analysts, and other stakeholders to deliver high-quality data solutions.
Responsibilities:

Key Responsibilities: -<ul><li>
Design, develop, and maintain scalable data pipelines using Apache Spark and Scala.</li><li>
Collaborate with cross-functional teams to understand data requirements and deliver data solutions that meet business needs.</li><li>
Optimize Spark jobs for performance and cost-efficiency in a distributed computing environment.</li><li>
Implement best practices for data modeling, ETL processes, and data governance.</li><li>
Monitor and troubleshoot data processing workflows to ensure data integrity and availability.</li><li>
Work with cloud platforms (AWS, Azure, or GCP) to implement big data solutions.</li><li>
Stay up to date with industry trends and emerging technologies in big data and analytics.</li></ul><div><div>
</div></div> 
<div></div>
Requirements:
<ul><li>7-10 years of experience in Big Data technologies, with a strong focus on Apache Spark and Scala.</li><li>
Proficiency in data processing frameworks (Hadoop, Spark) and languages (Scala, Java).</li><li>
Experience with data warehousing solutions (Snowflake, Redshift, etc.) and SQL.</li><li>
Knowledge of data modeling, ETL processes, and data visualization tools (Tableau, Power BI).</li><li>
Familiarity with cloud services (AWS, Azure, Google Cloud) and containerization (Docker, Kubernetes).</li><li>
Strong analytical skills and the ability to work with large datasets.</li><li>
Excellent communication and teamwork skills.</li></ul></div><div></div>

BigData- Spark/Scala

S&P Global

<div>
 
 About The Role
 
 
 
 
 Grade Level (for internal use):
 
 
 
 11
 
 
 
 The Role: Lead Software Developer
 
 
 
 
 Location:
 
 Islamabad, Pakistan
 
 
 
 The Team
 
 : We are looking for a highly motivated, enthusiastic, and skilled software engineer for S&amp;P Global Market Intelligence. This developer would help us to accelerate our Metals and Mining project GAIA
 
 
 
 The Impact:
 
 As a Software Developer, you will be part of the development team that manages and supports the Internal applications that are supporting Metals and Mining Dataset.
 
 
 
 What&rsquo;s in it for you:
 
 It&rsquo;s a fast-paced agile environment that deals with huge volumes of data, so you&rsquo;ll have an opportunity to sharpen your data skills and work on an emerging technology stack.
 
 
 
 Responsibilities
 
 
 
 <ul>
 <li>
 Solve problems, analyze, and isolate issues
 </li>
 <li>
 Build solutions to develop/support key business needs
 </li>
 <li>
 Engineer components and common services based on standard development models, languages, and tools
 </li>
 <li>
 Produce system design documents
 </li>
 <li>
 Collaborate effectively with technical and non-technical partners
 </li>
 <li>
 Quickly learn new and internal technologies.
 
 
 </li>
 </ul>
 
 Basic Qualifications
 
 
 
 
 What We&rsquo;re Looking For:
 
 
 
 <ul>
 <li>
 Bachelor's / master&rsquo;s degree in computer science, Information Systems or equivalent.
 </li>
 <li>
 Minimum of 5 years of experience in application development using Microsoft and Big Data Technologies.
 </li>
 <li>
 Strong Knowledge of object-oriented design, .NET, .NET Core, C#, SQL Server, and design patterns including MVVM.
 </li>
 <li>
 Must have worked on multi-tier applications.
 </li>
 <li>
 Experience building applications using Win forms, WPF, ADO.net, and Entity Framework.
 </li>
 <li>
 Have worked on Windows services and scheduler applications using Dotnet and C#.
 </li>
 <li>
 Must have good command on Bigdata and Cloud-based technologies including Snowflake, AWS, OData, Postgres, Databricks
 </li>
 <li>
 AWS knowledge and expertise is a MUST for this position.
 </li>
 <li>
 Experienced with software development lifecycle (SDLC) methodologies like Agile and Test-driven development. Agile experience is mandatory.
 </li>
 <li>
 Experienced with developing solutions involving relational database technologies on SQL Server platform, stored procedure programming experience using Transact SQL.
 </li>
 <li>
 Experience with troubleshooting applications, debugging, logging, performance monitoring, data visualization, and data analytics
 </li>
 <li>
 Must be a team player and quick learner, and willing to take on difficult tasks as and when required.
 </li>
 <li>
 Experience with other technologies including Azure Cloud, Google Cloud, Scala, and Docker is a plus but not mandatory.
 
 
 </li>
 </ul>
 
 Preferred Qualifications
 
 
 
 <ul>
 <li>
 Mapping technologies, Containerization, Docker,
 
 
 </li>
 </ul>
 #
 
 
 
 About S&amp;P Global Commodity Insights
 
 
 
 At S&amp;P Global Commodity Insights, our complete view of global energy and commodities markets enables our customers to make decisions with conviction and create long-term, sustainable value.
 
 
 We&rsquo;re a trusted connector that brings together thought leaders, market participants, governments, and regulators to co-create solutions that lead to progress. Vital to navigating Energy Transition, S&amp;P Global Commodity Insights&rsquo; coverage includes oil and gas, power, chemicals, metals, agriculture and shipping.
 
 
 S&amp;P Global Commodity Insights is a division of S&amp;P Global (NYSE: SPGI). S&amp;P Global is the world&rsquo;s foremost provider of credit ratings, benchmarks, analytics and workflow solutions in the global capital, commodity and automotive markets. With every one of our offerings, we help many of the world&rsquo;s leading organizations navigate the economic landscape so they can plan for tomorrow, today.
 
 
 For more information, visit http://www.spglobal.com/commodity-insights.
 
 
 
 What&rsquo;s In It For
 
 
 You?
 
 
 
 
 Our Purpose
 
 
 
 Progress is not a self-starter. It requires a catalyst to be set in motion. Information, imagination, people, technology&ndash;the right combination can unlock possibility and change the world.
 
 
 Our world is in transition and getting more complex by the day. We push past expected observations and seek out new levels of understanding so that we can help companies, governments and individuals make an impact on tomorrow. At S&amp;P Global we transform data into Essential Intelligence&reg;, pinpointing risks and opening possibilities. We Accelerate Progress.
 
 
 
 Our People
 
 
 
 We're more than 35,000 strong worldwide&mdash;so we're able to understand nuances while having a broad perspective. Our team is driven by curiosity and a shared belief that Essential Intelligence can help build a more prosperous future for us all.
 
 
 From finding new ways to measure sustainability to analyzing energy transition across the supply chain to building workflow solutions that make it easy to tap into insight and apply it. We are changing the way people see things and empowering them to make an impact on the world we live in. We&rsquo;re committed to a more equitable future and to helping our customers find new, sustainable ways of doing business. We&rsquo;re constantly seeking new solutions that have progress in mind. Join us and help create the critical insights that truly make a difference.
 
 
 
 Our Values
 
 
 
 
 Integrity, Discovery, Partnership
 
 
 
 At S&amp;P Global, we focus on Powering Global Markets. Throughout our history, the world's leading organizations have relied on us for the Essential Intelligence they need to make confident decisions about the road ahead. We start with a foundation of
 
 integrity
 
 in all we do, bring a spirit of
 
 discovery
 
 to our work, and collaborate in close
 
 partnership
 
 with each other and our customers to achieve shared goals.
 
 
 
 Benefits
 
 
 
 We take care of you, so you can take care of business. We care about our people. That&rsquo;s why we provide everything you&mdash;and your career&mdash;need to thrive at S&amp;P Global.
 
 
 
 Our Benefits Include
 
 
 
 <ul>
 <li>
 Health &amp; Wellness: Health care coverage designed for the mind and body.
 </li>
 <li>
 Flexible Downtime: Generous time off helps keep you energized for your time on.
 </li>
 <li>
 Continuous Learning: Access a wealth of resources to grow your career and learn valuable new skills.
 </li>
 <li>
 Invest in Your Future: Secure your financial future through competitive pay, retirement planning, a continuing education program with a company-matched student loan contribution, and financial wellness programs.
 </li>
 <li>
 Family Friendly Perks: It&rsquo;s not just about you. S&amp;P Global has perks for your partners and little ones, too, with some best-in class benefits for families.
 </li>
 <li>
 Beyond the Basics: From retail discounts to referral incentive awards&mdash;small perks can make a big difference.
 
 
 </li>
 </ul>
 For more information on benefits by country visit: https://spgbenefits.com/benefit-summaries
 
 
 
 Global Hiring And Opportunity At S&amp;P Global
 
 
 
 At S&amp;P Global, we are committed to fostering a connected and engaged workplace where all individuals have access to opportunities based on their skills, experience, and contributions. Our hiring practices emphasize fairness, transparency, and merit, ensuring that we attract and retain top talent. By valuing different perspectives and promoting a culture of respect and collaboration, we drive innovation and power global markets.
 
 
 
 Recruitment Fraud Alert
 
 
 
 If you receive an email from a spglobalind.com domain or any other regionally based domains, it is a scam and should be reported to reportfraud@spglobal.com. S&amp;P Global never requires any candidate to pay money for job applications, interviews, offer letters, &ldquo;pre-employment training&rdquo; or for equipment/delivery of equipment. Stay informed and protect yourself from recruitment fraud by reviewing our guidelines, fraudulent domains, and how to report suspicious activity here.
 
 
 
 Equal Opportunity Employer
 
 
 
 S&amp;P Global is an equal opportunity employer and all qualified candidates will receive consideration for employment without regard to race/ethnicity, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, marital status, military veteran status, unemployment status, or any other status protected by law. Only electronic job submissions will be considered for employment.
 
 
 If you need an accommodation during the application process due to a disability, please send an email to: EEO.Compliance@spglobal.com and your request will be forwarded to the appropriate person.
 
 
 
 US Candidates Only:
 
 The EEO is the Law Poster http://www.dol.gov/ofccp/regs/compliance/posters/pdf/eeopost.pdf describes discrimination protections under federal law. Pay Transparency Nondiscrimination Provision - https://www.dol.gov/sites/dolgov/files/ofccp/pdf/pay-transp_%20English_formattedESQA508c.pdf
 
 
 20 - Professional (EEO-2 Job Categories-United States of America), IFTECH202.2 - Middle Professional Tier II (EEO Job Group), SWP Priority &ndash; Ratings - (Strategic Workforce Planning)
 
 
 
 Job ID:
 
 320198
 
 
 
 Posted On:
 
 2025-09-24
 
 
 
 Location:
 
 Islamabad, Islamabad, Pakistan
 
 
</div>

Lead Software Developer - Backend

QuantumBlack, AI by McKinsey

<div>
 
 Who You'll Work With
 
 
 
 Driving lasting impact and building long-term capabilities with our clients is not easy work. You are the kind of person who thrives in a high performance/high reward culture - doing hard things, picking yourself up when you stumble, and having the resilience to try another way forward.
 
 
 In return for your drive, determination, and curiosity, we'll provide the resources, mentorship, and opportunities you need to become a stronger leader faster than you ever thought possible. Your colleagues&mdash;at all levels&mdash;will invest deeply in your development, just as much as they invest in delivering exceptional results for clients. Every day, you'll receive apprenticeship, coaching, and exposure that will accelerate your growth in ways you won&rsquo;t find anywhere else.
 
 
 When you join us, you will have:
 
 
 <ul>
 <li>
 Continuous learning: Our learning and apprenticeship culture, backed by structured programs, is all about helping you grow while creating an environment where feedback is clear, actionable, and focused on your development. The real magic happens when you take the input from others to heart and embrace the fast-paced learning experience, owning your journey.
 </li>
 <li>
 A voice that matters: From day one, we value your ideas and contributions. You&rsquo;ll make a tangible impact by offering innovative ideas and practical solutions. We not only encourage diverse perspectives, but they are critical in driving us toward the best possible outcomes.
 </li>
 <li>
 Global community: With colleagues across 65+ countries and over 100 different nationalities, our firm&rsquo;s diversity fuels creativity and helps us come up with the best solutions for our clients. Plus, you&rsquo;ll have the opportunity to learn from exceptional colleagues with diverse backgrounds and experiences.
 </li>
 <li>
 World-class benefits: On top of a competitive salary (based on your location, experience, and skills), we provide a comprehensive benefits package to enable holistic well-being for you and your family.
 
 
 </li>
 </ul>
 
 Your Impact
 
 
 
 You will apply your passion for finding opportunities in data by using established tools and techniques or by innovating/building new ones.
 
 
 In this role, you will collaborate with highly talented business consultants and data-scientists to design and drive architecture and implementation planning for our clients and act as a big data evangelist, educating McKinsey's and client's teams on data engineering topics. You&rsquo;ll actively cooperate with data scientists and digital consultants to incorporate clients&rsquo; requirements and impactful analysis into our data products. You&rsquo;ll also be an expert in data engineering and advise relevant people at the firm on technical aspects of analytics engagements.
 
 
 You will shape the future of what data-driven organizations look like, drive processes for extracting and using that data in creative ways, and create new lines of thinking within an infinite number of clients and situations.
 
 
 Lastly, you'll do meaningful work that makes a real difference with our clients. This is a hybrid client-facing/technical role using state-of-the-art technologies, whilst also being able to communicate complex intractable ideas to non-technical audiences.
 
 
 
 Your Qualifications and Skills
 
 
 
 <ul>
 <li>
 Bachelor's degree in computer science, engineering, mathematics or another relevant subject
 </li>
 <li>
 3+ years of experience in big data related software development; experience with data modeling, design patterns and building highly scalable and secured solutions
 </li>
 <li>
 Background in development with practical experience with at least one general purpose programming languages (Python, R, C, Java, Scala, etc.)
 </li>
 <li>
 Familiar with big data platforms (SQL, Teradata, Hadoop, Spark)
 </li>
 <li>
 Experience preparing data for analytics and following a data science workflow
 </li>
 <li>
 Advanced knowledge of data structures and algorithms, networking, operating systems, standard unix/linux operating systems, version control and DevSecOps tool chains, etc.
 </li>
 <li>
 Ability to work with a myriad of data types (structured and unstructured), extracting information from modern and legacy data platforms and identify and perform linkages across disparate data sets
 </li>
 <li>
 Meaningful experience in extracting and loading data from multiple database technologies like RDBMS (MS SQL Server, Oracle, MySQL, PostgreSQL), MPP (AWS Redshift, Teradata, Snowflake) and optionally NoSQL (MongoDB, DynamoDB, Cassandra, Neo4J, Titan)
 </li>
 <li>
 Experience and interest in Cloud platforms such as AWS, Azure, GCP
 </li>
 <li>
 Commercial client-facing project experience is helpful
 </li>
 <li>
 Proven record of leadership in a work setting and/or through extracurricular activities
 </li>
 <li>
 Ability to work collaboratively in a team environment and effectively with people at all levels in an organization
 </li>
 <li>
 Ability to communicate complex ideas effectively, both verbally and in writing, in English; fluency in local office language(s) is a plus
 </li>
 </ul>
</div>

Data Engineer - QuantumBlack, AI by McKinsey

Pennant Technologies

<div><div>Job Title: Senior Big Data Engineer</div><div>
Department: Data &amp; Analytics</div><div>
Employment Type: Full&#8209;time</div><div>
Role Overview</div><div>
We are seeking an experienced Big Data Engineer to join our growing Data &amp; Analytics team. In this role, you&rsquo;ll design, build, and optimize our modern data platform&mdash;enabling scalable pipelines, reliable data lakes, and robust ML operations. You&rsquo;ll work closely with data scientists, analysts, and stakeholders to turn raw data into actionable insights.</div><div>
Key Responsibilities</div><ul><li>
Data Platform Development &amp; Maintenance</li><li>
Design, deploy, and maintain ETL/ELT pipelines using Apache Airflow and dbt/meltano.</li><li>
Architect and manage data lakes on HDFS and object storage (e.g., S3/HDFS).</li><li>
Implement table formats and governance with Apache Iceberg and Unity Catalog/Apache ranger/Atlas.</li><li>
Big Data Processing</li><li>
Develop distributed data processing jobs in Apache Spark (Scala/Java/Python).</li><li>
Optimize Spark applications for performance, cost, and scalability.</li><li>
Good working knowledge in Databases (Postgres, Mongo, SQL, Sql server etc)</li><li>
Collaboration with Data science teams</li><li>
Integrate MLflow for experiment tracking, model registry, and seamless deployment.</li><li>
Collaborate with data science teams to operationalize models in production.</li><li>
Containerization &amp; Orchestration</li><li>
Containerize data services and workflows using Docker/kubenetes.</li><li>
Orchestrate deployments on Kubernetes clusters (EKS/GKE/AKS).</li><li>
Center of Excellence &amp; Best Practices</li><li>
Contribute to and govern the dbt Center of Excellence: define standards, code reviews, and documentation.</li><li>
Advocate and enforce data engineering best practices: CI/CD, testing, monitoring, and observability.</li><li>
Cross&#8209;Functional Collaboration</li><li>
Partner with Analytics, Product, and Engineering teams to understand requirements and deliver solutions.</li><li>
Mentor junior engineers and share knowledge through code reviews and technical talks.</li></ul><div>
Required Skills &amp; Experience</div><ul><li>
5+ years of professional experience in Big Data engineering or similar roles.</li><li>
Airflow: Hands&#8209;on experience authoring DAGs, configuring operators/sensors, and managing scheduler/executor backends.</li><li>
Apache Spark: Proficient in building and tuning large&#8209;scale ETL jobs (Scala/Python).</li><li>
Apache Iceberg &amp; Unity Catalog: Familiarity with table formats, partitioning strategies, and catalog management.</li><li>
Data Lake &amp; HDFS: Solid understanding of storage architectures, file formats (Parquet/ORC), and data partitioning.</li><li>
MLflow: Working knowledge of experiment tracking, model registry, and deployment workflows.</li><li>
Kubernetes: Skilled in deploying, scaling, and debugging containerized applications and stateful workloads.</li><li>
dbt COE: Experience with dbt for data modeling, CI/CD integration, and establishing a CoE for standards.</li><li>
Strong programming skills in Python and/or Scala.</li><li>
Experience with version control (Git), CI/CD pipelines (Jenkins/GitHub Actions), and monitoring tools (Prometheus, Grafana).</li></ul><div>
Nice&#8209;to&#8209;Have</div><ul><li>
Familiarity with cloud platforms (AWS, GCP, or Azure) and managed services (EMR, Dataproc, EKS, GKE).</li><li>
Knowledge of additional data catalogs (e.g., AWS Glue, Hive Metastore).</li><li>
Background in event&#8209;driven architectures (Kafka, Pulsar).</li><li>
Prior involvement in data governance or compliance initiatives.</li></ul><div>
Education &amp; Certifications</div><ul><li>
Bachelor&rsquo;s or Master&rsquo;s degree in Computer Science, Engineering, or related field.</li><li>
Relevant certifications (e.g., Any cloud certification AWS/GCP/Azure/Databricks Certified Big Data &ndash; Specialty) are a plus.</li></ul><div>
Soft Skills</div><ul><li>
Excellent problem&#8209;solving and analytical abilities.</li><li>
Strong written and verbal communication; able to explain technical concepts to non&#8209;technical stakeholders.</li><li>
Proactive mindset with a passion for learning and driving innovation.</li><li>
Collaborative team player who thrives in agile, fast&#8209;paced environments.</li></ul><div>
Join us and help shape the future of our data platform&mdash;enabling smarter decisions, faster insights, and transformative business outcomes!</div></div><div></div>

Big Data Engineer

Search 10000+ Big data developer jobs near you this October. New roles added regularly across leading industries. Apply today.