Lead II - Software Engineering

7 - 9 Years
3 Openings
Hyderabad

Job Summary

We are seeking a skilled Data Engineer to design, build, and optimize scalable data pipelines and cloud-based data platforms. The role involves working with large-scale batch and real-time data processing systems, collaborating with cross-functional teams, and ensuring data reliability, security, and performance across the data lifecycle.

Key Responsibilities

ETL Pipeline Development & Optimization

• Design, develop, and maintain complex end-to-end ETL pipelines for large-scale data ingestion and processing.

• Optimize data pipelines for performance, scalability, fault tolerance, and reliability.

Big Data Processing

• Develop and optimize batch and real-time data processing solutions using Apache Spark (PySpark/Scala) and Apache Kafka.

• Ensure fault-tolerant, scalable, and high-performance data processing systems.

Cloud Infrastructure Development

• Build and manage scalable, cloud-native data infrastructure on AWS.

• Design resilient and cost-efficient data pipelines adaptable to varying data volume and formats.

Real-Time & Batch Data Integration

• Enable seamless ingestion and processing of real-time streaming and batch data sources (e.g., AWS MSK).

• Ensure consistency, data quality, and a unified view across multiple data sources and formats.

Data Analysis & Insights

• Partner with business teams and data scientists to understand data requirements.

• Perform in-depth data analysis to identify trends, patterns, and anomalies.

• Deliver high-quality datasets and present actionable insights to stakeholders.

CI/CD & Automation

• Implement and maintain CI/CD pipelines using Jenkins or similar tools.

• Automate testing, deployment, and monitoring to ensure smooth production releases.

Data Security & Compliance

• Collaborate with security teams to ensure compliance with organizational and regulatory standards (e.g., GDPR, HIPAA).

• Implement data governance practices ensuring data integrity, security, and traceability.

Troubleshooting & Performance Tuning

• Identify and resolve performance bottlenecks in data pipelines.

• Apply best practices for monitoring, tuning, and optimizing data ingestion and storage.

Collaboration & Cross-Functional Work

• Work closely with engineers, data scientists, product managers, and business stakeholders.

• Participate in agile ceremonies, sprint planning, and architectural discussions.

Skills & Qualifications

Mandatory (Must-Have) Skills

Good-to-Have (Preferred) Skills

• Knowledge of Java for data processing applications.

• Experience with NoSQL databases (e.g., DynamoDB, Cassandra, MongoDB).

• Familiarity with data governance frameworks and compliance tooling.

• Experience with monitoring and observability tools such as AWS CloudWatch, Splunk, or Dynatrace.

• Exposure to cost optimization strategies for large-scale cloud data platforms.

big data,scala spark,apache spark,etl pipeline development,