Data Engineer

Job Summary:

We are seeking a skilled Big Data Engineer with expertise in Python, Hadoop, Spark, Shell scripting, PostgreSQL, and Linux to join our data engineering team. The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines and platforms to support our data analytics and business intelligence needs.

Key Responsibilities:

● Develop, maintain, and optimize large-scale data processing workflows using Hadoop and Apache Spark.

● Write efficient, reusable, and maintainable code in Python and Shell scripting for data ingestion, transformation, and automation tasks.

● Design and manage PostgreSQL databases including schema design, query optimization, and performance tuning.

● Manage Linux-based environments and tools for data platform deployment, monitoring, and troubleshooting.

● Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver reliable data solutions.

● Implement data quality checks and data validation processes.

● Monitor and optimize data pipeline performance and resource utilization.

● Document workflows, processes, and system architecture to ensure maintainability and knowledge sharing.

Required Skills and Qualifications:

● Strong programming skills in Python for data processing and automation.

● Hands-on experience with Hadoop ecosystem components (HDFS, MapReduce, YARN).

● Proficiency in Apache Spark for distributed data processing and analytics.

● Expertise in Shell scripting (Bash, etc.) for task automation and system management.

● Experience working with PostgreSQL database systems, including advanced SQL and performance tuning.

● Strong familiarity with Linux operating systems and command-line tools.

● Understanding of data modeling, ETL processes, and big data architecture.

● Familiarity with version control systems such as Git.

● Excellent problem-solving skills and ability to work in a collaborative team environment.

● Good communication skills, both written and verbal.

Preferred Qualifications:

● Bachelors Degree in Computer Science or related field.

● Experience with containerization (Docker, Kubernetes) and cloud platforms (AWS, Azure, GCP).

● Knowledge of other big data tools like Kafka, Hive, or Airflow.

● Prior experience working in Agile/Scrum teams.

Job Type: Full-time

Pay: ₹500,000.00 - ₹2,000,000.00 per year

Work Location: In person