Hadoop Administrator / Big Data Platform Engineer

We are looking for an experienced Hadoop Administrator to design, build, and maintain our Hadoop ecosystem based on the Open Data Platform (ODP) framework. The ideal candidate will have hands-on experience in the installation, configuration, tuning, and administration of key Hadoop components, including HDFS, YARN, Hive, Spark, Ranger, and Oozie.

This role involves end-to-end platform setup, ongoing maintenance, performance optimization, and support for enterprise big data workloads.

Key Responsibilities

• Install, configure, and manage Hadoop clusters and ecosystem components (HDFS, YARN, Hive, Spark, Zookeeper, Oozie, etc.) on ODP-compliant distributions.
• Build and deploy Hadoop stacks from scratch, including hardware sizing, capacity planning, and architecture design.
• Implement cluster high availability (HA), backup/recovery, and disaster recovery strategies.
• Manage user access, security policies, and Kerberos/Ranger configurations.
• Perform cluster performance tuning, troubleshooting, and log analysis to ensure system stability.
• Monitor system health and optimize resource utilization using Ambari, Cloudera Manager, or other monitoring tools.
• Automate cluster operations using shell scripts or Python for deployment, maintenance, and patching.
• Collaborate with data engineering and infrastructure teams for upgrades, migrations, and platform integrations.
• Maintain detailed documentation for architecture, configurations, and operational runbooks.

Required Skills & Experience

• 5–10 years of experience in Hadoop ecosystem administration.
• Proven experience building Hadoop clusters from scratch using ODP distributions (Hortonworks, Cloudera, or similar).
• Strong expertise in:
  • HDFS, YARN, Hive, Spark, Zookeeper, Oozie
  • Ambari or Cloudera Manager (installation, service management, and monitoring)
  • Kerberos, Ranger, or Sentry for security and authorization.
  • Proficiency in Linux system administration, shell scripting, and configuration management.
  • Experience with performance tuning, capacity planning, and troubleshooting in production environments.
  • Familiarity with HA configurations, NameNode failover, and cluster scaling.

Education

• Regular MCA or Bachelor’s degree in Computer Science, Information Technology, Engineering, or equivalent experience.

Nice to Have

• Experience with cloud-based Hadoop environments (AWS EMR, Azure HDInsight, GCP Dataproc).
• Exposure to containerized big data platforms (Kubernetes, Docker).
• Knowledge of automation tools (Ansible, Terraform, Puppet).
• Experience with Kafka, Airflow, or NiFi for data pipeline integration.\
• Understanding of data governance, auditing, and monitoring best practices.