Big Data Platform Developer
- Doha, Qatar
Posted yesterday
About the role
We are looking for an experienced and innovative Big Data Platform Developer to join our technology and data engineering team.
The successful candidate will be responsible for designing, developing, implementing, and maintaining scalable big data platforms and data processing solutions. This role will work closely with data engineers, software developers, data scientists, architects, and business stakeholders to build high-performance data platforms that support analytics, reporting, artificial intelligence, and business intelligence initiatives.
The ideal candidate should have strong hands-on experience with big data technologies, distributed computing, data pipelines, cloud platforms, and modern data architecture.
Key Responsibilities
- Design, develop, and maintain scalable Big Data platforms and data processing solutions.
- Develop and optimize high-volume, high-performance data pipelines for structured and unstructured data.
- Build reliable ETL/ELT pipelines for data ingestion, transformation, processing, and integration.
- Work with distributed data processing technologies such as Apache Spark, Hadoop, Kafka, and related platforms.
- Design and implement data ingestion solutions from databases, APIs, applications, logs, IoT devices, and other data sources.
- Develop batch and real-time/streaming data processing solutions.
- Optimize data processing jobs, queries, storage, and platform performance.
- Implement data quality, validation, monitoring, and error-handling mechanisms.
- Develop scalable data storage solutions using data lakes, data warehouses, and modern data platforms.
- Integrate big data platforms with business intelligence, analytics, AI, and machine learning environments.
- Implement security, access control, encryption, and governance requirements across data platforms.
- Monitor platform performance, availability, capacity, and reliability.
- Troubleshoot and resolve data pipeline and platform-related issues.
- Automate deployment, testing, monitoring, and operational processes.
- Collaborate with DevOps and Cloud teams to implement CI/CD and infrastructure automation.
- Develop technical documentation, architecture diagrams, standards, and operational procedures.
- Participate in technology evaluations and recommend solutions to improve scalability, performance, and cost efficiency.
- Stay current with emerging big data, cloud, data engineering, and analytics technologies.
Required Qualifications
- Bachelor's degree in Computer Science, Information Technology, Software Engineering, Data Engineering, or a related field.
- 5+ years of experience in data engineering, big data development, data platforms, or a related technical role.
- Strong programming experience in Python, Java, or Scala.
- Strong experience with Apache Spark and distributed data processing.
- Hands-on experience developing and managing ETL/ELT data pipelines.
- Strong knowledge of SQL and relational databases.
- Experience with large-scale data storage and processing architectures.
- Good understanding of data modeling, data integration, and data warehousing concepts.
- Experience working with Linux/Unix environments.
- Strong understanding of software development lifecycle, version control, testing, and deployment practices.
- Strong analytical and problem-solving skills.
Work Location: In person