Big Data Engineer

Responsibilities:

· Develop, test, and maintain data processing pipelines and workflows, ensuring optimal performance and reliability.

· Collaborate with data scientists and analysts to understand data requirements and design appropriate data models and schemas.

· Optimize and tune existing data pipelines, storage systems, and analytical frameworks to improve efficiency and reduce processing time.

· Monitor and troubleshoot data processing jobs, performance bottlenecks, and data quality issues.

· Stay up-to-date with the latest advancements in big data technologies and recommend innovative solutions to improve our data infrastructure.

· Collaborate with cross-functional teams to gather requirements and define technical specifications for data-related projects.

· Document design decisions, technical specifications, and standard operating procedures for the big data infrastructure.

· Mentor junior engineers and provide guidance on best practices for big data engineering and development.

Requirements:

· Bachelor's or Master's degree in Computer Science, Information Systems, or a related field.

· Proven experience as a Big Data Engineer or similar role with expertise in Azure and Hadoop technologies.

· Strong knowledge of the Azure cloud platform, including Azure Data Lake Storage, Azure Data Factory, Azure Databricks, and Azure HDInsight.

· Proficiency in Hadoop ecosystem technologies such as HDFS, MapReduce, Hive, Pig, Spark, and HBase.

· Experience with big data processing frameworks like Apache Kafka and Apache Storm is a plus.

· Solid understanding of data modeling, data warehousing, and ETL processes.

· Strong programming skills in languages such as Java, Scala, or Python.

· Familiarity with SQL and NoSQL databases.

· Knowledge of data governance, data privacy, and data security best practices.

· Excellent problem-solving and analytical skills with a strong attention to detail.

· Ability to work effectively in a fast-paced, collaborative environment and manage multiple priorities.

· Strong communication and interpersonal skills, with the ability to effectively communicate technical concepts to both technical and non-technical stakeholders.

Job Type: Full-time

Work Location: In person