Lead Data Engineer - Spark, Scala

Role:Lead Data Engineer - Spark, Scala


Responsibilities

• Design, development and testing of components / modules involving Spark, Scala, Hive

and related big-data technologies in a datalake architecture

Implement design, development and deployment of new Data pipelines and maintain the
existing ones.

• Lead migrations such as move to public cloud ( Azure ), version upgrades.

• Oversee team of Data Engineers located in both offshore and onshore.

• Building views that are plugged by Data Visualization tools such as Power BI Release &

Deployment - Deploy using CD/CI practices and tools in various environments -
development, UAT and production and follow production processes.

• Ensure Craftsmanship practices are followed.

• Follow Agile@Scale process in terms of participation in PI Planning and follow-up, Sprint

planning, Back-log maintenance in Jira.

• Develop/propose tactical solutions for Project specific needs