Lead Engineer (AI/ML - Notebook)

• Develop, maintain, and enhance the Notebook platform built on JupyterLab, supporting both cloud and on-premises deployments.
• Integrate and optimize connections to diverse data sources (BigQuery, Hive, Teradata, Hadoop, Spark, etc.).
• Enable and support distributed data processing and analytics using Spark/PySpark and Dataproc.
• Implement and maintain platform features for code execution, visualization, collaboration, and scheduling (e.g., Airflow integration).
• Ensure platform security, including SSO, role-based access control, and compliance with governance and data privacy standards.
• Support and automate ML workflows, including model training, experiment tracking (e.g., MLflow), and deployment.
• Monitor platform health, troubleshoot issues, and optimize for performance, scalability, and reliability (e.g., GKE, Kubernetes, Openshift).
• Collaborate with data scientists, engineers, and product teams to deliver robust, user-friendly solutions.
• Contribute to documentation, onboarding guides, and best practices for users and developers.

Required Technical Skills

• Proficiency in Python, especially for backend development and data engineering.
• Experience with JupyterLab/Jupyter Notebooks, including kernel and extension development.
• Familiarity with distributed data processing frameworks (Spark, PySpark, Dataproc).
• Experience with cloud platforms (GCP preferred), Kubernetes, and containerization (Docker).
• Strong understanding of data source integration (BigQuery, Hive, Teradata, Hadoop).
• Experience with CI/CD, DevOps, and monitoring tools for cloud-based platforms.
• Knowledge of security best practices (SSO, RBAC, credential management).
• Familiarity with ML/AI workflows, experiment tracking (MLflow), and model deployment.
• Excellent troubleshooting, debugging, and communication skills.

Preferred Qualifications

• Experience with platform governance, code classification, and compliance automation.
• Familiarity with VS Code integration, workflow automation tools (Zonkey, Airflow), and dashboarding (Tableau).
• Experience supporting multi-tenant, highly available, and scalable notebook environments.
• Exposure to large-scale financial/ML platforms.