Senior Databricks AI Engineer

Overview:

The Senior AI/ML Engineer will modernize and scale the company’s enterprise data and AI platform by designing AI-ready data models, operationalizing ML systems, and enabling natural-language analytics through Databricks Genie or equivalent AI tooling.

This role exists to shift the organization from dashboard-driven analytics to AI-powered decision intelligence at enterprise scale.

Responsibilities:

Performance Objectives

Modernize and Operationalize the Analytics Data Platform

Within 6–9 months, design and implement a scalable medallion-based architecture (Bronze/Silver/Gold) in Databricks or Snowflake that supports AI-ready datasets, improves query performance by 30%, and reduces data reliability incidents by 40%.

Subtasks:

• Redesign analytical data models for AI/ML consumption
• Implement governance using Unity Catalog or Snowflake controls
• Optimize distributed compute performance
• Establish monitoring and quality validation checkpoints

Enable AI-Ready Data Modeling & Governance

Within 6 months, establish semantic models and metadata standards that enable business-facing AI querying with 95% data trust rating from stakeholders.

Subtasks:

• Standardize schema design for ML and GenAI workloads
• Align business definitions with governed datasets
• Implement lineage and access controls
• Reduce duplicate or conflicting metric definitions

Build and Deploy Production-Grade ML Pipelines

Within 9–12 months, implement reusable ML lifecycle pipelines (experimentation training evaluation deployment) that reduce time-to-production for ML models by 50%.

Subtasks:

• Standardize MLflow/Feature Store workflows
• Implement CI/CD for ML
• Improve model observability and drift monitoring
• Establish model documentation standards

Implement Natural Language AI Analytics (Databricks Genie Enablement)

Within 6 months, deploy and optimize Databricks Genie (or equivalent AI query interface) enabling business users to generate accurate plain-language insights with 80% adoption across target user groups.

Subtasks:

• Translate business questions into semantic AI-ready datasets
• Improve response accuracy through model + metadata tuning
• Partner with Product on use-case prioritization
• Track and improve AI query accuracy and user engagement

Democratize AI Across Business Teams

Within 12 months, embed AI-driven analytics into at least 3 core business workflows, demonstrating measurable business impact (e.g., cost reduction, revenue lift, or decision cycle time improvement).

Subtasks:

• Identify high-value AI use cases
• Collaborate cross-functionally
• Deliver production-ready AI solutions
• Document business ROI outcomes

Establish Enterprise AI Platform Standards

Within 12 months, define and institutionalize architectural standards, best practices, and governance frameworks adopted across Engineering and Analytics teams.

Subtasks:

• Publish architecture reference patterns
• Lead design reviews
• Mentor engineers
• Influence long-term AI strategy

Success Metrics Summary

• 30%+ performance improvement in analytics workloads
• 40%+ reduction in data quality incidents
• 50% reduction in ML deployment cycle time
• 80% Genie adoption in target group
• 3+ AI use cases with measurable ROI
• • 95% stakeholder trust in AI-generated insights

Qualifications:

Growth & Career Move

This is a high-impact platform leadership role enabling enterprise AI transformation. The individual will shape architecture standards, influence executive AI strategy, and lead the shift from traditional BI to AI-powered decision intelligence.

Required Qualifications

• 10+ years of experience in Data Analytics, Data Engineering, ML Engineering, or AI Engineering
• Strong hands-on experience with Databricks or Snowflake in production environments
• Expertise in SQL, Python, and distributed data processing (Spark preferred)
• Strong understanding of data modeling for analytics and AI
• Experience building and deploying ML models in real-world systems
• Familiarity with LLMs, GenAI concepts, and AI-assisted analytics
• Experience with ML lifecycle tools (MLflow, Feature Stores, CI/CD for ML)

• Direct experience with Databricks Genie or AI-powered BI tools
• Experience with Unity Catalog, Delta Live Tables, or Snowflake governance features
• Exposure to Azure, AWS, or GCP cloud platforms
• Experience working in regulated or enterprise SaaS environments
• Ability to explain complex technical concepts to non-technical stakeholders

• Business users can ask questions in plain English and get trusted, accurate insights
• Data models are AI-ready, scalable, and well-governed
• ML models move smoothly from experimentation to production
• Databricks Genie adoption grows with measurable business impact
• AI is embedded into analytics not bolted on

• Work on real AI/ML problems at enterprise scale
• Influence the evolution of a modern data + AI platform
• Partner with senior leaders shaping the company’s AI-first future
• Build systems that turn data into decisions not dashboards

Please note that this is a hybrid role that involves a combination of onsite work from our corporate office as well as work from home. While we strive to accommodate flexible working arrangements when sensible, there will be times when onsite work is required. This could include scheduled office days, team meetings, client meetings, or special events.