Azure Data Engineer

Data Engineer (Databricks)

Azure Databricks Environment

Austin, TX - Hybrid

General Information

Description:

The Databricks Data and AI Agent Automation Engineer is responsible for designing, developing, and deploying intelligent agents that automate data engineering development (dev) and quality assurance (QA) tasks within the Azure Databricks environment. This role bridges software engineering and data platform operations, leveraging AI/ML capabilities, prompt engineering, and low-code/no-code automation frameworks to accelerate platform development cycles, reduce manual toil, and improve code quality. The engineer will work closely with the Data Platform team to build, test, and operationalize agents that enhance developer productivity, streamline CI/CD pipelines, and automate governance and compliance checks.

Responsibilities

· Design and implement intelligent agents (LLM-powered, rule-based, or hybrid) that automate data engineering workflows in Azure Databricks

· Develop agents for automated code generation, refactoring, and optimization of PySpark and SQL workloads

· Build agents that automate data quality validation, testing, and reconciliation tasks in dev and QA environments

· Create agents for metadata management, lineage tracking, and governance automation within Databricks

· Develop autonomous CI/CD pipeline tasks, including automated test case generation, deployment validation, and rollback logic

· Implement agents that perform intelligent monitoring, anomaly detection, and root-cause analysis for data pipelines

· Build prompt engineering frameworks and agent orchestration patterns tailored to data engineering use cases

· Collaborate with data engineers and architects to identify high-impact automation opportunities and prioritize agent development roadmap

· Establish best practices for agent development, testing, deployment, and observability within Azure Databricks

· Document agent workflows, decision logic, and operational runbooks for team knowledge sharing

· Evaluate emerging AI/agent frameworks and technologies; recommend adoption where applicable

Job Requirements

Education

· Bachelor's degree in Computer Science, Software Engineering, Data Science, or related field (Master's preferred)

Experience

· 5+ years of data engineering or software engineering experience, with hands-on work in cloud data platforms (Databricks, Snowflake, Azure)

· Proven experience developing automation solutions, agents, or AI-powered tools (e.g., using LLMs, prompt engineering, or orchestration frameworks like Databricks Agent Framework (or LangChain, AutoGen)

· Strong proficiency in Python and SQL with focus on production-grade code quality

· Demonstrated experience with Azure Databricks (notebooks, jobs, workflows, DLT) or equivalent modern data platforms

· Hands-on experience with CI/CD/CT tools (GitHub Actions, Azure DevOps Pipelines) and automation frameworks

· Experience building data quality frameworks, testing patterns, or QA automation

· Prior work in regulated or risk-sensitive environments preferred

Skills & Competencies

· Expert-level Python programming with strong software engineering practices (testing, documentation, version control)

· Strong SQL and PySpark optimization skills

· Hands-on experience with LLM APIs, prompt engineering, and AI agent frameworks (e.g., Databricks Agent Bricks, LangChain, OpenAI, Azure OpenAI)

· Proficiency in Azure Databricks, including Delta Lake, workflows, and Databricks Asset Bundles

· Experience designing and implementing automation workflows using orchestration tools or serverless platforms

· Strong problem-solving and systems thinking skills; ability to decompose complex workflows into automatable components

· Excellent communication and documentation skills; ability to articulate technical concepts and agent design rationale

· Self-directed learner with curiosity about emerging AI/automation technologies

· Ability to collaborate effectively with cross-functional teams (data engineers, architects, QA, operations)

Pay: $65.00 - $70.00 per hour

Work Location: Hybrid remote in Austin, TX 78701