Data Engineer

Description:

About Heritage Auctions

Start your career with one of the nation’s fastest-growing auction houses. Heritage Auctions is a global auction house driven by curiosity, innovation, and the stories behind extraordinary objects. For nearly 50 years, Heritage has connected collectors and consignors through a dynamic marketplace that celebrates history, culture, and craftsmanship. Headquartered in Dallas, Texas, with offices in New York, Chicago, Beverly Hills, London, Hong Kong, and other major cities, we support a worldwide network of experts and enthusiasts across more than 50 auction categories. From Sports and Entertainment memorabilia to Luxury Handbags, Fine Art, Jewelry, Coins, Wine, and rare historical artifacts, every item tells a story. We are dedicated to bringing those stories to life through deep research, cutting-edge technology, and thoughtful presentation. Known for our expertise and innovation, we offer a dynamic environment where passion for history, culture, and storytelling brings collectors together to discover, learn, and collaborate.

SUMMARY: Responsible for the development of all phases in SQL, T-SQL, relational database programming/design, APIs, SSIS, and SSRS reporting.

LOCATION: This is a Hybrid position and requires one day in the office located South of Dallas/Fort Worth International Airport, NW corner of W. Airport Freeway (Hwy. 183) and Valley View Lane

BENEFITS:

• Medical, Dental, Vision coverage
• Paid time off
• 401k savings plans
• Onsite personal trainer and gym

COMPENSATION: Commensurate based on experience

ESSENTIAL DUTIES & RESPONSIBILITIES:

SQL Development:

• Write and optimize stored procedures, functions, table alterations, and indexes in SQL Server 2019 and 2022.
• Design and implement efficient queries using inner/outer joins and set-based logic.
• Performance tuning of stored procedures, set-based code, and indexing strategies.
• Leverage execution plans, statistics, and plan diagrams for troubleshooting and optimization

Data Integration & Pipeline Development:

• Develop and manage data pipelines using SSIS and Azure Data Factory.
• Consume and process data from APIs, handling JSON data efficiently.
• Configure and promote objects through development to production environments using version control and coding standards.
• Microsoft Fabric: Design and manage data solutions within the Fabric ecosystem, including OneLake, Data Factory (Gen2 Pipelines), and Lakehouses.

ElasticSearch Expertise:

• Design and manage indexes and lifecycle management for optimal search performance.
• Develop ElasticSearch queries, including keyword, text, and vector search, and manage document ingestion.
• Leverage ML inference models within ElasticSearch pipelines to enhance retrieval and ranking mechanisms.

Infrastructure as Code (IaC) & DevOps:

• Terraform: Utilize Terraform to provision and manage Azure resources (SQL databases, Fabric capacities, and ElasticSearch clusters).
• Configure and promote objects through development to production environments using version control (Git) and automated deployment standards.

Machine Learning Integration:

• Build and integrate machine learning models into search pipelines using Python.
• Experience with libraries such as TensorFlow, PyTorch, or scikit-learn.
• Familiarity with natural language processing (NLP) and large language models (LLMs).

Advanced Search and Retrieval Systems:

• Integrate large language models (LLMs) for retrieval-augmented generation (RAG) solutions.
• Design and manage scalable, high-availability systems for efficient query processing and document retrieval.
• Utilize knowledge graphs and dynamic query generation to enhance search functionality.

Agentic AI Systems:

• Build multi-agent systems for advanced search and recommendation engines.
• Use frameworks like LangChain or LangGraph for orchestrating agent reasoning and decision-making processes.
• Optimize multi-turn conversations, state management, and task collaboration across distributed agents.

Collaboration and Ownership:

• Own the design and improvement of database schema and architecture.
• Translate business requirements into scalable system designs.
• Interact with cross-functional teams and act as a subject matter expert for data engineering and search technologies.

Requirements:

EDUCATION & EXPERIENCE:

• Bachelor’s degree in Computer Science required Master’s degree preferred
• Minimum 5+ years of experience as an MS SQL programmer in a fast-paced, dynamic environment using a structured software development methodology
• Experience designing and implementing large-scale database solutions that are reliable, scalable, and perform at high levels
• Working knowledge of security and privacy issues, SQL 2019, and SQL 2022

WORK ENVIRONMENT: Individuals must be team-quality players in a fast-paced environment. The candidate must work in an environment with 50% development and 50% production support.

At Heritage Auctions, we celebrate diversity and are committed to fostering an inclusive environment. We welcome and encourage applications from women, minorities, veterans, mature workers, and persons with disabilities. EOE