Data Engineering Consultant

Company Description

Job Description

Data Engineering Consultant designs, implements, and optimizes scalable data pipelines and architectures. This role bridges raw data and actionable insights, ensuring robustness, performance, and data governance. Collaboration with analysts and scientists is central to delivering high-quality solutions aligned with business objectives.

Key Responsibilities:

1. Data Pipeline Development

• Architect, implement and maintain real-time and batch data pipelines to handle large datasets efficiently.
• Employ frameworks such as Apache Spark, Databricks, Snowflake or Airflow to automate ingestion, transformation, and delivery.
• Design and implement fault-tolerant, self-healing data pipelines to ensure continuous data flow.
• Optimize data pipeline performance through parallel processing and distributed computing techniques.

• Data Integration & Transformation

• Work with Data Analysts to understand source-to-target mappings and quality requirements.
• Build ETL/ELT workflows, validation checks, and cleaning steps for data reliability.
• Develop and maintain data dictionaries and metadata repositories for improved data understanding.
• Implement data quality monitoring tools and processes to ensure data integrity throughout the pipeline.

• Automation & Process Optimization

• Automate data reconciliation, metadata management, and error-handling procedures.
• Continuously refine pipeline performance, scalability, and cost-efficiency.
• Implement monitoring and alerting systems for early detection of pipeline issues.
• Develop and maintain documentation for all automated processes and workflows.

• Collaboration & Leadership

• Coordinate with Data Scientists, Data Architects, and Analysts to ensure alignment with business goals.
• Mentor junior engineers and enforce best practices (version control, CI/CD for data pipelines).
• Participate in technical presales activities and client engagement initiatives.
• Lead cross-functional team meetings to discuss data engineering challenges and solutions.
• Contribute to the development of data engineering standards and best practices within the organization.

• Governance & Compliance

• Apply robust security measures (RBAC, encryption) and ensure regulatory compliance (GDPR).
• Document data lineage and recommend improvements for data ownership and stewardship.
• Implement data masking and anonymization techniques for sensitive information.
• Collaborate with legal and compliance teams to ensure adherence to data protection regulations.

• Performance Tuning & Optimization

• Conduct regular performance audits of data pipelines and storage systems.
• Implement caching strategies and query optimization techniques to improve data access speeds.
• Utilize partitioning and indexing strategies to enhance query performance on large datasets.
• Optimize resource allocation and utilization in cloud environments to manage costs effectively.

• Emerging Technologies & Innovation

• Stay abreast of emerging data engineering technologies and methodologies.
• Evaluate and recommend new tools and frameworks that can enhance data engineering capabilities.
• Conduct proof-of-concept projects to test innovative data engineering solutions.
• Contribute to the company's thought leadership through blog posts, whitepapers, or conference presentations on data engineering topics.

Qualifications

Qualifications:

Required:

• Bachelor's degree in Computer Science, Information Technology, or a related field; Master's degree preferred
• 5-10 years of experience in data engineering, with a strong focus on big data technologies
• Expert-level proficiency in Python/PySpark
• Extensive experience with cloud platforms, particularly AWS-or- Azure-or- GCP
• Advanced knowledge of data warehousing solutions such as Databricks -or- Snowflake-or- BigQuery -or- Redshift
• Proven track record in designing, implementing, and optimizing large-scale data pipelines
• Strong understanding of data modeling, ETL/ELT processes, and data governance principles
• Proficiency in SQL and NoSQL databases
• Familiarity with data visualization tools and techniques

Preferred:

• Relevant certifications such as AWS Certified Data Analytics - Specialty, Azure Data Engineer Associate, or Google Cloud Professional Data Engineer
• Experience with real-time data streaming technologies (e.g., Kafka, Apache Flink)
• Knowledge of machine learning concepts and their application in data engineering
• Familiarity with data lake architectures and associated technologies
• Experience with CI/CD practices for data pipelines
• Understanding of data security and compliance requirements (e.g., GDPR)
• Strong problem-solving skills and ability to work independently
• Excellent communication skills, both written and verbal
• Experience mentoring junior engineers and collaborating with cross-functional teams

Additional Information

At Endava, we’re committed to creating an open, inclusive, and respectful environment where everyone feels safe, valued, and empowered to be their best. We welcome applications from people of all backgrounds, experiences, and perspectives—because we know that inclusive teams help us deliver smarter, more innovative solutions for our customers. Hiring decisions are based on merit, skills, qualifications, and potential. If you need adjustments or support during the recruitment process, please let us know.