Data Engineer

Key Responsibilities

Data Engineering & Pipelines

Build and enhance data pipelines and automation workflows on Azure for large-scale data ingestion and processing
Maintain and optimise existing toolkits, databases, dashboards, and API-based solutions
Develop end-to-end automated data solutions across commodities (Gas/LNG, Power, Weather)
Design and implement web scraping pipelines (e.g. US LNG data)

Analytics & Real-Time Systems

Support real-time and batch analytics use cases in a dynamic trading environment
Enable and support analytics dashboards and GUI tools with reliable backend infrastructure
Develop real-time flow tracking solutions at critical market points

Data Modelling & Architecture

Design and manage Snowflake data models, ensuring performance and scalability
Integrate external data sources including ENTSOE, EPEX, KPLER, MetDesk, WoodMac, and others
Collaborate with analytics teams to structure data for models and advanced analytics use cases
Build and maintain data platforms supporting ML models such as demand forecasting

Key Deliverables

Required Skills & Experience

Technical

Strong proficiency in Python for data engineering and pipeline development
Hands-on experience with Azure data services (Data Factory, Databricks, Blob Storage, etc.)
Experience designing and optimising Snowflake data models
Proven track record building and maintaining data pipelines at scale
Experience integrating third-party market data APIs and external data feeds
Familiarity with SQL and data warehousing concepts

Domain

Experience or strong interest in energy, commodities, or financial markets data
Understanding of real-time and batch data processing patterns in a trading context
Familiarity with market data providers such as ENTSOE, EPEX, KPLER, or similar

Nice to Have

Experience with web scraping frameworks (e.g. Scrapy, BeautifulSoup, Playwright)
Knowledge of LNG, Gas, or Power market fundamentals
Experience supporting ML model pipelines or demand forecasting workflows
Exposure to GUI/dashboard tools such as Grafana, Power BI, or Streamlit
Knowledge of ship tracking data or AIS feeds

Similar jobs