Lead Data Engineer

Canada Life
London, United Kingdom
Today
£70,000 – £100,000 pa

Salary

£70,000 – £100,000 pa

Job Type
Permanent
Work Pattern
Full-time
Work Location
Hybrid
Seniority
Lead
Education
Degree
Posted
30 Apr 2026 (Today)

Location: London, Watford or Bristol (Hybrid working options available)

The Lead Data Engineer will provide hands-on technical leadership in Azure cloud and Databricks-based solutions within our Enterprise Data Platform. The role requires strong expertise in Azure cloud services, Databricks, data engineering, and DevOps, leading a cross-functional team to build, deploy, and support high-performance data-driven solutions.

The role involves:

  • Interpreting Outcomes and user stories and translating them into technical solutions.
  • Creating innovative solution designs for domain and enterprise data products.
  • Overseeing Data Analysts to support detailed data discovery.
  • Overseeing data modelling for Finance and Enterprise data products.
  • Leading Product Increment planning to break down solutions into Features and Epics for incremental delivery.
  • Designing and implementing scalable data solutions on Azure and Databricks within the assigned domain.
  • Ensuring appropriate engineering standards are applied to maintain data quality, performance and reliability.

Duties/Responsibilities

  • Work with Product Owners and Business Analysts to understand Outcomes, refine user stories.
  • Lead solution design for Finance and Enterprise data products, ensuring alignment with enterprise patterns and guardrails.
  • Direct and collaborate with Data Analysts on detailed data discovery, source understanding and requirements refinement.
  • Oversee logical and physical data modelling for Finance and Enterprise data products, working closely with architecture where required.
  • Implement and maintain data pipelines and ETL workflows in Databricks (PySpark, Delta Lake).
  • Contribute to CI/CD pipelines for data applications using Azure DevOps and infrastructure-as-code (Terraform) in line with established patterns.
  • Apply security, access control and compliance standards for Azure and Databricks in collaboration with platform and security teams.
  • Support monitoring, logging and basic cost optimisation for the team’s data products.
  • Support the development of DevOps practices within the team, including reducing technical debt and improving automation over time.

Skills, Knowledge and Experience

Lead Data Engineers are expected to have strong capability in at least three of the following areas of engineering practice.

Core skills

  • Automation including testing of data pipelines and data products.
  • Strong teamwork, communication and problem-solving skills to collaborate effectively with cross-functional teams.
  • Awareness of security principles and best practices to ensure secure data solutions.
  • Commitment to continuous learning and staying current with Azure, Databricks and data engineering trends.
  • Strong experience working within an agile development methodology, ideally Scaled Agile (SAFe or similar).
  • Excellent time and self-management through effective planning and prioritisation of tasks.
  • Proven and demonstrable data engineering capability.
  • Ability to influence within the team and communicate clearly with technical and non-technical stakeholders.

Data Engineer (New Technology / Microsoft)

  • Strong experience with Databricks (Spark, PySpark, Delta Lake, and Unity Catalog advantageous).
  • Proficiency in Azure data services (Azure Data Factory, Data Lake, Azure Functions advantageous).
  • Experience contributing to CI/CD pipelines (Azure DevOps, GitHub Actions, Terraform).
  • Scripting and programming skills (Python advantageous).
  • Good understanding of DevOps and automation concepts (e.g. YAML pipelines, IaC).
  • Solid understanding of cloud security, compliance and governance principles.
  • Experience working with Databricks and Azure in a product or Scaled Agile delivery environment.

Qualifications

  • Degree level IT or technical/scientific subject (or equivalent experience).
  • Microsoft Azure Data Engineer or Solutions Architect certification (desirable).
  • Databricks Certified Data Engineer or Machine Learning Associate (desirable).
  • Experience with streaming solutions (Kafka, Event Hubs, Spark Streaming) (desirable).
  • Knowledge of machine learning and AI on Databricks (desirable).

Related Jobs

View all jobs

Senior Data Scientist

Adria Solutions Manchester, United Kingdom
Permanent

Data Scientist / AI Engineer

Searchability NS&D Cheltenham, United Kingdom
£45,000 – £95,000 pa On-site Clearance Required

Data Engineer, Strategic Account Services

Amazon London, United Kingdom
Permanent

Senior Data Engineer - Microsoft Fabric

Harvey Nash London, United Kingdom
£80,000 – £90,000 pa Hybrid

Lead Data Scientist

Technify Talent Limited United Kingdom
£80,000 – £90,000 pa Remote

Lead Data Scientist

Exposed Solutions London, United Kingdom
£60,000 – £80,000 pa Remote

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Where to Advertise Machine Learning Jobs in the UK (2026 Guide)

Advertising machine learning jobs in the UK requires a different approach to most technical hiring. The candidate pool is small, highly specialised and in demand across AI labs, financial services, healthcare, autonomous systems and consumer technology simultaneously. Machine learning engineers and researchers move between roles through professional networks, conference communities and specialist platforms — not general job boards where ML roles compete with unrelated software engineering positions for the same audience. This guide, published by MachineLearningJobs.co.uk, covers where to advertise machine learning roles in the UK in 2026, how the main platforms compare, what employers should expect to pay, and what the data says about hiring across different role types.

New Machine Learning Employers to Watch in 2026: UK and Global Companies Driving ML Innovation

Machine learning (ML) has transitioned from a specialised field into a core business capability. In 2026, organisations across healthcare, finance, robotics, autonomous systems, natural language processing, and analytics are expanding their machine learning teams to build scalable intelligent products and services. For professionals exploring opportunities on www.MachineLearningJobs.co.uk , understanding the companies that are scaling, winning investment, or securing high‑impact contracts is crucial. This article highlights the new and high‑growth machine learning employers to watch in 2026, focusing on UK innovators, international firms with significant UK presence, and global platforms investing in machine learning talent locally.

How Many Machine Learning Tools Do You Need to Know to Get a Machine Learning Job?

Machine learning is one of the most exciting and rapidly growing areas of tech. But for job seekers it can also feel like a maze of tools, frameworks and platforms. One job advert wants TensorFlow and Keras. Another mentions PyTorch, scikit-learn and Spark. A third lists Mlflow, Docker, Kubernetes and more. With so many names out there, it’s easy to fall into the trap of thinking you must learn everything just to be competitive. Here’s the honest truth most machine learning hiring managers won’t say out loud: 👉 They don’t hire you because you know every tool. They hire you because you can solve real problems with the tools you know. Tools are important — no doubt — but context, judgement and outcomes matter far more. So how many machine learning tools do you actually need to know to get a job? For most job seekers, the real number is far smaller than you think — and more logically grouped. This guide breaks down exactly what employers expect, which tools are core, which are role-specific, and how to structure your learning for real career results.