Be at the heart of actionFly remote-controlled drones into enemy territory to gather vital information.

Apply Now

Senior Data Engineer - Azure

London
4 days ago
Create job alert

Senior Databricks Data Engineer

2-3 days on-site - central London

up to £75k + 20% Bonus + Excellent Benefits

Our client is a leading global Retail/FMCG brand undergoing an exciting period of rapid growth and transformation. With significant investment in data and technology, they are building a world-class data platform to power decision-making across every area of the business - from supply chain and logistics to marketing, customer sales and in-store operations.

We are seeking an experienced Senior Data Engineer with deep expertise in Databricks and Medallion Architecture (Bronze-Silver-Gold) to design, build, and optimize our next-generation data platform. This role will be pivotal in developing scalable data pipelines, enabling advanced analytics, and driving data quality and governance across the organization.

You'll work closely with data scientists, analysts, and business stakeholders to transform raw data into trusted, actionable insights that power critical business decisions.

Key Responsibilities

Design and implement scalable data pipelines and ETL/ELT workflows in Databricks using PySpark, SQL, and Delta Lake.
Architect and manage the Medallion (Bronze, Silver, Gold) data architecture for optimal data organization, transformation, and consumption.
Develop and maintain data models, schemas, and data quality frameworks across multiple domains.
Integrate data from a variety of structured and unstructured sources, including APIs, relational databases, and streaming data.
Optimize performance, scalability, and cost efficiency of Databricks clusters and workflows.
Collaborate with cross-functional teams to support analytics, machine learning, and business intelligence use cases.
Implement data governance, lineage, and observability best practices using tools such as Unity Catalog, DataHub, or Collibra.
Mentor junior engineers, fostering best practices in data engineering, testing, and DevOps for data (DataOps).
Stay current with emerging technologies in cloud data platforms, Lakehouse architecture, and data engineering frameworks.Required Qualifications

6+ years of experience in data engineering
3+ years of hands-on experience with Databricks, Delta Lake, and Spark (PySpark preferred).
Proven track record implementing Medallion Architecture (Bronze, Silver, Gold layers) in production environments.
Strong knowledge of data modeling, ETL/ELT design, and data lakehouse concepts.
Proficiency in Python, SQL, and Spark optimization techniques.
Experience working with cloud data platforms such as Azure Data Lake, AWS S3, or GCP BigQuery.
Strong understanding of data quality frameworks, testing, and CI/CD pipelines for data workflows.
Excellent communication skills and ability to collaborate across teams.Preferred Qualifications

Experience with Databricks Unity Catalog and Delta Live Tables.
Familiarity with streaming frameworks (Structured Streaming, Kafka, etc.).
Background in data observability and metadata management.
Exposure to machine learning pipelines or MLflow within Databricks.
Knowledge of infrastructure as code (IaC) using Terraform or similar tools. To apply for this role please email across your CV ASAP

Related Jobs

View all jobs

Senior Data Engineer

Senior Data Engineer

Senior Data Engineer

Senior Data Engineer

Senior Data Engineer

Senior Data Engineer

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Machine Learning Recruitment Trends 2025 (UK): What Job Seekers Need To Know About Today’s Hiring Process

Summary: UK machine learning hiring has shifted from title‑led CV screens to capability‑driven assessments that emphasise shipped ML/LLM features, robust evaluation, observability, safety/governance, cost control and measurable business impact. This guide explains what’s changed, what to expect in interviews & how to prepare—especially for ML engineers, applied scientists, LLM application engineers, ML platform/MLOps engineers and AI product managers. Who this is for: ML engineers, applied ML/LLM engineers, LLM/retrieval engineers, ML platform/MLOps/SRE, data scientists transitioning to production ML, AI product managers & tech‑lead candidates targeting roles in the UK.

Why Machine Learning Careers in the UK Are Becoming More Multidisciplinary

Machine learning (ML) has moved from research labs into mainstream UK businesses. From healthcare diagnostics to fraud detection, autonomous vehicles to recommendation engines, ML underpins critical services and consumer experiences. But the skillset required of today’s machine learning professionals is no longer purely technical. Employers increasingly seek multidisciplinary expertise: not only coding, algorithms & statistics, but also knowledge of law, ethics, psychology, linguistics & design. This article explores why UK machine learning careers are becoming more multidisciplinary, how these fields intersect with ML roles, and what both job-seekers & employers need to understand to succeed in a rapidly changing landscape.

Machine Learning Team Structures Explained: Who Does What in a Modern Machine Learning Department

Machine learning is now central to many advanced data-driven products and services across the UK. Whether you work in finance, healthcare, retail, autonomous vehicles, recommendation systems, robotics, or consumer applications, there’s a need for dedicated machine learning teams that can deliver models into production, maintain them, keep them secure, efficient, fair, and aligned with business objectives. If you’re hiring for or applying to ML roles via MachineLearningJobs.co.uk, this article will help you understand what roles are typically present in a mature machine learning department, how they collaborate through project lifecycles, what skills and qualifications UK employers look for, what the career paths and salaries are, current trends and challenges, and how to build an effective ML team.