National AI Awards 2025Discover AI's trailblazers! Join us to celebrate innovation and nominate industry leaders.

Nominate & Attend

Senior Data Engineer - (Azure/Databricks)

Axis Capital
London
1 month ago
Create job alert

Responsibilities

:
Design and implement data solutions using Azure services, including Azure Databricks, ADF, and Data Lake Storage. Develop and maintain ETL/ELT pipelines to process structured and unstructured data from multiple sources. - Automate loads using Databricks workflows and Jobs Develop, test and build CI/CD pipelines using Azure DevOps to automate deployment and monitoring of data solutions to all environments. Provide knowledge sharing to data operations teams on release management and maintenance. Manage platform administration, ensuring optimal performance, availability, and scalability of Azure data services. Implement end-to-end data pipelines, ensuring data quality, data integrity and data security. Troubleshoot and resolve data pipeline issues while ensuring data integrity and quality. Implement and enforce data security best practices, including role-based access control (RBAC), encryption, andpliance with industry standards. Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality data solutions. Monitor and optimize Databricks performance, including cost management guidance and cluster tuning. Stay up to date with Azure cloud innovations and rmend improvements to existing architectures. Assist data analysts with technical input.
You may also be required to take on additional duties, responsibilities and activities appropriate to the nature of this role.

About You:

We encourage you to bring your own experience and expertise to the table, so while there are some qualifications and experiences, we need you to have, we are open to discussing how your individual knowledge might lend itself to fulfilling this role and help us achieve our goals.

Required Skills & Experience:
5 plus years Azure & Data Engineering Expertise: Proven experience in designing and managing large-scale data solutions on Microsoft Azure. Unity Catalog Mastery: In-depth knowledge of setting up, configuring, and utilizing Unity Catalog for robust dataernance, access control, and metadata management in a Databricks environment. Databricks Proficiency: Demonstrated ability to optimize and tune Databricks notebooks and workflows to maximize performance and efficiency. Experience with performance troubleshooting and best practices for scalable data processing is essential. Additional Requirements: Strong problem-solving skills, ability to work in agile environments, and effective collaboration with cross-functional teams. Experience with implementing a Data Lakehouse solution with Azure Databricks, data modeling, warehousing, and real-time streaming. Knowledge of developing and processing full and incremental loads. Experience of automated loads using Databricks workflows and Jobs Expertise in Azure Databricks, including Delta Lake, Spark optimizations, and MLflow. Strong experience with Azure Data Factory (ADF) for data integration and orchestration. Hands-on experience with Azure DevOps, including pipelines, repos, and infrastructure as code (IaC). Solid understanding of platform administration, including monitoring, logging, and cost management. Knowledge of data security,pliance, andernance in Azure, including Azure Active Directory (AAD), RBAC, and encryption. Experience working with big data technologies (Spark, Python, Scala, SQL). Strong problem-solving and troubleshooting skills. Excellentmunication skills with the ability to collaborate with cross-functional teams to understand requirements, data solutions, data models and mapping documents.Preferred Qualifications:Azure certifications (, Azure Data Engineer Associate, Azure Solutions Architect). Experience with Terraform, ARM templates, or Bicep for infrastructure automation. Experience integrating Azure Data Services with Power BI and AI/ML workflows.Role Factors:

The position is full-time with remote work options, requiring in-office presence three days per week

What we offer:

You will be eligible for aprehensive andpetitive benefits package which includes medical plans for you and your family, health and wellness programs, retirement plans, tuition reimbursement, paid annual leave, and much more. Job ID REQ05851

Related Jobs

View all jobs

Senior Data Engineer

Senior Data Engineer

Senior Data Engineer

Senior Data Engineer

Senior Data Engineer - Azure - Leeds

Senior Data Engineer - Snowflake - £100,000 - London - Hybrid

National AI Awards 2025

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

How to Present Machine Learning Solutions to Non-Technical Audiences: A Public Speaking Guide for Job Seekers

Machine learning is driving change across nearly every industry—from retail and finance to health and logistics. But while the technology continues to evolve rapidly, the ability to communicate it clearly has become just as important as building the models themselves. Whether you're applying for a junior ML engineer role, a research position, or a client-facing AI consultant job, UK employers increasingly expect candidates to explain complex machine learning solutions to non-technical audiences. In this guide, you’ll learn how to confidently present your work, structure your message, use simple visuals, and explain the real-world value of machine learning in a way that makes sense to people without a background in data science.

Machine Learning Jobs UK 2025: 50 Companies Hiring Now

Bookmark this page—we refresh the Hotlist every quarter so you always know who’s really scaling their ML teams. The UK’s National AI Strategy, a £2 billion GenAI accelerator fund and a record flow of private capital have kicked ML hiring into overdrive for 2025. Whether you build production‑grade LLM services or optimise on‑device models for edge hardware, employers need your skills now. Below you’ll find 50 organisations that advertised UK‑based machine‑learning vacancies or announced head‑count growth during the past eight weeks. They’re grouped into five quick‑scan categories so you can jump straight to the type of employer—and mission—that excites you. For each company we list: Main UK hub Example live or recent vacancy Why it’s worth a look (stack, impact, culture) Search any employer on MachineLearningJobs.co.uk to see real‑time adverts, or set a free alert so fresh openings drop straight in your inbox.

Return-to-Work Pathways: Relaunch Your Machine Learning Career with Returnships, Flexible & Hybrid Roles

Returning to work after an extended break can feel like starting from scratch—especially in a specialist field like machine learning. Whether you paused your career for parenting, caring responsibilities or another life chapter, the UK’s machine learning sector now offers a variety of return-to-work pathways. From structured returnships to flexible and hybrid roles, these programmes recognise the transferable skills and resilience you’ve developed, pairing you with mentorship, upskilling and supportive networks to ease your transition back. In this guide, you’ll discover how to: Understand the current demand for machine learning talent in the UK Leverage your organisational, communication and analytical skills in ML contexts Overcome common re-entry challenges with practical solutions Refresh your technical knowledge through targeted learning Access returnship and re-entry programmes tailored to machine learning Find roles that fit around family commitments—whether flexible, hybrid or full-time Balance your career relaunch with caring responsibilities Master applications, interviews and networking specific to ML Learn from inspiring returner success stories Get answers to common questions in our FAQ section Whether you aim to return as an ML engineer, research scientist, MLOps specialist or data scientist with an ML focus, this article will map out the steps and resources you need to reignite your machine learning career.