Data Engineer - Databricks / AI

JACOBS
Manchester
1 day ago
Create job alert
Overview

At Jacobs, we're challenging today to reinvent tomorrow by solving the world's most critical problems for thriving cities, resilient environments, mission-critical outcomes, operational advancement, scientific discovery and cutting-edge manufacturing, turning abstract ideas into realities that transform the world for good.

Your impact

Enjoy designing elegant data systems, shipping production-grade code, and seeing your work make a measurable difference, then this role is for you!

Our team build data platforms and AI solutions that power critical infrastructure, transform operations, and move entire industries.

In this role you will work with a broad set of clients on high-scale problems, with the backing of a global organisation, investing heavily in Azure, Databricks, and applied AI. Working primarily on Azure + Databricks (Spark, Delta Lake, Unity Catalog) - Ship modern ELT/ETL, streaming and batch data products, and ML/AI pipelines, operating at serious scale across water, transport, energy, and more. Join a collaborative, engineering-led culture with real investment in platforms & tooling.

Stack
  • Azure: ADLS Gen2, Event Hubs, ADF/Azure Data Factory or Synapse pipelines, Functions, Key Vault, VNets
  • Databricks: Spark, Delta Lake, Unity Catalog, Workflows, MLflow (experiments, model registry)
  • Languages: Python (PySpark), SQL (Delta SQL), optional Scala
  • Engineering: Git, pull requests, code review, unit/integration tests, dbx, notebooks as code
  • Platform & Ops: Azure DevOps/GitHub, CI/CD, Terraform or Bicep, monitoring/alerting
Responsibilities
  • Design & build robust data platforms and pipelines on Azure and Databricks (batch + streaming) using Python/SQL, Spark, Delta Lake, and Data Lakehouse patterns.
  • Develop AI-enabling foundations: feature stores, ML-ready datasets, and automated model-serving pathways (MLflow, model registries, CI/CD).
  • Own quality & reliability: testing (dbx/pytest), observability (metrics, logging, lineage), and cost/performance optimisation.
  • Harden for enterprise: security-by-design, access patterns with Unity Catalog, data governance, and reproducible environments.
  • Automate the boring stuff: IaC (Terraform/Bicep), CI/CD (Azure DevOps/GitHub Actions), and templated project scaffolding.
  • Partner with clients: translate business problems into technical plans, run workshops, and present trade-offs with clarity.
  • Ship value continuously: iterate, review, and release frequently; measure outcomes, not just outputs.
Qualifications
  • Utilising SQL and Python for building reliable data pipelines.
  • Hands-on with Spark (preferably Databricks) and modern data modelling (e.g., Kimball/Inmon/Data Vault, lakehouse).
  • Experience running on a cloud data platform (ideally Azure). Sound software delivery practices: Git, CI/CD, testing, Agile ways of working.
  • Streaming/event-driven designs (Event Hubs, Kafka, Structured Streaming).
  • MPP/Data Warehouses (Synapse, Snowflake, Redshift) and NoSQL (Cosmos DB).
  • ML enablement: feature engineering at scale, MLflow, basic model lifecycle know-how.
  • Infrastructure-as-code (Terraform/Bicep) and platform hardening.

Don’t meet every single bullet? We’d still love to hear from you. We hire for mindset and potential as much as current skills!

Culture & inclusion

Joining Jacobs not only connects you locally but globally. Our values stand on a foundation of safety, integrity, inclusion and belonging. We put people at the heart of our business, and we truly believe that by supporting one another through our culture of caring, we all succeed. We value positive mental health and a sense of belonging for all employees.

With safety and flexibility always top of mind, we’ve gone beyond traditional ways of working so you have the support, means and space to maximize your potential. You’ll uncover flexible working arrangements, benefits, and opportunities, from well-being benefits to our global giving and volunteering program, to exploring new and inventive ways to help our clients make the world a better place. No matter what drives you, you’ll discover how you can cultivate, nurture, and achieve your goals – all at a single global company. Find out more about life at Jacobs.

We aim to embed inclusion and belonging in everything we do. We know that if we are inclusive, we’re more connected and creative. We are committed to building vibrant communities within Jacobs, including through our Jacobs Employee Networks, Communities of Practice and our Find Your Community initiatives, allowing every employee to find connection, purpose, and belonging. Find out more about our Jacobs Employee Networks here.

Jacobs partners with VERCIDA to help us attract and retain talent from a wide range of backgrounds. For greater online accessibility please visit https://www.vercida.com/uk/employers/jacobs to view and access our roles.

As a disability confident employer, we will interview disabled candidates who best meet the criteria. We welcome applications from candidates who are seeking flexible working and from those who may not meet all the listed requirements for a role.

We value collaboration and believe that in-person interactions are crucial for both our culture and client delivery. We empower employees with our hybrid working policy, allowing them to split their work week between Jacobs offices/projects and remote locations enabling them to deliver their best work.

Your application experience is important to us, and we’re keen to adapt to make every interaction even better. If you require further support or reasonable adjustments with regards to the recruitment process (for example, you require the application form in a different format), please contact the team via Careers Support.


#J-18808-Ljbffr

Related Jobs

View all jobs

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

The Skills Gap in Machine Learning Jobs: What Universities Aren’t Teaching

Machine learning has moved from academic research into the core of modern business. From recommendation engines and fraud detection to medical imaging, autonomous systems and language models, machine learning now underpins many of the UK’s most critical technologies. Universities have responded quickly. Machine learning modules are now standard in computer science degrees, specialist MSc programmes have proliferated, and online courses promise to fast-track careers in the field. And yet, despite this growth in education, UK employers consistently report the same problem: Many candidates with machine learning qualifications are not job-ready. Roles remain open for months. Interview processes filter out large numbers of applicants. Graduates with strong theoretical knowledge struggle when faced with practical tasks. The issue is not intelligence or effort. It is a persistent skills gap between university-level machine learning education and real-world machine learning jobs. This article explores that gap in depth: what universities teach well, what they routinely miss, why the gap exists, what employers actually want, and how jobseekers can bridge the divide to build successful careers in machine learning.

Machine Learning Jobs for Career Switchers in Their 30s, 40s & 50s (UK Reality Check)

Are you considering a career change into machine learning in your 30s, 40s or 50s? You’re not alone. In the UK, organisations across industries such as finance, healthcare, retail, government & technology are investing in machine learning to improve decisions, automate processes & unlock new insights. But with all the hype, it can be hard to tell which roles are real job opportunities and which are just buzzwords. This article gives you a practical, UK-focused reality check: which machine learning roles truly exist, what skills employers really hire for, how long retraining realistically takes, how to position your experience and whether age matters in your favour or not. Whether you come from analytics, engineering, operations, research, compliance or business strategy, there is a credible route into machine learning if you approach it strategically.

How to Write a Machine Learning Job Ad That Attracts the Right People

Machine learning now sits at the heart of many UK organisations, powering everything from recommendation engines and fraud detection to forecasting, automation and decision support. As adoption grows, so does demand for skilled machine learning professionals. Yet many employers struggle to attract the right candidates. Machine learning job adverts often generate high volumes of applications, but few applicants have the blend of modelling skill, engineering awareness and real-world experience the role actually requires. Meanwhile, strong machine learning engineers and scientists quietly avoid adverts that feel vague, inflated or confused. In most cases, the issue is not the talent market — it is the job advert itself. Machine learning professionals are analytical, technically rigorous and highly selective. A poorly written job ad signals unclear expectations and low ML maturity. A well-written one signals credibility, focus and a serious approach to applied machine learning. This guide explains how to write a machine learning job ad that attracts the right people, improves applicant quality and strengthens your employer brand.