Data Engineer - Databricks Specialist - NonVolume

The Automobile Association
Basingstoke
1 day ago
Create job alert

Company description

Location: Basingstoke (hybrid working 3 office days per week)

Employment Type: Permanent, full time

Additional Benefits: Annual Bonus

Join Our Data & Analytics Team: Transforming Data into Our Superpower!

Are you passionate about data and eager to make a significant impact? The AA is a well-loved brand with a range of driver services much wider than most people realise. We have an enviable set of data assets from breakdown, service, repair, insurance, telematics, digital interactions, car dealers and driving school!

If that’s not enough, we’re focused on making Data our new Superpower as one of only 4 strategic priorities for the Group. Our growing team is modernising our data infrastructure to a cutting-edge cloud platform and enabling machine learning and GenAI. Join us at an exciting time and be part of a team that is driving meaningful change for our customers, colleagues and shareholders.

#LI-Hybrid

This is the job

At The AA, our purpose is to create confidence for drivers now and for the future. Data plays a critical role in delivering that purpose, and we are investing heavily in a modern, Databricks-centric data platform to unlock the full value of our connected car and insurance data.

This role is for an experienced Databricks Data Engineer. You will be working day-to-day designing, building and operating production-grade Databricks Lakehouse solutions, including structured streaming pipelines, Unity Catalog governance, and Spark-based data engineering at scale.

This is not a role for someone who has “touched” Databricks occasionally or supported it at the edges. We are looking for someone who has built, owned and evolved Databricks solutions, and who can confidently articulate design decisions, trade-offs, and best practice.

If Databricks is central to how you work and can demonstrate this natural, we want to hear from you.

What will I be doing?

  • Designing, building and operating production-grade Databricks Lakehouse solutions , including structured streaming pipelines using Python and PySpark
  • Owning and evolving Unity Catalog–based governance , ensuring secure, discoverable and well-managed data assets
  • Developing and maintaining event-driven data pipelines , integrating closely with backend engineering teams
  • Implementing and supporting CI/CD pipelines in Azure DevOps to enable reliable, automated Databricks deployments
  • Creating high-quality, analytics-ready datasets that deliver actionable business insight at scale
  • Proactively improving performance, reliability, automation and observability across the Databricks data platform

What do I need?

We are intentionally setting a high bar for Databricks experience. You should be able to demonstrate deep, hands-on capability, not just theoretical knowledge.

Essential experience

  • Significant, hands-on experience with Azure Databricks in a production environment
  • Proven experience building Spark / Databricks pipelines using Python and PySpark
  • Strong experience with structured streaming and event-driven architectures
  • Practical experience implementing and operating Unity Catalog
  • Solid understanding of Lakehouse design principles, including dimensional and analytical modelling
  • Experience building and maintaining CI/CD pipelines, ideally using Azure DevOps
  • Confidence working with large-scale data, performance tuning, and troubleshooting complex pipelines

What we mean by “Databricks experience”

You have:

  • Designed and built Databricks pipelines end-to-end
  • Made architectural decisions within Databricks environments
  • Worked with Spark internals, optimisation techniques and cluster configuration
  • Operated Databricks solutions in live, business-critical contexts

If your Databricks exposure has been limited to minor contributions, proof-of-concepts, or occasional usage alongside other tools, this role is unlikely to be the right fit.

Additional information

We’re always looking to recognise and reward our employees for the work they do. As a valued member of The AA team, you’ll have access to a range of benefits including:

  • 25 days annual leave plus bank holidays + holiday buying scheme
  • Worksave pension scheme with up to 7% employer contribution
  • Free AA breakdown membership from Day 1 plus 50% discount for family and friends
  • Discounts on AA products including car and home insurance
  • Employee discount scheme that gives you access to a car salary sacrifice scheme plus great discounts on healthcare, shopping, holidays and more
  • Company funded life assurance
  • Diverse learning and development opportunities to support you to progress in your career
  • Dedicated Employee Assistance Programme and a 24/7 remote GP service for you and your family

Plus, so much more!

We’re an equal opportunities employer and welcome applications from everyone. The AA values diversity and the difference this brings to our culture and our customers. We actively seek people from diverse backgrounds to join us and become part of an inclusive company where you can be yourself, be empowered to be your best and feel like you truly belong. We have five communities to bring together people with shared characteristics and backgrounds and drive positive change.

#LI-HH

Related Jobs

View all jobs

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Maths for Machine Learning Jobs: The Only Topics You Actually Need (& How to Learn Them)

Machine learning job adverts in the UK love vague phrases like “strong maths” or “solid fundamentals”. That can make the whole field feel gatekept especially if you are a career changer or a student who has not touched maths since A level. Here is the practical truth. For most roles on MachineLearningJobs.co.uk such as Machine Learning Engineer, Applied Scientist, Data Scientist, NLP Engineer, Computer Vision Engineer or MLOps Engineer with modelling responsibilities the maths you actually use is concentrated in four areas: Linear algebra essentials (vectors, matrices, projections, PCA intuition) Probability & statistics (uncertainty, metrics, sampling, base rates) Calculus essentials (derivatives, chain rule, gradients, backprop intuition) Basic optimisation (loss functions, gradient descent, regularisation, tuning) If you can do those four things well you can build models, debug training, evaluate properly, explain trade-offs & sound credible in interviews. This guide gives you a clear scope plus a six-week learning plan, portfolio projects & resources so you can learn with momentum rather than drowning in theory.

Neurodiversity in Machine Learning Careers: Turning Different Thinking into a Superpower

Machine learning is about more than just models & metrics. It’s about spotting patterns others miss, asking better questions, challenging assumptions & building systems that work reliably in the real world. That makes it a natural home for many neurodivergent people. If you live with ADHD, autism or dyslexia, you may have been told your brain is “too distracted”, “too literal” or “too disorganised” for a technical career. In reality, many of the traits that can make school or traditional offices hard are exactly the traits that make for excellent ML engineers, applied scientists & MLOps specialists. This guide is written for neurodivergent ML job seekers in the UK. We’ll explore: What neurodiversity means in a machine learning context How ADHD, autism & dyslexia strengths map to ML roles Practical workplace adjustments you can ask for under UK law How to talk about neurodivergence in applications & interviews By the end, you’ll have a clearer sense of where you might thrive in ML – & how to turn “different thinking” into a genuine career advantage.

Machine Learning Hiring Trends 2026: What to Watch Out For (For Job Seekers & Recruiters)

As we move into 2026, the machine learning jobs market in the UK is going through another big shift. Foundation models and generative AI are everywhere, companies are under pressure to show real ROI from AI, and cloud costs are being scrutinised like never before. Some organisations are slowing hiring or merging teams. Others are doubling down on machine learning, MLOps and AI platform engineering to stay competitive. The end result? Fewer fluffy “AI” roles, more focused machine learning roles with clear ownership and expectations. Whether you are a machine learning job seeker planning your next move, or a recruiter trying to build ML teams, understanding the key machine learning hiring trends for 2026 will help you stay ahead.