Be at the heart of actionFly remote-controlled drones into enemy territory to gather vital information.

Apply Now

Principal Data Engineer

Arqiva
City of London
1 week ago
Create job alert
Job Description
Location

We operate a flexible, hybrid working environment with the candidate required to travel to either our Winchester or London office once or twice a week.


We offer

  • Competitive Salary
  • 10% bonus
  • 6% pension contribution
  • Private Medical
  • 25 days annual leave
  • Access to our comprehensive flexible benefits including discounts on big brands, wellness and employee assistance programmes, gymflex, buy and sell annual leave, travel and dental insurance
  • Work. Life. Smarter. Our commitment to a flexible and hybrid working culture

Purpose

We're looking for a hands‑on Principal Data Engineer to lead the design and development of scalable, secure cloud‑based data platforms and pipelines that power enterprise analytics, reporting, and real‑time decision‑making.


This is a strategic and technical leadership role, ideal for someone with deep expertise in AWS, Databricks, Apache Spark, and Python/SQL, who can mentor teams, influence architecture, and deliver end‑to‑end data solutions across a modern tech stack.


Accountabilities

  • Design and deliver enterprise‑scale data architectures on AWS using services like S3, EMR, Glue, Redshift, Lambda, and IAM.
  • Build and manage robust ETL/ELT pipelines supporting both batch and real‑time data ingestion and transformation.
  • Deploy and optimise Apache Spark jobs, leveraging Databricks and Delta Lake for distributed data processing.
  • Lead the implementation of unified analytics workflows, integrating Databricks with the wider AWS environment.
  • Write clean, efficient, and reusable code in Python, building production‑ready data transformation scripts.
  • Develop complex SQL queries across multiple RDBMS and cloud‑native warehouses.
  • Drive data quality, lineage, and governance standards across structured and unstructured datasets.
  • Collaborate with data scientists, analysts, and business stakeholders to define and deliver scalable solutions.
  • Provide technical mentorship and thought leadership to data engineers and wider delivery teams.
  • Promote automation and operational efficiency through CI/CD, containerisation (e.g. Docker/ECS), and infrastructure as code.

Skills

  • Python, SQL, PySpark, Scala, Java
  • AWS (S3, Glue, Redshift, Lambda, EMR, IAM, ECS, SageMaker), Databricks, Delta Lake
  • Apache Spark, Airflow, Dagster, Rundeck
  • Kafka, Logstash, NiFi
  • PostgreSQL, Oracle DB, MongoDB
  • ELK Stack (Elasticsearch, Logstash, Kibana), Dynatrace, CloudWatch
  • GitLab, CI/CD, Docker, Infrastructure as Code
  • Agile/Scrum

Knowledge & Experience

  • Senior/principal data engineering roles with hands‑on leadership experience.
  • Proven success in designing and deploying cloud‑native data solutions, particularly on AWS.
  • Advanced experience with Apache Spark, Databricks, and real‑time data processing.
  • Strong capability in writing and optimising complex SQL and Python code.
  • Background in handling large‑scale data transformations and integrating multiple data sources.
  • Experience with data lakes, Delta Lake, and modern analytics platforms.
  • Skilled in stakeholder engagement, mentoring engineers, and delivering high‑quality solutions in fast‑paced environments.
  • A passion for simplifying complex problems and building future‑proof data infrastructure.


#J-18808-Ljbffr

Related Jobs

View all jobs

Principal Data Engineer/Architect

Principal Data Engineer

Principal Data Engineer

Principal Data Engineer

Principal Data Engineer

Principal Data Engineer

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Machine Learning Recruitment Trends 2025 (UK): What Job Seekers Need To Know About Today’s Hiring Process

Summary: UK machine learning hiring has shifted from title‑led CV screens to capability‑driven assessments that emphasise shipped ML/LLM features, robust evaluation, observability, safety/governance, cost control and measurable business impact. This guide explains what’s changed, what to expect in interviews & how to prepare—especially for ML engineers, applied scientists, LLM application engineers, ML platform/MLOps engineers and AI product managers. Who this is for: ML engineers, applied ML/LLM engineers, LLM/retrieval engineers, ML platform/MLOps/SRE, data scientists transitioning to production ML, AI product managers & tech‑lead candidates targeting roles in the UK.

Why Machine Learning Careers in the UK Are Becoming More Multidisciplinary

Machine learning (ML) has moved from research labs into mainstream UK businesses. From healthcare diagnostics to fraud detection, autonomous vehicles to recommendation engines, ML underpins critical services and consumer experiences. But the skillset required of today’s machine learning professionals is no longer purely technical. Employers increasingly seek multidisciplinary expertise: not only coding, algorithms & statistics, but also knowledge of law, ethics, psychology, linguistics & design. This article explores why UK machine learning careers are becoming more multidisciplinary, how these fields intersect with ML roles, and what both job-seekers & employers need to understand to succeed in a rapidly changing landscape.

Machine Learning Team Structures Explained: Who Does What in a Modern Machine Learning Department

Machine learning is now central to many advanced data-driven products and services across the UK. Whether you work in finance, healthcare, retail, autonomous vehicles, recommendation systems, robotics, or consumer applications, there’s a need for dedicated machine learning teams that can deliver models into production, maintain them, keep them secure, efficient, fair, and aligned with business objectives. If you’re hiring for or applying to ML roles via MachineLearningJobs.co.uk, this article will help you understand what roles are typically present in a mature machine learning department, how they collaborate through project lifecycles, what skills and qualifications UK employers look for, what the career paths and salaries are, current trends and challenges, and how to build an effective ML team.