Senior Data Engineer

Office for National Statistics
united kingdom
7 months ago
Create job alert

Job summary

This is an exciting opportunity to join our Integrated Data Sharing platform (IDS) Data Engineering team. The IDS Programme aims to enable data sharing and innovative cross-cutting analysis across and at the heart of government through a single platform, and represents a large step-change in how government utilises data.

This role will be at the core of this, designing, maintaining existing and building new systems, pipelines and data products within a team of GCP Data Engineers.

As well as the day-to-day processing and delivery, we work as a team to develop the capability of the organisation � developing re-usable and increasingly automated code, tools, and best practice that we and others across the organisation can use to build the data pipelines needed to support the growing work of a changing organisation.

Job description

Our tech stack is 60% Python, PySpark and SQL and the remaining 40% GCP eco system tools. We�re looking for someone with relevant technical skills but also a great attitude, who is proactive, takes pride in the quality of their team�s work and has an eagerness to develop their own technical skills and their team�s.

Training will be provided in the aspects of the work the successful candidate is not familiar with and they will be buddied with an experienced peer. This role will give exposure to the GCP eco system, Pub/Sub, Cloud functions, AirFlow, GitHub actions, CI/CD, DevOps practices and Agile project delivery with cross-government stakeholders; all while working alongside other highly skilled GCP Data and Platform Engineers.

We encourage training and development so that we can stay on top of changing technology. We invest in the development of our staff with extensive internal and external training opportunities.

Responsibilities

Responsibilities include but are not limited to:

Designing and building robust data pipelines, ETL processes, data validation and other products using Python, SQL, BigQuery, Spark, DataProc, AirFlow and wider GCP eco system. Maintenance of existing pipelines. Orchestrating services and systems using Pub/Sub and AirFlow. Manage our data estate. Investigation and mitigation of data quality issues by implementing appropriate solutions. Actively seek to improve existing pattern and best practices. Encourage and lead discussions of team standards and their implementation collaboratively. Producing a high standard of technical work and non-technical, setting an example for the team. Horizon scanning for opportunities and risks, planning future work and actions to mitigate risks. Proactively using self-development time to grow own technical and non-technical skillsets. Encourage initiatives that build a culture of self-development across the team. Becoming an expert in areas key to the project.�

Person specification

Essential Skills Criteria:

Technical Skills:Experience utilising Python, PySpark, SQL and GIT.Programming and build: You know how to use agreed standards and tools to design, code, test, correct and document moderate-to-complex programs and scripts from agreed specifications and subsequent iterations. You can collaborate with others to review specifications where appropriate.Data development process:You know how to establish enterprise-scale data integration procedures across the data development life cycle and ensure that teams adhere to these. You can manage resources to ensure that data services work effectively.Problem resolution:You can ensure that the most appropriate actions are taken to resolve problems as they occur. You know how to coordinates teams to resolve problems and implement solutions and preventative measures.Communication skills:You can listen to the needs of technical and business stakeholders and interpret them. You are flexible and capable of proactive and reactive communication. You know how to facilitate difficult discussions within the team or with diverse senior stakeholders.

Desirable Skills:

Experience working on GCP

Behaviours

We'll assess you against these behaviours during the selection process:

Making Effective Decisions Changing and Improving Working Together

Technical skills

We'll assess you against these technical skills during the selection process:

Programming and build - Experience of designing and building data pipelines using Python and Spark eco systems following industry best practices. Problem resolution - Problem-solving skills and ability to troubleshoot data pipeline issues, analyse logs and error messages, and implement effective solutions to ensure continuous data flow.

Benefits

Alongside your salary of �53,352, Office for National Statistics contributes �15,456 towards you being a member of the Civil Service Defined Benefit Pension scheme.

The is part of the Civil Service, and as such we share a number of key benefits with other departments, whilst also having our own unique offerings to support our 5400 valued colleagues across the business.

This role is part of the cross-government Government Digital and Data (formerly DDaT) profession framework. As a role within Government Digital and Data (formerly DDaT) at the ONS, we also offer benefits such as:

� Protected Learning Time to spend on your personal development and side-projects.

� A supportive and active Community of Practice which you will be expected to contribute to, helping ensure you and your colleagues get the training, development and opportunities you need to progress your careers.

ONS are committed to flexible ways of working that support a healthy work-life balance. ONS has already considered how this job could be right sized for applicants working flexibly and we are happy to explore options with you about working part time, in a job share or flexibly, in line with our hybrid working policies.�

Whether you are hearing about us for the first time or already know a bit about our organisation, we hope that the benefits pack attached (bottom of page) will give you a great insight into the benefits and facilities available to our colleagues and our fantastic working culture.

Inclusion & Accessibility

At ONS we are always looking to attract the very best people from the widest possible talent pool, and we are proud to be an inclusive, equal opportunities employer. As a member of the Business Disability Forum and a Disability Confident Leader we�re committed to ensuring that all candidates are treated fairly throughout the recruitment process.

As part of our application process, you will be prompted to provide details of any reasonable adjustments to our recruitment process that you need. If you would like to discuss any reasonable adjustments before applying, please contact the recruitment team in the first instance.

If you would like an accessible version of any of the attachments or recruitment documents below or linked to in this advert, please contact the recruitment team who will be happy to assist.

Related Jobs

View all jobs

Senior Data Engineer

Senior Data Engineer

Senior Data Engineer

Senior Data Engineer

Senior Data Engineer - Databricks

Senior Data Engineer - DV Cleared

Get the latest insights and jobs direct. Sign up for our newsletter.

By subscribing you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Negotiating Your Machine Learning Job Offer: Equity, Bonuses & Perks Explained

How to Secure a Compensation Package That Matches Your Technical Mastery and Strategic Influence in the UK’s ML Landscape Machine learning (ML) has rapidly shifted from an emerging discipline to a mission-critical function in modern enterprises. From optimising e-commerce recommendations to powering autonomous vehicles and driving innovation in healthcare, ML experts hold the keys to transformative outcomes. As a mid‑senior professional in this field, you’re not only crafting sophisticated algorithms; you’re often guiding strategic decisions about data pipelines, model deployment, and product direction. With such a powerful impact on business results, companies across the UK are going beyond standard salary structures to attract top ML talent. Negotiating a compensation package that truly reflects your value means looking beyond the numbers on your monthly payslip. In addition to a competitive base salary, you could be securing equity, performance-based bonuses, and perks that support your ongoing research, development, and growth. However, many mid‑senior ML professionals leave these additional benefits on the table—either because they’re unsure how to negotiate them or they simply underestimate their long-term worth. This guide explores every critical aspect of negotiating a machine learning job offer. Whether you’re joining an AI-focused start-up or a major tech player expanding its ML capabilities, understanding equity structures, bonus schemes, and strategic perks will help you lock in a package that matches your technical expertise and strategic influence. Let’s dive in.

Machine Learning Jobs in the Public Sector: Opportunities Across GDS, NHS, MOD, and More

Machine learning (ML) has rapidly moved from academic research labs to the heart of industrial and governmental operations. Its ability to uncover patterns, predict outcomes, and automate complex tasks has revolutionised industries ranging from finance to retail. Now, the public sector—encompassing government departments, healthcare systems, and defence agencies—has become an increasingly fertile ground for machine learning jobs. Why? Because government bodies oversee vast datasets, manage critical services for millions of citizens, and must operate efficiently under tight resource constraints. From using ML algorithms to improve patient outcomes in the NHS, to enhancing cybersecurity within the Ministry of Defence (MOD), there’s a growing demand for skilled ML professionals in UK public sector roles. If you’re passionate about harnessing data-driven insights to solve large-scale problems and contribute to societal well-being, machine learning jobs in the public sector offer an unparalleled blend of challenge and impact. In this article, we’ll explore the key reasons behind the public sector’s investment in ML, highlight the leading organisations, outline common job roles, and provide practical guidance on securing a machine learning position that helps shape the future of government services.

Contract vs Permanent Machine Learning Jobs: Which Pays Better in 2025?

Machine learning (ML) has swiftly become one of the most transformative forces in the UK technology landscape. From conversational AI and autonomous vehicles to fraud detection and personalised recommendations, ML algorithms are reshaping how organisations operate and how consumers experience products and services. In response, job opportunities in machine learning—including roles in data science, MLOps, natural language processing (NLP), computer vision, and more—have risen dramatically. Yet, as the demand for ML expertise booms, professionals face a pivotal choice about how they want to work. Some choose day‑rate contracting, leveraging short-term projects for potentially higher immediate pay. Others embrace fixed-term contract (FTC) roles for mid-range stability, or permanent positions for comprehensive benefits and a well-defined career path. In this article, we will explore these different employment models, highlighting the pros and cons of each, offering sample take‑home pay scenarios, and providing insights into which path might pay better in 2025. Whether you’re a new graduate with a machine learning degree or an experienced practitioner pivoting into an ML-heavy role, understanding these options is key to making informed career decisions.