Principal Software Engineer - Data Platform

Rapid7
Belfast
6 months ago
Create job alert

As a Principal Engineer, you’ll get the opportunity to be a hands-on engineer, learning best practice engineering processes and approaches whilst receiving ongoing development through coaching, mentoring and pairing with other engineers on your team. From problem-solving to challenging old ways of thinking, you will have the opportunity to unleash your full potential and creativity whilst working with cutting edge technologies in a dynamic and collaborative team.

About the Team

The Data Platform team is responsible for building ETL Pipelines that fuel the Data Platform at Rapid7. Moving Product Data into our Data Platform for product teams to develop new features, enhance existing features and build shared experiences to create value for customers across the world.

We have a cutting edge data stack including Kafka, K8s, Spark and Iceberg.

About the Role


The Principal Engineer role is a part of our Data Platform Engineering team. In this role you will be focussed on helping our product teams move data into our Data Platform for in product experiences and product analytics.

As a Principal Engineer on the Data Platform Engineering team, you will be responsible for architecting and scaling streaming and batch data pipelines, while also designing the CI/CD infrastructure that ensures efficient development and deployment of data services. You will play a key role in shaping the architecture of our data platform, collaborating with cross-functional teams to deliver highly available, performant, and scalable solutions for both real-time and large-scale data processing.

In this role, you will:

Architect and implement a highly scalable Data Platform that supports Change Data Capture (CDC) using Debezium and Kafka for data replication across different databases and services.

Design and maintain large-scale data lakes using Apache Iceberg, ensuring efficient data partitioning, versioning, and schema evolution to support real-time analytics and historical data access.

Build and optimize CI/CD pipelines for the deployment and automation of data platform services using tools like Jenkins.

Lead the integration of Apache Spark for large-scale data processing and ensure that both batch and streaming workloads are handled efficiently.

Collaborate with our Platform Delivery teams to ensure high availability and performance of the data platform, implementing monitoring, disaster recovery, and automated testing frameworks.

Provide technical leadership and mentoring to junior engineers, promoting best practices in CDC architecture, distributed systems, and CI/CD automation.

Ensure that the platform adheres to data governance principles, including data lineage tracking, auditing, and compliance with regulatory requirements.

Stay informed about the latest advancements in CDC, data engineering, and infrastructure automation to guide future platform improvements.

Work closely with product and data science teams to understand business requirements and translate them into scalable and efficient data platform solutions.

Stay current with the latest trends in data engineering and infrastructure, making recommendations for improvements and introducing new technologies as appropriate.

The skills you’ll bring include:

10+ years of experience in software engineering with a focus on data platform engineering, data infrastructure, or distributed systems.

Expertise in building data pipelines using Apache Kafka or similar for ingesting, processing, and distributing high-throughput data.

Strong experience designing and managing CI/CD pipelines for data platform services using tools such as Jenkins.

Experience with Apache Iceberg (or similar Delta Lake/Apache Hudi) for managing versioned, partitioned datasets in data lakes with an understanding of Apache Spark for both batch and streaming data processing, including optimization strategies for distributed data workloads.

Expertise in designing distributed systems and managing high-throughput, fault-tolerant, and low-latency data architectures.

Strong programming skills in Java, Scala, or Python.

Experience with cloud-based environments (AWS, GCP, Azure) and containerized infrastructure using Kubernetes and Docker.

The attitude and ability to thrive in a high-growth, evolving environment

Collaborative team player who has the ability to partner with others and drive toward solutions

Strong creative problem solving skills

Solid communicator with excellent written and verbal communications skills both within the team and cross functionally

Passionate about delighting customers, puts the customer needs at the forefront of all decision making

Excellent attention to detail


We know that the best ideas and solutions come from multi-dimensional teams. That’s because these teams reflect a variety of backgrounds and professional experiences. If you are excited about this role and feel your experience can make an impact, please don’t be shy - apply today.

#LI_FB1

Related Jobs

View all jobs

Senior Software Engineer (GO/PHP)

Principal Engineer – Software Engineering

[Immediate Start] Principal Staff Software Engineer, AI andData Infrastructure

▷ [15h Left] Manager, Software Development, TransportationFinancial Systems

Principal Product Analytics Developer /Team Lead, Newcastle

Prinicpal Pricing Analyst - Actuarial Pricing

Get the latest insights and jobs direct. Sign up for our newsletter.

By subscribing you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Machine‑Learning Jobs for Non‑Technical Professionals: Where Do You Fit In?

The Model Needs More Than Math When ChatGPT went viral and London start‑ups raised seed rounds around “foundation models,” many professionals asked, “Do I need to learn PyTorch to work in machine learning?” The answer is no. According to the Turing Institute’s UK ML Industry Survey 2024, 39 % of advertised ML roles focus on strategy, compliance, product or operations rather than writing code. As models move from proof‑of‑concept to production, demand surges for specialists who translate algorithms into business value, manage risk and drive adoption. This guide reveals the fastest‑growing non‑coding ML roles, the transferable skills you may already have, real transition stories and a 90‑day action plan—no gradient descent necessary.

Quantexa Machine‑Learning Jobs in 2025: Your Complete UK Guide to Joining the Decision‑Intelligence Revolution

Money‑laundering rings, sanctioned entities, synthetic identities—complex risks hide in plain sight inside data. Quantexa, a London‑born scale‑up now valued at US $2.2 bn (Series F, August 2024), solves that problem with contextual decision‑intelligence (DI): graph analytics, entity resolution and machine learning stitched into a single platform. Banks, insurers, telecoms and governments from HSBC to HMRC use Quantexa to spot fraud, combat financial crime and optimise customer engagement. With the launch of Quantexa AI Studio in February 2025—bringing generative AI co‑pilots and large‑scale Graph Neural Networks (GNNs) to the platform—the company is hiring at record pace. The Quantexa careers portal lists 450+ open roles worldwide, over 220 in the UK across data science, software engineering, ML Ops and client delivery. Whether you are a graduate data scientist fluent in Python, a Scala veteran who loves Spark or a solutions architect who can turn messy data into knowledge graphs, this guide explains how to land a Quantexa machine‑learning job in 2025.

Machine Learning vs. Deep Learning vs. MLOps Jobs: Which Path Should You Choose?

Machine Learning (ML) continues to transform how businesses operate, from personalised product recommendations to automated fraud detection. As ML adoption accelerates in nearly every industry—finance, healthcare, retail, automotive, and beyond—the demand for professionals with specialised ML skills is surging. Yet as you browse Machine Learning jobs on www.machinelearningjobs.co.uk, you may encounter multiple sub-disciplines, such as Deep Learning and MLOps. Each of these fields offers unique challenges, requires a distinct skill set, and can lead to a rewarding career path. So how do Machine Learning, Deep Learning, and MLOps differ? And which area best aligns with your talents and aspirations? This comprehensive guide will define each field, highlight overlaps and differences, discuss salary ranges and typical responsibilities, and explore real-world examples. By the end, you’ll have a clearer vision of which career track suits you—whether you prefer building foundational ML models, pushing the boundaries of neural network performance, or orchestrating robust ML pipelines at scale.