About the Team

The Rapid7 Data Platform is a unified, integrated platform powered by Rapid7’s product suite providing our customers enhanced visibility into their attack surface, operational efficiency, risk management, and decision-making capabilities. 

Our teams are responsible for consolidating data from all Rapid7 products, transforming it for optimised retrieval, and ensuring high-performance and seamless access to our customers. This role is crucial to the platform’s success as it focuses on building a highly scalable and reliable data mesh that powers cross-product use cases through a distributed query engine for big data analytics.

About the Role

We are seeking an innovative, self-motivated Data and Performance Engineer who will act as a technical leader to collaborate with our product teams to optimise their data pipelines and retrieval processes for performance and efficiency. You will work with the Data Platform teams to implement monitoring and testing strategies to ensure the performance of the data and their queries as well as identify optimisations.

Technologies you will work with:

Trino

Iceberg

Parquet

Spark

Airflow

Kafka

AWS services such as Glue, S3, EKS

In this role, you will:

Analyse and optimise distributed SQL queries to improve performance

Suggest optimisations to our data pipelines

Provide recommendations for efficient partitioning strategies and schema designs

Conduct performance tuning for the data pipelines and queries

Develop performance monitoring strategies and tools

The skills you'll bring include:

5+ years of hands-on software engineering experience, with a specific focus on database query optimization

Strong database system expertise in query execution planning, query optimization, performance tuning, parallel computing, and schema design

Experience in continuously monitoring and optimising data pipelines for performance and cost-effectiveness

Ability to design, develop, implement, and operate highly reliable large-scale data lake systems in cooperation with product teams

Skills to analyse and performance test the data mesh performance and scalability, identify bottlenecks, recommend and develop improvements

Mentorship and guidance of junior engineers, providing technical leadership and fostering a culture of continuous improvement and innovation

Excellent verbal and written communication skills.

Strong, creative problem solving ability.

Nice to haves:

Trino/Presto data-mesh

AWS, Terraform, Kubernetes

Java

Kafka

We believe the best ideas and solutions come from diverse teams. If you're excited about this role and feel your experience can make an impact, don't hesitate – apply today!