About the Team
The Rapid7 Data Platform is a unified, integrated platform powered by Rapid7’s product suite providing our customers enhanced visibility into their attack surface, operational efficiency, risk management, and decision-making capabilities.
Our teams are responsible for consolidating data from all Rapid7 products, transforming it for optimised retrieval, and ensuring high-performance and seamless access to our customers. This role is crucial to the platform’s success as it focuses on building a highly scalable and reliable data mesh that powers cross-product use cases through a distributed query engine for big data analytics.
About the Role
We are seeking an innovative, self-motivated Data and Performance Engineer who will act as a technical leader to collaborate with our product teams to optimise their data pipelines and retrieval processes for performance and efficiency. You will work with the Data Platform teams to implement monitoring and testing strategies to ensure the performance of the data and their queries as well as identify optimisations.
Technologies you will work with:
Trino
Iceberg
Parquet
Spark
Airflow
Kafka
AWS services such as Glue, S3, EKS
In this role, you will:
Analyse and optimise distributed SQL queries to improve performance
Suggest optimisations to our data pipelines
Provide recommendations for efficient partitioning strategies and schema designs
Conduct performance tuning for the data pipelines and queries
Develop performance monitoring strategies and tools
The skills you'll bring include:
5+ years of hands-on software engineering experience, with a specific focus on database query optimization
Strong database system expertise in query execution planning, query optimization, performance tuning, parallel computing, and schema design
Experience in continuously monitoring and optimising data pipelines for performance and cost-effectiveness
Ability to design, develop, implement, and operate highly reliable large-scale data lake systems in cooperation with product teams
Skills to analyse and performance test the data mesh performance and scalability, identify bottlenecks, recommend and develop improvements
Mentorship and guidance of junior engineers, providing technical leadership and fostering a culture of continuous improvement and innovation
Excellent verbal and written communication skills.
Strong, creative problem solving ability.
Nice to haves:
Trino/Presto data-mesh
AWS, Terraform, Kubernetes
Java
Kafka
We believe the best ideas and solutions come from diverse teams. If you're excited about this role and feel your experience can make an impact, don't hesitate – apply today!
The latest news, articles, and resources, sent to your inbox weekly.
We have an opportunity to impact your career and provide an adventure where you can push the limits of what's possible. As a Python / Spark Big Data Lead Software Engineer at JPMorgan Chase within the Capital Technology team, you will play a crucial role in an agile team, focusing...
JPMorgan Chase & Co. Glasgow
Canonical is building a comprehensive suite of multi-cloud and on-premise data solutions for the enterprise. We want to make it easy to operate any database on any cloud, or on premise. The data platform team covers the full range of data stores and data technologies, spanning from big data, NoSQL,...
Canonical London
Responsibilitiesand Impact:You will enable S&P ratings business to provide best in class end-to-end analytical,mercial and customer experience by building feature rich solutions including big data engineering, analytics, business intelligence and workflow capabilities.What We're Looking For10+ years of software engineering experience as individual contributor and as people leader Familiarity with well...
S&P Global London
Platform Engineering Manager - £100K – Hybrid LondonAn excellent opportunity has arisen in a Financial Services organisation, who are looking for an experienced Platform Engineering Manager to join their dynamic team in London where you’ll be focusing mainly on infrastructure and security programmes.In this position, you will lead intricate, cross-functional...
Tower, Greater London
Platform Engineering Manager - £100K Hybrid LondonAn excellent opportunity has arisen in a Financial Services organisation, who are looking for an experienced Platform Engineering Manager to join their dynamic team in London where youll be focusing mainly on infrastructure and security programmes.In this position, you will lead intricate, cross-functional projects...
Intec London
Job DescriptionThe RoleAs a Lead Typescript / Node engineer for this AI cyber security business you will work to lead technical products around distributed systems and big data. This is very much a hands-on delivery role where you will be leading from a technical perspective and providing some mentorship to...
Propel London London
