Senior Data Engineer

About the Role

 

As a Data Engineer in the DataOps Team, your responsibilities will span the development and implementation of automated solutions for data integration, quality control, and continuous delivery. This role demands an excellent understanding of software engineering principles, strong programming skills, and good knowledge of DevOps tools.

You’ll be working in a small, highly skilled agile team, with ownership over the mission and your development practices and processes. You will collaborate with data engineers, data scientists, and data analysts both inside the team and across the technology department. Your colleagues will be UK-based, and you will work closely with the existing data engineering teams, as well as with a broader range of stakeholders distributed across the UK, Germany, Netherlands, and the USA.

Responsibilities
 

Designing, building, and maintaining efficient, reliable, and scalable data pipelines, based on both batch and streaming processing. Implementingtools and practices to monitor data quality, performance, and reliability across all data workflows. Visualize data quality through dashboards. Developinginfrastructure, automation, and integrate various data sources and tools to enhance data operations. Workingclosely with data scientists, data engineers, and other stakeholders to understand data needs and deliver optimal solutions. Establishingand enforcing data governance policies to ensure compliance with both internal and wider standards and regulations. Ensuring product implementation plans have suitable metrics to reflect data quality. Workingwithin agile practices. Foster a culture of continuous improvement by identifying and implementing process improvements. Drivinginnovation within data practices by exploring and adopting new technologies and methodologies. Optimizingboth existing and new pipelines. Ensure suitable logging and monitoring tools are evaluated and used by other teams.

Requirements
 

Demonstrate experience working with both Python and PySpark.  Demonstrate experience in implementing and managing data pipelines.  Show understanding of data quality, metrics, and logging in data pipelines.  Experience working with Databricks and its ecosystem is desirable.  Proficiency in cloud platforms such as AWS, Azure, or Google Cloud for deploying and managing scalable data infrastructure and services.  Knowledge of DevOps principles and practices for automating infrastructure provisioning, configuration management, and continuous integration/continuous deployment (CI/CD) pipelines.  Ability to collaborate with cross-functional teams including data engineers, data scientists, and data analysts, and to work with/across multiple teams.  Demonstrate problem-solving skills to troubleshoot data issues, optimize performance, and improve the reliability of data pipelines and infrastructure.  Understanding of continuous software delivery processes. 

Work in a way that works for you
 

We promote a healthy work/life balance across the organisation. We offer an appealing working prospect for our people. With numerous wellbeing initiatives, shared parental leave, study assistance and sabbaticals, we will help you meet your immediate responsibilities and your long-term goals.
 

Working flexible hours - flexing the times when you work in the day to help you fit everything in and work when you are the most productive

Working for you
 

We know that your wellbeing and happiness are key to a long and successful career. These are some of the benefits we are delighted to offer
- Generous holiday allowance with the option to buy additional days
- Health screening, eye care vouchers and private medical benefits
- Wellbeing programs
- Life assurance
- Access to a competitive contributory pension scheme
- Save As You Earn share option scheme
- Travel Season ticket loan
- Electric Vehicle Scheme
- Optional Dental Insurance
- Maternity, paternity and shared parental leave
- Employee Assistance Programme
- Access to emergency care for both the elderly and children
- RECARES days, giving you time to support the charities and causes that matter to you
- Access to employee resource groups with dedicated time to volunteer
- Access to extensive learning and development resources
- Access to employee discounts scheme via Perks at Work