Senior Data Engineer

DataOps Engineer Are you an experienced developer with a 'can do' attitude and enthusiasm that inspires others? Do you enjoy being part of a team that works with a diverse range of technology? About our Team In our new DataOps T eam, we are focused on leveraging data, technology, and governance to elevate the quality of our pipelines to the highest level. We are in search of a Data Engineer to revolutionize and streamline our new data workflows and bring full data observability to both our pipelines and artificial intelligence efforts. The DataOps T eam is young and still building its roadmap, and you will play a critical role in shaping the team's future. The team occupies a unique and exciting position within the technology department, setting the standards and governance for all other data engineering teams. Your contributions will be vital in enabling quick access to insights, fostering a culture of continuous improvement, and driving innovation within our data practices. About the Role As a Data Engineer in the DataOps Team , your responsibilities will span the development and implementation of automated solutions for data integration, quality control, and continuous delivery. This role demands an excellent understanding of software engineering principles, strong programming skills, and good knowledge of DevOps tools. You'll be working in a small, highly skilled agile team, with ownership over the mission and your development practices and processes. You will collaborate with data engineers, data scientists, and data analysts both inside the team and across the technology department. Your colleagues will be UK-based, and you will work closely with the existing data engineering teams, as well as with a broader range of stakeholders distributed across the UK, Germany, Netherlands, and the USA. Responsibilities Designing , building, and maintaining efficient, reliable, and scalable data pipelines, based on both batch and streaming processing. Implementing tools and practices to monitor data quality, performance, and reliability across all data workflows. Visualize data quality through dashboards. Developing infrastructure, automation, and integrate various data sources and tools to enhance data operations. Working closely with data scientists, data engineers, and other stakeholders to understand data needs and deliver optimal solutions. Establishing and enforcing data governance policies to ensure compliance with both internal and wider standards and regulations. Ensuring product implementation plans have suitable metrics to reflect data quality. Working within agile practices. Foster a culture of continuous improvement by identifying and implementing process improvements. Driving innovation within data practices by exploring and adopting new technologies and methodologies. Optimizing both existing and new pipelines. Ensure suitable logging and monitoring tools are evaluated and used by other teams. Requirements Demonstrate experience working with both Python and PySpark. Demonstrate experience in implementing and managing data pipelines. Show understanding of data quality, metrics, and logging in data pipelines. Experience working with Databricks and its ecosystem is desirable. Proficiency in cloud platforms such as AWS, Azure, or Google Cloud for deploying and managing scalable data infrastructure and services. Knowledge of DevOps principles and practices for automating infrastructure provisioning, configuration management, and continuous integration/continuous deployment (CI/CD) pipelines. Ability to collaborate with cross-functional teams including data engineers, data scientists, and data analysts, and to work with/across multiple teams. Demonstrate problem-solving skills to troubleshoot data issues, optimize performance, and improve the reliability of data pipelines and infrastructure. Understanding of continuous software delivery processes. Work in a way that works for you We promote a healthy work/life balance across the organisation. We offer an appealing working prospect for our people. With numerous wellbeing initiatives, shared parental leave, study assistance and sabbaticals, we will help you meet your immediate responsibilities and your long-term goals. Working flexible hours - flexing the times when you work in the day to help you fit everything in and work when you are the most productive Working for you We know that your wellbeing and happiness are key to a long and successful career. These are some of the benefits we are delighted to offer: - Generous holiday allowance with the option to buy additional days - Health screening, eye care vouchers and private medical benefits - Wellbeing programs - Life assurance - Access to a competitive contributory pension scheme - Save As You Earn share option scheme - Travel Season ticket loan - Electric Vehicle Scheme - Optional Dental Insurance - Maternity, paternity and shared parental leave - Employee Assistance Programme - Access to emergency care for both the elderly and children - RECARES days, giving you time to support the charities and causes that matter to you - Access to employee resource groups with dedicated time to volunteer - Access to extensive learning and development resources - Access to employee discounts scheme via Perks at Work About the Business LexisNexis Legal & Professional® provides legal, regulatory, and business information and analytics that help customers increase their productivity, improve decision-making, achieve better outcomes, and advance the rule of law around the world. As a digital pioneer, the company was the first to bring legal and business information online with its Lexis® and Nexis® services.