As part of this role, you'll be responsible for:
• Hands on Leadership of the Data and AI Engineering teams and ML Ops capability based in the UK and India
• Increase the capability and standards of our Engineering and ML Ops function
• Lead the evolution of our data engineering framework on Azure Databricks
• Work with the Principal AI engineer of the adaptation and evolution of our Gen AI Platform
• Develop and socialise Data, AI and ML Ops standards across our data engineering and data science practices
• Assist in the creation of solution designs for the approval by our architecture, governance and security functions
• Develop and implement patterns to ingest and analyse unstructured data
• Ensure cost efficient data operations on the platform
• Ensure delivery of robust and efficient data pipelines and architectures that support our data science teams
• Stay abreast of the latest industry trends and technologies, integrating new technology and best practices into the team's workflows where appropriate
• Implement, maintain and evangelise best practices for data engineering, including data governance, security, and compliance.
• Optimize performance, scalability, and efficiency of data pipelines and ML Ops processes
• Drive continuous improvement initiatives, leveraging feedback and performance metrics to enhance team effectiveness and project outcomes.
• Minimise consumption and storage costs across our Azure Databricks platform
This role will define, document and enforce data and ML engineering standards. It will have accountability for the performant and efficient build and operation of data components on our Azure Databricks Cloud platform. The role will be responsible for ensuring that we are utilizing the optimal software components on the platform and will own our data ingestion patterns and ML Ops practice.
You will need:
• Experience delivering data engineering solutions in cross functional environments
• Experience of leading data engineering teams / ML Ops functions and supporting Data Science teams in cross functional environments
• Deep understanding of Azure services, including Azure Databricks, Azure Data Factory, Azure Synapse Analytics, Azure Storage, and Azure ML
• Proficiency in managing and optimizing cloud resources for performance and cost-efficiency
• Extensive experience with Databricks, including Spark SQL, Delta Lake, and Databricks clusters
• Experience in deploying, monitoring, and optimizing machine learning models in a production environment
• Proficiency in programming languages such as Python, Scala, and SQL
• Knowledge of scripting languages for automation and orchestration
• Experience with DevOps practices
• Familiarity with containerization
• Excellent written, oral communication and advocacy skills, with demonstrable experience of prioritising effectively, managing diverse and multiple stakeholders
We know our people are the secret to our success. That's why we're always looking for ways to reward great work. You'll find a host of benefits designed to work for you, including:
- Company bonus
- Private Medical
- Pension
