Senior Data / AI Engineer

Location: London (Hybrid, 2 days per week onsite)
Salary: Up to £85,000

We are seeking a Senior Data / AI Engineer to play a key role in building and scaling innovative data products. This is more than a traditional data engineering position. You will work closely with product, commercial and technology teams to transform raw and third-party data into valuable, customer-facing solutions.

The role combines modern data engineering, AI capabilities and product thinking, offering the opportunity to shape how data is monetised, enriched and delivered to end users.

• Design and build scalable data products that support commercial and customer-facing solutions.
• Transform raw and third-party data into structured, enriched and product-ready datasets.
• Collaborate with product and commercial stakeholders to define how data is packaged, accessed and delivered.
• Develop data delivery solutions through APIs, internal tools and customer-facing platforms.

• Apply AI and LLM technologies to enrich datasets through classification, tagging, summarisation and insight generation.
• Design and develop LLM-powered product features.
• Implement Retrieval-Augmented Generation (RAG) solutions to enhance data quality and unlock new product capabilities.
• Support the development of AI-driven solutions including intelligent search, recommendation engines and insights platforms.
• Ensure AI systems are scalable, resilient and governed appropriately.

• Support the deployment, operation and lifecycle management of data and AI platforms.
• Own and evolve the Azure Databricks-based data platform architecture to support business growth.
• Drive performance and cost optimisation across Azure services through monitoring, tuning and automation.
• Build and maintain reliable batch and streaming data pipelines.
• Implement robust testing, observability and data quality controls.
• Promote engineering best practices through CI/CD, automation and platform standardisation.

• Partner with business and product stakeholders to identify high-value data opportunities.
• Translate business requirements into scalable technical solutions.
• Act as a bridge between commercial, product and engineering teams.
• Support prioritisation decisions based on business value and impact.

• Contribute to the wider data and AI strategy.
• Mentor junior engineers and promote engineering excellence.
• Help build a high-performing team with a strong product and commercial mindset.

• Extensive experience designing, building and operating production data platforms using Azure Databricks and Azure.
• Strong hands-on expertise with Apache Spark, PySpark, Spark SQL and Delta Lake.
• Experience implementing and managing Unity Catalog, including governance, lineage and access controls.
• Proven delivery of lakehouse and Medallion architectures.
• Advanced Python and SQL skills for scalable, production-grade solutions.
• Experience building modern batch and real-time data pipelines.
• Strong understanding of API design and data product delivery mechanisms.
• Experience deploying Databricks solutions using CI/CD pipelines, Azure DevOps and/or GitHub Actions.
• Experience integrating Databricks with ADLS Gen2, Azure Identity, Key Vault and external data sources.

• Experience building customer-facing data products or commercial data solutions.
• Strong understanding of how data can be structured, exposed and scaled for end users.
• Ability to focus on measurable business outcomes rather than solely technical delivery.
• Experience working closely with product, commercial or customer-facing teams.

• Practical experience with Generative AI and LLM technologies such as Azure OpenAI and LangChain.
• Experience applying AI solutions to real-world business challenges.
• Knowledge of embeddings, vector databases and semantic search.
• Experience working with unstructured data including text and document-based datasets.
• Understanding of AI governance, monitoring and ethical considerations.

• Airbyte or similar modern data ingestion platforms.
• Terraform and Infrastructure as Code.
• LLM frameworks such as LangChain or LlamaIndex.
• Experimentation, testing or A/B testing frameworks.
• Experience building AI-enabled or data-driven products.
• Background within Life Sciences, Healthcare or highly regulated industries.

• Product-focused and driven by business impact.
• Equally comfortable discussing technical solutions and commercial objectives.
• Passionate about delivering production-ready solutions rather than proofs of concept.
• Pragmatic, collaborative and delivery-oriented.
• Naturally curious about emerging AI technologies and how they can create commercial value.