Senior Data Engineer

Optima Partners
Edinburgh
4 days ago
Create job alert
Senior Data EngineerWho we are

We are an advanced data and business consultancy headquartered in Edinburgh, UK. We are a practitioner-led organisation that collaborates with top consumer brands to drive transformation and foster customer-centricity through our expertise in customer strategy, innovative design, and advanced data science and engineering.

We work with leading consumer brands to tackle and overcome complex business and customer problems to drive transformation and champion customer-centric agendas. We are proud to include some of the leading UK and global brands among our current clients such as Lloyds Banking Group, NatWest Group, Bank of Ireland, Nationwide, Aviva, Biogen, Eon Next, OVO, Virgin Media O2, BT, HMD Global, Centrica, and GSK. We are obsessive about delivering value for our clients and work in a collaborative, engaged, and creative way with our colleagues and clients. We strive to support the transition of knowledge and capability into strategic teams.

The opportunity

We have an exciting new opportunity for a Senior Data Engineer at Optima Partners. In this role, you will be instrumental in designing, building, and optimising scalable data pipelines, architectures, and workflows for our diverse client base. You’ll work within dynamic, multi-disciplinary teams to implement advanced data solutions while championing best practices and innovative trends in data engineering. This is a valuable opportunity for an experienced data professional to lead projects and drive technical excellence across a variety of commercial sectors.

At Optima, you’ll collaborate with experts across fields, gaining exposure to varied business challenges in an inclusive environment that fosters continuous learning, knowledge sharing, and clear career development.

What you will be doing
  • Data Pipeline Development: Design and implement robust, scalable, and efficient data pipelines to collect, transform, and integrate data from various sources.
  • Data Architecture: Develop and optimise data architectures, including data warehouses, data lakes, and other storage solutions.
  • ETL/ELT Processes: Create and maintain reliable ETL/ELT workflows to ensure data quality and accessibility.
  • Collaborate: Partner with data scientists, analysts, and client stakeholders to understand requirements and deliver impactful solutions.
  • Performance Optimisation: Monitor and enhance the performance of data systems to ensure minimal downtime and rapid query responses.
  • Automation and Tools: Identify automation opportunities and recommend tools to improve data engineering workflows.
  • Documentation: Maintain detailed technical documentation for all solutions and processes.
  • Leadership: Mentor junior engineers and lead technical initiatives within cross-functional teams.
What skills we would like you to have
  • Programming: Proficiency in Python, Java, Scala, or similar languages for data processing.
  • Big Data Technologies: Hands-on experience with tools such as Databricks, Apache Spark, and Hadoop.
  • Cloud Platforms: Familiarity with AWS, Azure, GCP, or other cloud ecosystems for data engineering tasks.
  • Database Management: Expertise in relational databases (e.g. Postgres, SQL Server).
  • Data Integration Tools: Knowledge of platforms like Airflow, Apache NiFi, or Talend.
  • Data Storage and Modelling: Experience with data warehousing tools (e.g. Snowflake, Redshift, BigQuery) and schema design.
  • Version Control and CI/CD: Familiarity with Git, Docker, and CI/CD pipelines for deployment.
  • Experience: 5+ years in data engineering or a related role, with a proven track record of delivering advanced data solutions in consulting or client-facing environments.
  • Methodologies: Experience with Agile or Scrum practices is beneficial.
Personal Qualities
  • Strong problem-solving and analytical skills.
  • Excellent communication abilities to explain complex technical concepts to non-technical stakeholders.
  • A collaborative, adaptable, and team-oriented mindset.


#J-18808-Ljbffr

Related Jobs

View all jobs

Senior Data Engineer

Senior Data Engineer

Senior Data Engineer

Senior Data Engineer

Senior Data Engineer - Energy

Senior Data Engineer, SQL, RDBMS, AWS, Python, Mainly Remote

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Machine Learning Jobs for Career Switchers in Their 30s, 40s & 50s (UK Reality Check)

Are you considering a career change into machine learning in your 30s, 40s or 50s? You’re not alone. In the UK, organisations across industries such as finance, healthcare, retail, government & technology are investing in machine learning to improve decisions, automate processes & unlock new insights. But with all the hype, it can be hard to tell which roles are real job opportunities and which are just buzzwords. This article gives you a practical, UK-focused reality check: which machine learning roles truly exist, what skills employers really hire for, how long retraining realistically takes, how to position your experience and whether age matters in your favour or not. Whether you come from analytics, engineering, operations, research, compliance or business strategy, there is a credible route into machine learning if you approach it strategically.

How to Write a Machine Learning Job Ad That Attracts the Right People

Machine learning now sits at the heart of many UK organisations, powering everything from recommendation engines and fraud detection to forecasting, automation and decision support. As adoption grows, so does demand for skilled machine learning professionals. Yet many employers struggle to attract the right candidates. Machine learning job adverts often generate high volumes of applications, but few applicants have the blend of modelling skill, engineering awareness and real-world experience the role actually requires. Meanwhile, strong machine learning engineers and scientists quietly avoid adverts that feel vague, inflated or confused. In most cases, the issue is not the talent market — it is the job advert itself. Machine learning professionals are analytical, technically rigorous and highly selective. A poorly written job ad signals unclear expectations and low ML maturity. A well-written one signals credibility, focus and a serious approach to applied machine learning. This guide explains how to write a machine learning job ad that attracts the right people, improves applicant quality and strengthens your employer brand.

Maths for Machine Learning Jobs: The Only Topics You Actually Need (& How to Learn Them)

Machine learning job adverts in the UK love vague phrases like “strong maths” or “solid fundamentals”. That can make the whole field feel gatekept especially if you are a career changer or a student who has not touched maths since A level. Here is the practical truth. For most roles on MachineLearningJobs.co.uk such as Machine Learning Engineer, Applied Scientist, Data Scientist, NLP Engineer, Computer Vision Engineer or MLOps Engineer with modelling responsibilities the maths you actually use is concentrated in four areas: Linear algebra essentials (vectors, matrices, projections, PCA intuition) Probability & statistics (uncertainty, metrics, sampling, base rates) Calculus essentials (derivatives, chain rule, gradients, backprop intuition) Basic optimisation (loss functions, gradient descent, regularisation, tuning) If you can do those four things well you can build models, debug training, evaluate properly, explain trade-offs & sound credible in interviews. This guide gives you a clear scope plus a six-week learning plan, portfolio projects & resources so you can learn with momentum rather than drowning in theory.