Lead Data Engineer - Commodities Trading | London, UK | Hybrid

twentyAI
London
1 week ago
Create job alert

Are you a Data Engineering leader looking for your next big challenge? I'm searching for a Lead Data Engineer to spearhead an exciting global trade surveillance initiative for a prestigious client within the commodity trading sector. This is an opportunity to shape the future of compliance-driven data solutions, leveraging cutting-edge technologies like Azure Synapse and DataBricks to optimise data ingestion and processing.

Role Responsibilities:

  • Leading a team of data engineers to design and implement scalable data pipelines for trade surveillance.
  • Owning the global roadmap for trade surveillance rollout, ensuring seamless integration with business initiatives.
  • Driving the evolution of the data lakehouse architecture, enhancing speed, efficiency, and quality.
  • Acting as the key technical point of contact, influencing strategic decision-making.
  • Fostering a high-performance, agile team culture that thrives on innovation.

Your background:

  • Proven experience leading high-performing data engineering teams in a fast-paced environment.
  • Expertise in Python, PySpark, SQL, Synapse, DataBricks, and Azure Data Factory (ADF).
  • Strong knowledge of CI/CD pipelines, Git, Jenkins, Docker, and test automation (PyTest).
  • Hands-on experience with Big Data architectures, Data Lakehousing, and real-time data processing.
  • Exceptional communication skills—translating complex technical challenges to both technical and non-technical stakeholders.
  • Bonus: Experience in commodity trading operations (Oil, Gas, Shipping, etc.).

#J-18808-Ljbffr

Related Jobs

View all jobs

Lead Data Engineer

Lead Data Engineer

Lead Data Engineer

Lead Data Engineer

Lead Data Engineer

Lead Data Engineer - Databricks

Get the latest insights and jobs direct. Sign up for our newsletter.

By subscribing you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Machine Learning Leadership for Managers: Strategies to Motivate, Mentor, and Set Realistic Goals in Data-Driven Teams

Machine learning (ML) has become an indispensable force in the modern business world, influencing everything from targeted marketing campaigns to advanced medical diagnostics. As industries integrate predictive algorithms and data-driven decision-making into their core operations, the need for effective leadership in machine learning environments has never been greater. Whether you’re overseeing a small team of data scientists or spearheading an enterprise-scale ML project, your leadership style must accommodate rapid innovation, complex problem-solving, and diverse stakeholder expectations. This guide provides actionable insights into how you can motivate, mentor, and establish achievable goals for your machine learning teams—ensuring they thrive in data-driven environments.

Top 10 Books to Advance Your Machine Learning Career in the UK

Machine learning (ML) remains one of the fastest-growing fields within technology, reshaping industries across the UK from finance and healthcare to e-commerce, telecommunications, and beyond. With increasing demand for ML specialists, job seekers who continually update their knowledge and skills hold a significant advantage. In this article, we've curated ten essential books every machine learning professional or aspiring ML engineer in the UK should read. Covering foundational theory, practical implementations, advanced techniques, and industry trends, these resources will equip you to excel in your machine learning career.

Navigating Machine Learning Career Fairs Like a Pro: Preparing Your Pitch, Questions to Ask, and Follow-Up Strategies to Stand Out

Machine learning (ML) has swiftly become one of the most in-demand skill areas across industries, with companies leveraging predictive models and data-driven insights to solve challenges in healthcare, finance, retail, manufacturing, and beyond. Whether you’re an early-career data scientist aiming to break into ML, a seasoned engineer branching into deep learning, or a product manager exploring AI-driven solutions, machine learning career fairs offer a powerful route to connect with prospective employers face-to-face. Attending these events can help you: Network with hiring managers and technical leads who make direct recruitment decisions. Gain insider insights on the latest ML trends and tools. Learn about emerging job roles and new industry verticals adopting machine learning. Showcase your interpersonal and communication skills, both of which are increasingly important in collaborative AI/ML environments. However, with many applicants vying for attention in a bustling hall, standing out isn’t always easy. In this detailed guide, we’ll walk you through how to prepare meticulously, pitch yourself confidently, ask relevant questions, and follow up effectively to land the machine learning opportunity that aligns with your ambitions.