Be at the heart of actionFly remote-controlled drones into enemy territory to gather vital information.

Apply Now

Senior Data Engineer

Glencore
London
1 week ago
Create job alert

SUMMARY

Glencore, headquartered in Baar, Switzerland is one of the world's leading integrated producers and marketers of over 90 commodities that industries around the world need. Glencore has worldwide activities in the production, sourcing, processing, refining, transporting, storage, financing and supply of metals and minerals, energy products and agricultural products. We strive to be a reliable and competitive partner in the markets in which we serve and to support our customers and suppliers at each stage of their expansion and development.

Our customers around the world operate in a wide range of industries such as oil, automotive, power generation, steel production and food processing. They rely upon our established global network for supply of metals and minerals, and energy products.

These commodities either originate from Glencore's own production assets or are sourced from third parties.

The Group also provides financing, logistics and other supply chain services to producers and consumers of commodities. Our diversified operations comprise of over 150 mining and metallurgical, oil refining and distribution assets. Our industrial and marketing activities are supported by a global network of more than 90 offices located in over 30+ countries. We employ around 150,000 people, including contractors.

Glencore's Information Technology group develops and supports a range of IT applications and infrastructure to meet the needs of a fast-moving, international commodity trading business. The Oil Assets IT team is focused primarily on supporting Glencore's downstream operations in South Africa and Brazil. The team covers activities across the IT landscape but has a strong data engineering and analytics capability. Glencore uses a wide range of modern IT technologies in London and its other offices and also uses a number of leading commercial applications, providing a diverse set of opportunities and challenges.

Glencore is increasing its cloud footprint and adoption of AI technologies. This role will suit an experienced cloud professional who can bring new knowledge into the team and provide consultancy and guidance to local operations in South African and Brazil.

THE ROLE

  • As a Senior Data Engineer, you will be part of a team who design, build and maintain scalable, secure, and high-performance data solutions on Microsoft Azure. You will support mission-critical operations across oil trading, refining, and retail fuel distribution by building robust data pipelines, integrating real-time telemetry, and ensuring compliance with regional and international regulations.
  • This role sits within the Oil Assets IT team, reporting to the Data Engineering Team Lead.
  • This is a hybrid working position with a minimum of three days a week in the London office, including every other Thursday for team sprint meetings. This is subject to change and you may be required to come in up to five days in a week.
  • Occasional travel to affiliates in South Africa and Brazil.

PROFILE

  • Seasoned data engineer with 7+ years of experience, and a strong foundation in designing and managing cloud-native data pipelines, data models, and analytics solutions. Brings hands-on expertise in Azure-based technologies including Synapse Analytics, Data Factory, and Azure SQL, with forward-looking experience in Microsoft Fabric and Databricks. Operates in a fast-paced, business-facing environment, delivering high-quality solutions across multiple parallel workstreams. Self-driven and adaptable, with a deep understanding of the downstream oil and gas domain and a commitment to continuous learning and innovation.

RESPONSIBILITIES

  • Design and manage scalable data pipelines and ETL/ELT processes using Azure Data Factory, Synapse Analytics, and emerging platforms such as Microsoft Fabric and Databricks.
  • Collaborate with peers to create and maintain data models and databases in Azure SQL DB and Azure Data Lake.
  • Ensure data quality, lineage, and availability through rigorous validation, testing, and monitoring practices.
  • Integrate data from ETRM systems, business applications, refinery control systems, and retail station point-of-sale networks.
  • Implement real-time analytics solutions using Azure Event Hubs, Stream Analytics, and IoT Hub.
  • Liaise with multiple functional groups across Glencore Group, the Oil Department and the industrial assets, to provision and deploy infrastructure within Azure.
  • Implement data governance and security policies using Microsoft Purview and Azure RBAC.
  • Ensure data quality, lineage, and availability for business-critical applications.
  • Deliver high-velocity solutions supported by strong coding practices and automation.
  • Build and maintain multiple pipelines in parallel, managing context switching effectively.
  • Stay current with modern data technologies and trends, including AI/ML, and advise on their responsible adoption.
  • Operate independently, driving individual workstreams while contributing to team-wide initiatives

COMPETENCIES

Core Competencies

  • Bachelor's degree in Computer Science, Information Technology, or equivalent experience.
  • Data Engineering and Modeling: Strong grasp of core data modeling concepts and techniques, experience designing and managing ETL/ELT pipelines using Azure Data Factory and Synapse Analytics.
  • Programming Languages: Expert proficiency in SQL, Python, and PySpark for data transformation, validation, and analytics.
  • Cloud Technologies (Azure):
    • Azure Data Factory (ADF)
    • Azure Synapse Analytics
    • Azure Databricks
    • Azure Data Lake Storage Gen2
    • Azure SQL
    • Azure Functions
  • Version Control and CI/CD: Experience with Git/GitHub and CI/CD pipelines using GitHub Actions.

Desirable

  • Visualisation and Reporting: Skilled in building dashboards and visual narratives using Power BI.
  • Infrastructure and Automation: Familiarity with infrastructure-as-code tools such as Terraform; experience provisioning resources in Azure.
  • Data Governance and Security: Experience implementing data lineage, cataloguing, and access controls using Microsoft Purview and Azure RBAC.
  • Real-Time and Streaming Data: Exposure to Azure Event Hubs, IoT Hub, and Azure Stream Analytics for real-time data processing.
  • Advanced Tools and Platforms:
    • Apache Airflow for workflow orchestration
    • Microsoft Power Platform for low-code solutions
    • Azure DevOps for collaborative development and deployment
    • NoSQL/Big Data technologies (e.g., Cosmos DB, Hadoop, or similar)
  • Machine Learning Engineering: Understanding of ML workflows and integration into data pipelines.

Certifications

  • Microsoft Certified: Azure Data Engineer Associate
  • Microsoft Certified: Azure Solutions Architect Expert
  • Microsoft Certified: Azure Fundamentals (AZ-900)
  • Microsoft Certified: Azure AI Engineer Associate
  • Apache Spark Developer Certification


#J-18808-Ljbffr

Related Jobs

View all jobs

Senior Data Engineer

Senior Data Engineer

Senior Data Engineer

Senior Data Engineer

Senior Data Engineer

Senior Data Engineer

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Top 10 Skills in Machine Learning According to LinkedIn & Indeed Job Postings

Machine learning (ML) is at the forefront of innovation, powering systems in finance, healthcare, retail, logistics, and beyond in the UK. As organisations leverage ML for predictive analytics, automation, and intelligent systems, demand for skilled practitioners continues to grow. So, which skills are most in demand? Drawing on insights from LinkedIn and Indeed, this article outlines the Top 10 machine learning skills UK employers are looking for in 2025. You'll learn how to demonstrate these capabilities through your CV, interviews, and real-world projects.

The Future of Machine Learning Jobs: Careers That Don’t Exist Yet

Machine learning (ML) has become one of the most powerful forces reshaping the modern world. From voice assistants and recommendation engines to fraud detection and medical imaging, it underpins countless applications. ML is no longer confined to research labs—it powers business models, public services, and consumer technologies across the globe. In the UK, demand for machine learning professionals has risen dramatically. Organisations in finance, retail, healthcare, and defence are embedding ML into their operations. Start-ups in Cambridge, London, and Edinburgh are pioneering innovations, while government-backed initiatives aim to position the UK as a global AI leader. Salaries for ML engineers and researchers are among the highest in the tech sector. Yet despite its current importance, machine learning is only at the beginning of its journey. Advances in generative AI, quantum computing, robotics, and ethical governance will reshape the profession. Many of the most vital machine learning jobs of the next two decades don’t exist today. This article explores why new careers will emerge, the roles likely to appear, how today’s roles will evolve, why the UK is well positioned, and how professionals can prepare now.

Seasonal Hiring Peaks for Machine Learning Jobs: The Best Months to Apply & Why

The UK's machine learning sector has evolved into one of Europe's most intellectually stimulating and financially rewarding technology markets, with roles spanning from junior ML engineers to principal machine learning scientists and heads of artificial intelligence research. With machine learning positions commanding salaries from £32,000 for graduate ML engineers to £160,000+ for senior principal scientists, understanding when organisations actively recruit can dramatically accelerate your career progression in this pioneering and rapidly evolving field. Unlike traditional software engineering roles, machine learning hiring follows distinct patterns influenced by AI research cycles, model development timelines, and algorithmic innovation schedules. The sector's unique combination of mathematical rigour, computational complexity, and real-world application requirements creates predictable hiring windows that strategic professionals can leverage to advance their careers in developing tomorrow's intelligent systems. This comprehensive guide explores the optimal timing for machine learning job applications in the UK, examining how enterprise AI strategies, academic research cycles, and deep learning initiatives influence recruitment patterns, and why strategic timing can determine whether you join a groundbreaking AI research team or miss the opportunity to develop the next generation of machine learning algorithms.