Data Engineer (Spark, Kafka)

Windsor
2 weeks ago
Create job alert

We are partnered with a leading global consultancy that is searching for a contractor with the following skillset to work on a LONG-TERM contract within the ENERGY sector:

Role: Data Engineer (Spark, Kafka)

Location: Windsor

Style: Hybrid

Rate: up to £500 per day (inside IR35)

Duration: 6 months (initially – view to extend)

Key responsibilities:

  • Design, implement, and manage Kafka-based data pipelines and messaging solutions to support critical business operations and enable real-time data processing.

  • Configure, deploy, and maintain Kafka clusters, ensuring high availability and scalability to maximize uptime and support business growth.

  • Monitor Kafka performance and troubleshoot issues to minimize downtime and ensure uninterrupted data flow, enhancing decision-making and operational efficiency.

  • Collaborate with development teams to integrate Kafka into applications and services.

  • Develop and maintain Kafka connectors such as JDBC, MongoDB, and S3 connectors, along with topics and schemas, to streamline data ingestion from databases,

  • NoSQL data stores, and cloud storage, enabling faster data insights.

  • Implement security measures to protect Kafka clusters and data streams, safeguarding sensitive information and maintaining regulatory compliance

    Key Skills:

  • Design, build, and maintain reliable, scalable data pipelines. Data Integration, Data Security and Compliance

  • Monitor and manage the performance of data systems and troubleshoot issues.

  • Strong knowledge of data engineering tools and technologies (e.g. SQL, ETL, data warehousing), Experience in tools like Azure ADF, Apache Kafka, Apache Spark SQL, Proficiency in programming languages such as Python, PySpark

  • Good written and verbal communication skill

  • Experience in managing business stakeholders for requirement clarification

    If you are interested and have the relevant experience, please apply promptly and we will contact you to discuss it further.

    Yilmaz Moore

    Senior Delivery Consultant

    London | Bristol | Amsterdam

Related Jobs

View all jobs

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Get the latest insights and jobs direct. Sign up for our newsletter.

By subscribing you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Job-Hunting During Economic Uncertainty: Machine Learning Edition

Machine learning (ML) has firmly established itself as a crucial part of modern technology, powering everything from personalised recommendations and fraud detection to advanced robotics and predictive maintenance. Both start-ups and multinational corporations depend on machine learning engineers and data experts to gain a competitive edge via data-driven insights and automation. However, even this high-demand sector can experience a downturn when broader economic forces—such as global recessions, wavering investor confidence, or unforeseen financial events—lead to more selective hiring, stricter budgets, and lengthier recruitment cycles. For ML professionals, the result can be fewer available positions, more rivals applying for each role, or narrower project scopes. Nevertheless, the paradox is that organisations still require skilled ML practitioners to optimise operations, explore new revenue channels, and cope with fast-changing market conditions. This guide aims to help you adjust your job-hunting tactics to these challenges, so you can still secure a fulfilling position despite uncertain economic headwinds. We will cover: How market volatility influences machine learning recruitment and your subsequent steps. Effective strategies to distinguish yourself when the field becomes more discerning. Ways to showcase your technical and interpersonal skills with tangible business impact. Methods for maintaining morale and momentum throughout potentially protracted hiring processes. How www.machinelearningjobs.co.uk can direct you towards the right opportunities in machine learning. By sharpening your professional profile, aligning your abilities with in-demand areas, and engaging with a focused ML community, you can position yourself for success—even in challenging financial conditions.

How to Achieve Work-Life Balance in Machine Learning Jobs: Realistic Strategies and Mental Health Tips

Machine Learning (ML) has become a cornerstone of modern innovation, powering everything from personalised recommendation engines and chatbots to autonomous vehicles and advanced data analytics. With numerous industries integrating ML into their core operations, the demand for skilled professionals—such as ML engineers, research scientists, and data strategists—continues to surge. High salaries, cutting-edge projects, and rapid professional growth attract talent in droves, creating a vibrant yet intensely competitive sector. But the dynamism of this field can cut both ways. Along with fulfilling opportunities comes the pressure of tight deadlines, complex problem-solving, continuous learning curves, and high-stakes project deliverables. It’s a setting where many professionals ask themselves, “Is true work-life balance even possible?” When new algorithms emerge daily and stakeholder expectations soar, the line between healthy dedication and perpetual overwork can become alarmingly thin. This comprehensive guide aims to shed light on how to achieve a healthy work-life balance in Machine Learning roles. We’ll discuss the distinctive pressures ML professionals face, realistic approaches to managing workloads, strategies for safeguarding mental health, and how boundary-setting can be the difference between sustained career growth and burnout. Whether you’re just getting started or have been at the forefront of ML for years, these insights will empower you to excel without sacrificing your well-being.

Transitioning from Academia to the Machine Learning Industry: How PhDs and Researchers Can Thrive in Commercial ML Settings

Machine learning (ML) has rapidly evolved from an academic discipline into a cornerstone of commercial innovation. From personalising online content to accelerating drug discovery, machine learning technologies permeate nearly every sector, creating exciting career avenues for talented researchers. If you’re a PhD or academic scientist thinking about leaping into this dynamic field, you’re not alone. Companies are eager to recruit professionals with a strong foundation in algorithms, statistical methods, and domain-specific knowledge to build the intelligent products of tomorrow. This article explores the essential steps academics can take to transition into industry roles in machine learning. We’ll discuss the differences between academic and commercial research, the skill sets most in demand, and how to optimise your CV and interview strategy. You’ll also find tips on networking, developing a commercial mindset, and navigating common challenges as you pivot your career from the halls of academia to the ML-driven tech sector.