Big Data Lead

FalconSmartIT
London
2 weeks ago
Create job alert

Job TItle: Big Data Lead

Job Type: Contract

Job Location: Wimbledon , UK


Job Description:

For this role, senior experience of Data Engineering and building automated data pipelines on IBM Datastage & DB2, AWS and Databricks from source to operational databases through to curation layer is expected using the latest cloud modern technologies where experience of delivering complex pipelines will be significantly valuable to how D&G maintain and deliver world class data pipelines.

Knowledge in the following areas essential:

Data Engineering Experience:

  • Databricks:Expertise in managing and scaling Databricks environments for ETL, data science, and analytics use cases.
  • AWS Cloud:Extensive experience with AWS services such as S3, Glue, Lambda, RDS, and IAM.
  • IBM Skills:DB2, Datastage, Tivoli Workload Scheduler, Urban Code
  • Programming Languages:Proficiency in Python, SQL.
  • Data Warehousing & ETL:Experience with modern ETL frameworks and data warehousing techniques.
  • DevOps & CI/CD:Familiarity with DevOps practices for data engineering, including infrastructure-as-code (e.g., Terraform, CloudFormation), CI/CD pipelines, and monitoring (e.g., CloudWatch, Datadog).
  • Familiarity with big data technologies like Apache Spark, Hadoop, or similar.
  • Test automation skills
  • ETL/ELT tools and creating common data sets across on-prem (IBMDatastage ETL) and cloud data stores
  • Leadership & Strategy:Lead Data Engineering team(s) in designing, developing, and maintaining highly scalable and performant data infrastructures.
  • Customer Data Platform Development:Architect and manage our data platforms using IBM (legacy platform) & Databricks on AWS technologies (e.g., S3, Lambda, Glacier, Glue, EventBridge, RDS) to support real-time and batch data processing needs.
  • Data Governance & Best Practices:Implement best practices for data governance, security, and data quality across our data platform. Ensure data is well-documented, accessible, and meets compliance standards.
  • Pipeline Automation & Optimisation:Drive the automation of data pipelines and workflows to improve efficiency and reliability.
  • Team Management:Mentor and grow a team of data engineers, ensuring alignment with business goals, delivery timelines, and technical standards.
  • Cross Company Collaboration:Work closely with all levels of business stakeholder including data scientists, finance analysts, MI and cross-functional teams to ensure seamless data access and integration with various tools and systems.
  • Cloud Management:Lead efforts to integrate and scale cloud data services on AWS, optimising costs and ensuring the resilience of the platform.
  • Performance Monitoring:Establish monitoring and alerting solutions to ensure the high performance and availability of data pipelines and systems to ensure no impact to downstream consumers.

Related Jobs

View all jobs

Lead Data Engineer

Lead Data Engineer

Lead Data Engineer

Clinical Data Enablement, Data Aggregation, Lead

Data Engineer Lead

Senior Data Engineer

Get the latest insights and jobs direct. Sign up for our newsletter.

By subscribing you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Tips for Staying Inspired: How Machine Learning Pros Fuel Creativity and Innovation

Machine learning (ML) continues to reshape industries—from personalised e-commerce recommendations and autonomous vehicles to advanced healthcare diagnostics and predictive maintenance in manufacturing. Yet behind every revolutionary model lies a challenging and sometimes repetitive process: data cleaning, hyperparameter tuning, infrastructure management, stakeholder communications, and constant performance monitoring. It’s no wonder many ML professionals can experience creative fatigue or get stuck in the daily grind. So, how do machine learning experts keep their spark alive and continually generate fresh ideas? Below, you’ll find ten actionable strategies that successful ML engineers, data scientists, and research scientists use to stay innovative and push boundaries. Whether you’re an experienced practitioner or just breaking into the field, these tips can help you fuel creativity and discover new angles for solving complex problems.

Top 10 Machine Learning Career Myths Debunked: Key Facts for Aspiring Professionals

Machine learning (ML) has become one of the hottest fields in technology—touching everything from recommendation engines and self-driving cars to language translation and healthcare diagnostics. The immense potential of ML, combined with attractive compensation packages and high-profile success stories, has spurred countless professionals and students to explore this career path. Yet, despite the boom in demand and innovation, machine learning is not exempt from myths and misconceptions. At MachineLearningJobs.co.uk, we’ve had front-row seats to the real-life career journeys and hiring needs in this field. We see, time and again, that outdated assumptions—like needing a PhD from a top university or that ML is purely about deep neural networks—can mislead new entrants and even deter seasoned professionals from making a successful transition. If you’re curious about a career in machine learning or looking to take your existing ML expertise to the next level, this article is for you. Below, we debunk 10 of the most persistent myths about machine learning careers and offer a clear-eyed view of the essential skills, opportunities, and realistic paths forward. By the end, you’ll be better equipped to make informed decisions about your future in this dynamic and rewarding domain.

Global vs. Local: Comparing the UK Machine Learning Job Market to International Landscapes

How to evaluate opportunities, salaries, and work culture in machine learning across the UK, the US, Europe, and Asia Machine learning (ML) has rapidly transcended the research labs of academia to become a foundational pillar of modern technology. From recommendation engines and autonomous vehicles to fraud detection and personalised healthcare, machine learning techniques are increasingly ubiquitous, transforming how organisations operate. This surge in applications has fuelled an extraordinary global demand for ML professionals—data scientists, ML engineers, research scientists, and more. In this article, we’ll examine how the UK machine learning job market compares to prominent international hubs, including the United States, Europe, and Asia. We’ll explore hiring trends, salary ranges, workplace cultures, and the nuances of remote and overseas roles. Whether you’re a fresh graduate aiming to break into the field, a software engineer with an ML specialisation, or a seasoned professional seeking your next challenge, understanding the global ML landscape is essential for making an informed career move. By the end of this overview, you’ll be equipped with insights into which regions offer the best blend of salaries, work-life balance, and cutting-edge projects—plus practical tips on how to succeed in a domain that’s constantly evolving. Let’s dive in.