Job Title: Data lakes, Hadoop Developer
Location: London
Work model: Hybrid
Key Responsibilities
Design, build, and manage scalable Data Lakes to support large-scale data processing and analytics.
Develop and maintain Big Data solutions using the Hadoop ecosystem (HDFS, Hive, HBase, Spark, Pig, MapReduce, etc.).
Implement data ingestion pipelines and workflows for structured, semi-structured, and unstructured data.
Optimize data processing and storage to ensure high performance and low latency.
Collaborate with data engineers, analysts, and scientists to provide robust and efficient data access solutions.
Monitor and troubleshoot data pipelines and applications to ensure reliability and accuracy.
Implement data security, governance, and compliance practices across the data lake and Hadoop systems.
Stay updated with emerging Big Data technologies and recommend tools or approaches to enhance the data platform.
Required Skills and Qualifications
Proven experience with Hadoop ecosystems, including HDFS, YARN, Hive, HBase, MapReduce, and Spark.
Expertise in Data Lake architectures and principles.
Proficiency in programming languages such as Python, Java, or Scala for Big Data processing.
Hands-on experience with ETL tools, data ingestion frameworks, and workflow schedulers (e.g., Apache Nifi, Airflow).
Strong knowledge of cloud platforms such as AWS (S3, EMR, Glue), Azure (Data Lake Storage, Synapse), or Google Cloud (BigQuery, Dataflow).
Familiarity with query languages like SQL, HiveQL, or Presto.
Understanding of data governance, security, and compliance (e.g., GDPR, HIPAA).
Excellent problem-solving skills and the ability to debug and resolve issues in distributed systems.
Preferred Qualifications
Experience with Kubernetes, Docker, or other containerization technologies for Big Data deployments.
Knowledge of streaming frameworks like Kafka, Flume, or Spark Streaming.
Hands-on experience in implementing machine learning workflows in a Big Data environment.
Certifications in Big Data technologies or cloud platforms (e.g., AWS Big Data Specialty, Cloudera Certified Professional).
Familiarity with tools like Databricks, Delta Lake, or Snowflake.
The latest news, articles, and resources, sent to your inbox weekly.
Job Title: Data lakes, Hadoop DeveloperLocation: LondonWork model: HybridKey ResponsibilitiesDesign, build, and manage scalable Data Lakes to support large-scale data processing and analytics.Develop and maintain Big Data solutions using the Hadoop ecosystem (HDFS, Hive, HBase, Spark, Pig, MapReduce, etc.).Implement data ingestion pipelines and workflows for structured, semi-structured, and unstructured data.Optimize...
N Consulting Ltd London
Job Title:Python Developer with PySparkLocation:NorthomptonJob Type:ContractAbout the Role:We are seeking a skilled Python Developer with expertise in PySpark to join our dynamic team. The ideal candidate will have strong experience in building and optimizing large-scale data processing pipelines and a deep understanding of distributed data systems. You will play a...
N Consulting Ltd
DescriptionJOB TITLE:Data Engineer (GCP/Kafka)SALARY:Bristol £68,202-£75,780 per annumLondon £78,849-87,610 per annumLOCATION(S):Bristol or LondonHOURS:Full-time – 35 hours per weekWORKING PATTERN:Our work style is hybrid, which involves spending at least two days per week currently, or 40% of our time, at our Bristol or London office.Please note that we will not be able...
Lloyds Banking Group London
Role SummaryAubay is seeking an experiencedData Solution Architectwith a strong background in financial services to join our team. This pivotal role is part of a transformational journey, where you will shape the data architecture landscape to meet evolving business needs. You will lead the design of large-scale data solutions, ensuring...
Aubay UK Newcastle upon Tyne
Senior Lead Data EngineerRemote/Reading/ExeterThe OpportunityAre you ready to make a significant impact on the future of data and engineering? We have an excitingSenior Leadposition in ourData Engineeringteam, perfect for experienced, hands-on professionals in leading and innovating. As a 100% data-driven company, we pride ourselves on employing the best engineering practices...
Landmark Information
Salary: £85,000 to £100,000 depending on experienceLocation: Hybrid working in either Bristol, Manchester, London.Are you a skilled Data Engineer who can deliver Data Platforms?Are you familiar of working on agile delivery-led projects?Data Engineers at Made Tech help build data platforms to enable predictive government services that make a positive impact...
Made Tech Bristol
