Data Engineer

Position: Data Engineer

Overview of the Role
An established organisation is currently undergoing a data transformation and is seeking a skilled Data Engineer to join its growing data team. This role plays a key part in building and deploying modern data solutions based on Azure Databricks, enabling faster and more informed business decisions.

You'll work hands-on with Azure Databricks, Azure Data Factory, Delta Lake, and Power BI to design scalable data pipelines, implement efficient data models, and ensure high-quality data delivery. This is a great opportunity to shape the future of data within the organisation while working with advanced cloud technologies.

Key Responsibilities and Deliverables

• Design, develop, and optimise end-to-end data pipelines (batch & streaming) using Azure Databricks, Spark, and Delta Lake.

• Implement Medallion Architecture to structure raw, enriched, and curated data layers efficiently.

• Build scalable ETL/ELT processes with Azure Data Factory and PySpark.

• Support data governance initiatives using tools like Azure Purview and Unity Catalog for metadata management, lineage, and access control.

• Ensure consistency, accuracy, and reliability across data pipelines.

• Collaborate with analysts to validate and refine datasets for reporting.

• Apply DevOps and CI/CD best practices (Git, Azure DevOps) for automated testing and deployment.

• Optimise Spark jobs, Delta Lake tables, and SQL queries for performance and cost-effectiveness.

• Troubleshoot and proactively resolve data pipeline issues.

• Partner with data architects, analysts, and business teams to deliver end-to-end data solutions.

• Stay current with emerging data technologies (e.g., Kafka/Event Hubs for streaming, Knowledge Graphs).

• Promote best practices in data engineering across the team.

Skills & Experience

• Hands-on experience with Azure Databricks, Delta Lake, Data Factory, and Synapse.

• Strong understanding of Lakehouse architecture and medallion design patterns.

• Proficient in Python, PySpark, and SQL, with advanced query optimisation skills.

• Proven experience building scalable ETL pipelines and managing data transformations.

• Familiarity with data quality frameworks and monitoring tools.

• Experience working with Git, CI/CD pipelines, and in Agile environments.

• Ability to write clean, maintainable, and well-documented code.

• Exposure to Power BI or similar data visualisation tools.

• Knowledge of IoT data pipelines is a plus.

#J-18808-Ljbffr