Data Engineer

The Job 

 As a Data Engineer, you will be working in the Red EngineBusinessInsightsteam, helping to build out the existing data and analytics platform. This role is placed within a small team, allowing the successful candidateto work closely with the senior engineersin implementing bespoke features and enhancements to our data platform using the latest technology. In this role you willassistthesenior and leadengineers aswellassistindrivingthe development of the data platform.This willincludingmeeting with key business stakeholders to gather technical requirements andconvertingtheserequirements into technical data solutionsunder the guidance of the lead engineerwithin the Data & Analytics platform. 

Key responsibilities will include:  

•   Developing andmaintainingdata pipelines to orchestrate the ingestion of data from disparate source systems into a centralised data analytics platform. 

• Work with the Lead Engineer to designing and implementdata engineering solutions using T-SQL, Python,PySparkand DBT in the Azure cloud environment. 

• Working with Data Analysts in promoting business logic into the analytics platform, to support business reports and dashboards. 

• Maintaining andleveragingCI/CD deployment pipelines to promote application code into higher tier environments. 

• Maintaining and making additions to the technical documentation suite in Azure DevOps. 

To be successful in this role,you’ll:  

Essential Skills: 

• Demonstrated experience developing of ETL/ELT ingestion pipelines to handle data movement,transformationand visualisations from structured and unstructured sources.  

• Experience with the Azure Platform including: 

• Data Ingestion: Azure Data Factory (ADF),  Databricks, Logic Apps and Function App 

• Data Storage: ADLS, SQL Server and UnityCatalog(Medallion Architecture) 

• Data Analysis: Databricks Notebooks, SQL Queries, Data visualisation 

• Strong understanding of the Databricks Platform including managing,developingand deploying workflows,jobsand notebooks.  

• Proven experience in modelling data in a Data warehouse using an Inmon or Kimball approaches.   

• Database development experience in SQL Server including Store Procedures and T-SQL or similar enterprise database toolset. 

• Experience working in an Agile software development framework. 

• Experience working with Data Build Tool (DBT) including building data models, contracts, tests, validation, and transformations.  

• Experience working with modern data distributed file formats (i.e., Parquet, Delta, Iceberg, Hudi). 

• Demonstrated experience with building data ingestion pipelines from REST API data sources. 

• Strong ability to produce technical documentation that can be understood for both technical and non-technical audiences.  

Desirable Skills: 

• Experience withIaCsolutions using Terraform,Pulumior similar tools. 

• Experience with modern CI/CD DevOps frameworks.  

• Experience in developing data visualisations usingPowerBi, Tableau or similar tools