Data Engineer

The Job 

 As a Data Engineer, you will be working in the Red Engine Business Insightsteam, helping to build out the existing data and analytics platform. This role is placed within a small team, allowing the successful candidate to work closely with the senior engineers in implementing bespoke features and enhancements to our data platform using the latest technology. In this role you will assist the senior and leadengineers as well assist in driving the development of the data platform. This will including meeting with key business stakeholders to gather technical requirements and converting these requirements into technical data solutions under the guidance of the lead engineer within the Data & Analytics platform. 

Key responsibilities will include:  

•   Developing and maintaining data pipelines to orchestrate the ingestion of data from disparate source systems into a centralised data analytics platform. 

• Work with the Lead Engineer to designing and implementdata engineering solutions using T-SQL, Python, PySpark and DBT in the Azure cloud environment. 

• Working with Data Analysts in promoting business logic into the analytics platform, to support business reports and dashboards. 

• Maintaining and leveraging CI/CD deployment pipelines to promote application code into higher tier environments. 

• Maintaining and making additions to the technical documentation suite in Azure DevOps. 

To be successful in this role, you’ll:  

Essential Skills: 

• Demonstrated experience developing of ETL/ELT ingestion pipelines to handle data movement, transformation and visualisations from structured and unstructured sources.  

• Experience with the Azure Platform including: 

• Data Ingestion: Azure Data Factory (ADF),  Databricks, Logic Apps and Function App 

• Data Storage: ADLS, SQL Server and Unity Catalog (Medallion Architecture) 

• Data Analysis: Databricks Notebooks, SQL Queries, Data visualisation 

• Strong understanding of the Databricks Platform including managing, developing and deploying workflows, jobs and notebooks.  

• Proven experience in modelling data in a Data warehouse using an Inmon or Kimball approaches.   

• Database development experience in SQL Server including Store Procedures and T-SQL or similar enterprise database toolset. 

• Experience working in an Agile software development framework. 

• Experience working with Data Build Tool (DBT) including building data models, contracts, tests, validation, and transformations.  

• Experience working with modern data distributed file formats (i.e., Parquet, Delta, Iceberg, Hudi). 

• Demonstrated experience with building data ingestion pipelines from REST API data sources. 

• Strong ability to produce technical documentation that can be understood for both technical and non-technical audiences.  

Desirable Skills: 

• Experience with IaC solutions using Terraform, Pulumi or similar tools. 

• Experience with modern CI/CD DevOps frameworks.  

• Experience in developing data visualisations using PowerBi, Tableau or similar tools