Data Engineer - HIRING ASAP

Start date:ASAP
Duration:12 Months
Location:Hybrid Working in London office

Rate:£700-£768 per day inside IR35

Summary

The International Engineering team is responsible for building and running the UK and Canadian apps. As a Data Engineer, you will own engineering efforts to provide partner teams within the organisation with data sources that build our AI/ML user experience and help run our client robustly and at scale.

Responsibilities:

• Build and maintain scalable data pipelines for BigQuery using Cloud Composer and Airflow;
• Provide observability and monitoring using Monte Carlo and Looker as well as operational tools such as NewRelic and Splunk, driving reliability, data quality, data quality and robustness;
• Build critical data quality checks to ensure that the data we process and provide to partner teams meets security and compliance requirements;
• Participate in code and technical design reviews;
• Scope and plan work together with product engineering, data science and the business; support other teams for all tasks related to data tracking, collection, ingestion, quality monitoring and alerting.

Skills:

• 3+ years of experience managing batching and/or streaming data pipelines and infrastructure using technologies such as GCP, Dataflow/Apache Beam, Dataproc, Spark/Flink, dbt;
• Working knowledge of Python, SQL and shell scripting;
• Experience with Docker/k8s and "infrastructure as a code" tools such as Terraform, Salt or similar;
• 3+ years of significant professional experience developing applications written in a modern programming environment - preferably Scala/Java and/or Python;
• Professional experience across the entire software development life cycle, from gathering requirements to production deployment using CI & CD;
• Confident at articulating complex technical content to peers and partners across the organization via Technical Design Documents.

Bonus Skills:

• Experience optimising complex OLAP queries in a DW system such BigQuery, SnowFlake, Druid, etc;
• Experience with Looker;
• Experience with the MLOps life cycle;
• DevOps experience with Google Cloud Platform, AWS, or Azure;
• Data quality experience especially using & improving internal tools to automatically detect issues.