Lead Data Engineer (AWS)

Permanent

Up to £108,000 per annum

Remote

Digital Skills are working with a growing tech company that builds and provides tools and services to a range of end clients. The company specializes in designing, building and continuously improving data-enabling products within highly governed sectors. We are currently recruiting for a Data Engineer who will assist in the development of a secure batch/Real Time data platform for a number of clients leveraging the latest commercial and open-source technologies.

This involves getting data from a variety of different sources, getting it into the right format, assuring alignment to target models, and ensuring that data can be made available to downstream consumers.

Responsibilities

• Code, test, and document new or modified data pipelines that meet functional/non- functional business requirements
• Conduct logical and physical database design
• Expand and grows data platform capabilities to solve new data and analytics problems
• Conduct data analysis, identifying feasible solutions and enhancements to data processing challenges
• Ensure that data models are consistent with the data architecture (eg entity names, relationships and definitions)
• Perform root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement

Qualifications

• Data pipeline development using data processing technologies and frameworks
• Agile or other rapid application development methods
• Datamodelling and understanding of different data structures and their benefits and limitations under particular use cases
• Experience in Public Cloud services, such as AWS. Practical experience with core services such as EC2, RDS, Lambda, Athena & Glue would be even better!
• Configuring and tuning Relational and NoSQL databases, including both query processing and query planning, or other data processing infrastructure
• Programming or Scripting languages, such as Python
• Exposure to Kafka would be ideal.
• Test Driven Development with appropriate tools and frameworks

Please don't hesitate to apply if this role seems right for you.