Senior Data Engineer

Senior Data Engineer

Bristol

6-month contract

Paying up to £80p/h (Outside IR35)

*** Please note that due to the nature of the work you must hold or be eligible to obtain DV level security clearance ***

Responsibilities:

• Design, develop, and maintain secure and scalable data pipelines using the Elastic Stack (Elasticsearch, Logstash, Kibana) and Apache NiFi.
• Implement data ingestion, transformation, and integration processes, ensuring data quality and security.
• Collaborate with data architects and security teams to ensure compliance with security policies and data governance standards.
• Manage and monitor large-scale data flows in real-time, ensuring system performance, reliability, and data integrity.
• Develop robust data models to support analytics and reporting within secure environments.
• Perform troubleshooting, debugging, and performance tuning of data pipelines and the Elastic Stack.
• Build dashboards and visualizations in Kibana to enable data-driven decision-making.
• Ensure high availability and disaster recovery for data systems, implementing appropriate backup and replication strategies.
• Document data architecture, workflows, and security protocols to ensure smooth operational handover and audit readiness.

Experience required:

• Experience working as a Data Engineer in secure or regulated environments.
• Expertise in the Elastic Stack (Elasticsearch, Logstash, Kibana) for data ingestion, transformation, indexing, and visualization.
• Strong experience with Apache NiFi for building and managing complex data flows and integration processes.
• Familiarity with containerization and orchestration tools such as Docker and Kubernetes.
• Experience with monitoring and alerting tools such as Prometheus, Grafana, or ELK for data infrastructure
• Knowledge of security practices for handling sensitive data, including encryption, anonymization, and access control.
• Familiarity with data governance, data quality management, and compliance standards in secure environments.
• Experience in managing large-scale, real-time data pipelines and ensuring their performance and reliability.