Software Engineer (SC Cleared)- Azure Data & Devops

scrumconnect ltd
London
2 months ago
Applications closed

Related Jobs

View all jobs

Software Engineer

Software Engineer in AI

Software Engineer (we have office locations in Cambridge, Leeds & London)

Software Manager

Software Engineer - Maths, Stats, Machine Learning

Software Engineer II - Data Engineer, Python, SQL - Associate

About the Role

Scrumconnect Consulting is looking for a Software Engineer to work on a Strategic Data Platform with expertise in Azure Data Factory (ADF), Python, PySpark, Java, Terraform, and Azure DevOps. The ideal candidate will have experience in cloud-based data engineering, automation, and infrastructure provisioning while working in an Azure environment.

You will be responsible for developing scalable data pipelines, integrating cloud services, automating deployments, and supporting DevOps workflows.

Key Responsibilities:

  1. Software Engineer

    • Develop, test, and deploy scalable data pipelines using Azure Data Factory (ADF), Python, PySpark, and Java.
    • Implement data transformation, ETL/ELT workflows, and data integration solutions.
    • Optimize data flow and performance for cloud-based data processing.

  2. Cloud & Infrastructure Automation

    • Use Terraform to provision and manage Azure resources.
    • Implement infrastructure as code (IaC) best practices for automated cloud deployments.
    • Ensure efficient resource scaling and cost optimization.

  3. DevOps & CI/CD Automation

    • Collaborate with DevOps teams to build automated CI/CD pipelines in Azure DevOps.
    • Deploy and manage containerized applications using Docker & Kubernetes.
    • Monitor and troubleshoot build, deployment, and infrastructure issues.

  4. Performance Optimisation & Security

    • Optimize data pipeline performance in Azure.
    • Implement cloud security best practices and ensure compliance with data governance policies.
    • Troubleshoot data and infrastructure-related performance issues.

  5. Collaboration & Documentation

    • Work closely with data engineers, cloud architects, and DevOps teams to design solutions.
    • Participate in agile ceremonies, sprint planning, and technical discussions.
    • Maintain technical documentation and best practices.

Required Skills & Experience:

  1. Software Development & Data Engineering

    • Strong experience in: Python, PySpark, Java, Azure Data Factory (ADF).
    • Data Processing & Pipelines: ETL/ELT development, big data frameworks.
    • Cloud Services: Hands-on experience with Azure Data Lake, Synapse, and Azure Functions.

  2. Infrastructure as Code (IaC) & Automation

    • Terraform expertise for Azure resource provisioning.
    • Experience in cloud infrastructure automation and DevOps workflows.

  3. DevOps & CI/CD

    • Experience with Azure DevOps, Git, YAML pipelines.
    • Ability to work with Docker, Kubernetes, and containerized applications.

  4. Other Skills

    • Strong problem-solving and debugging skills.
    • Experience working in Agile/Scrum environments.
    • Excellent communication and collaboration skills.

Nice to Have:

  • Experience with Databricks, Apache Spark, or ML workloads.
  • Knowledge of security best practices in cloud environments.
  • Azure, Terraform, or DevOps-related certifications.

#J-18808-Ljbffr

Get the latest insights and jobs direct. Sign up for our newsletter.

By subscribing you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Jobs

Top 10 Mistakes Candidates Make When Applying for Machine-Learning Jobs—And How to Avoid Them

Landing a machine-learning job in the UK is competitive. Learn the 10 biggest mistakes applicants make—plus tested fixes, expert resources and live links that will help you secure your next ML role. Introduction From fintechs in London’s Square Mile to advanced-research hubs in Cambridge, demand for machine-learning talent is exploding. Job boards such as MachineLearningJobs.co.uk list new vacancies daily, and LinkedIn shows more than 10,000 open ML roles across the UK right now. Yet hiring managers still reject most CVs long before interview—often for avoidable errors. Below are the ten most common mistakes we see, each paired with a practical fix and a live resource link so you can dive deeper.

Careers Education

Top 10 Best UK Universities for Machine Learning Degrees (2025 Guide)

Explore ten UK universities that deliver world-class machine-learning degrees in 2025. Compare entry requirements, course content, research strength and industry links to find the programme that fits your goals. Machine learning (ML) has shifted from academic curiosity to the engine powering everything from personalised medicine to autonomous vehicles. UK universities have long been pioneers in the field, and their programmes now blend rigorous theory with hands-on practice on industrial-scale datasets. Below, we highlight ten institutions whose undergraduate or postgraduate pathways focus squarely on machine learning. League tables move each year, but these universities consistently excel in teaching, research and collaboration with industry.

Jobs

How to Write a Winning Cover Letter for Machine Learning Jobs: Proven 4-Paragraph Structure

Learn how to craft the perfect cover letter for machine learning jobs with this proven 4-paragraph structure. Ideal for entry-level candidates, career switchers, and professionals looking to advance in the machine learning sector. When applying for a machine learning job, your cover letter is a vital part of your application. Machine learning is an exciting and rapidly evolving field, and your cover letter offers the chance to demonstrate your technical expertise, passion for AI, and your ability to apply machine learning techniques to solve real-world problems. Writing a cover letter for machine learning roles may feel intimidating, but by following a clear structure, you can showcase your strengths effectively. Whether you're just entering the field, transitioning from another role, or looking to advance your career in machine learning, this article will guide you through a proven four-paragraph structure. We’ll provide practical tips and sample lines to help you create a compelling cover letter that catches the attention of hiring managers in the machine learning job market.

🍪 We use cookies to enhance your browsing experience on our website. By continuing to use this site, you consent to the use of cookies as described in our Cookie Policy. You can review our Cookie Policy for more information.