Job Opportunity: Data Engineer (Databricks Specialist)

Location: Manchester |Employment Type: Full-time.

Are you a passionate Data & AI expert with a strong Databricks background? A leading consulting firm is seeking a person skilled inData Engineeringto join their growing team. In this senior role, you’ll work with clients to design and implement scalable, high-performance Databricks-based platforms that drive impactful business outcomes

Key Responsibilities

As a Data Engineer, you will be responsible for designing, implementing, and optimizing data workflows that drive critical insights across the business. You'll work with cutting-edge technologies to build scalable, high-performance data solutions that meet the evolving needs of our clients.

• Data Pipeline Development:Design and implement robust ETL/ELT workflows usingDatabricksto process and transform large datasets efficiently, enabling real-time insights and decision-making.
• Data Integration:Develop scalable data ingestion processes from multiple sources and integrate them intoDelta Lakeordata lakesfor unified storage and analysis.
• Collaboration:Partner closely with data architects, data scientists, and business analysts to understand requirements and deliver data solutions that align with business objectives and enhance analytics capabilities.
• Performance Optimization:OptimizeDatabricksworkflows for performance, scalability, and cost efficiency, includingSpark tuningandcluster management.
• Cloud Integration:Build cloud-native data engineering solutions on leading cloud platforms likeAzure, AWS, or GCPto ensure seamless, secure, and efficient data management.
• Governance & Quality:Implement and maintain data governance best practices, ensuring data consistency, quality, and security across the organization.
• Automation:Automate repetitive tasks, develop robust CI/CD pipelines, and streamline data workflows to improve operational efficiency and reduce manual effort.
• Documentation:Create and maintain clear, detailed documentation of processes, workflows, and architectures to ensure knowledge sharing and promote best practices within the team.

Skills & Qualifications

Required:

• Experience:
• 3+ years of experience indata engineering, with hands-on expertise inDatabricks.
• Proven ability to deliver large-scale data pipelines in consulting or enterprise environments.
• Technical Expertise:
• Proficiency inDatabricks, Delta Lake, andApache Spark.
• Strong programming skills inPythonandSQL, with optional experience inScala.
• Experience with cloud platforms such asAzure(preferred),AWS, orGCP.
• Strong understanding ofrelational and non-relational databases,data lakes, andreal-time data processing.
• Soft Skills:
• Strong problem-solving skills and a proactive approach to addressing complex challenges.
• Excellent communication skills and the ability to collaborate effectively with cross-functional teams.

Preferred:

• Experience with tools likeMLflowor otherAI/ML frameworks.
• Familiarity with data visualization tools such asPower BIorTableau.
• CertificationinDatabricksor cloud platforms (e.g.,Azure Data Engineer Associate).
• Experience inautomationandCI/CD practicesfor data engineering pipelines.

Why Join Us?

• Challenging Projects:Work on diverse and impactful projects for clients across various industries, contributing to real-world data solutions.
• Growth Opportunities:Gain exposure to cutting-edge technologies and continue developing your technical expertise within a fast-paced, innovative environment.
• Supportive Culture:Join a collaborative team that values creativity, innovation, and continuous learning.
• Competitive Package:We offer an attractive salary, comprehensive benefits, and flexible working options, including remote work opportunities.

If you're passionate aboutdata engineering, enjoy tackling complex challenges, and want to be part of a forward-thinking team, we want to hear from you!