Databricks Unity Catalog Engineer
The Company:
At Amber Labs, we are a cutting-edge UK and European technology consultancy that prioritises empowering autonomy, promoting experimentation, and facilitating rapid learning to provide exceptional value to our clients. Our company culture is centred around collaboration, where all colleagues, regardless of their role, work together to minimise risk and shorten delivery times. Our team consists of highly-skilled cross-functional consultants, analysts, and support staff.
We are looking for an experienced Databricks Unity Catalog Engineer to join our data engineering team. The ideal candidate will be responsible for designing, implementing, and managing the Databricks Unity Catalog to ensure seamless governance and secure access to data across multiple environments. You will work closely with data architects, data engineers, and cloud administrators to implement governance policies, catalog management, and provide secure data access across the organization's Databricks ecosystem.
As a Unity Catalog Engineer, you will play a key role in enhancing data governance, ensuring compliance, and implementing best practices for managing large-scale data lakes and enterprise data in a multi-cloud environment.
Key Responsibilities:
- Implement and manage Databricks Unity Catalog, ensuring effective data governance and secure access to sensitive datasets.
- Design and establish data governance policies, including role-based access control (RBAC), data lineage, data masking, and metadata management within Unity Catalog.
- Collaborate with data engineering and cloud teams to integrate Unity Catalog with data lakes, ETL pipelines, and data warehouse architectures across AWS, Azure, or GCP environments.
- Monitor and optimize the performance of Unity Catalog, ensuring that data classification and access policies are applied and enforced correctly.
- Work closely with security teams to ensure data governance and security policies align with compliance standards such as GDPR, CCPA, HIPAA, or industry-specific regulations.
- Establish best practices for data discovery and cataloging, enabling data scientists and analysts to quickly access the data they need.
- Implement data lineage tracking within Unity Catalog to ensure transparency and traceability of data usage across the organization.
- Manage access control configurations and ensure smooth integration with existing IAM solutions such as Azure AD, AWS IAM, or other identity providers.
- Develop and maintain detailed technical documentation on the configuration and management of Unity Catalog, including processes for onboarding new datasets.
- Troubleshoot and resolve issues related to Unity Catalog configurations, data access policies, and integration with other Databricks features such as Delta Lake and MLflow.
- Provide training and support to data engineering and analytics teams on utilizing Unity Catalog for their data governance and access needs.
Essential Skills and Qualifications:
- Bachelor's or Master's degree in Computer Science, Information Systems, Data Engineering, or related field.
- Proven experience with Databricks and hands-on knowledge of implementing Databricks Unity Catalog.
- Strong experience with cloud platforms (Azure, AWS, GCP), with a focus on Databricks integration.
- In-depth understanding of data governance principles, including data access control, data lineage, and metadata management.
- Experience with role-based access control (RBAC), IAM, and cloud security policies to manage secure data access.
- Proficiency in working with large-scale data lakes and Delta Lake architectures.
- Solid understanding of ETL/ELT processes, data ingestion, and transformation workflows in cloud environments.
- Hands-on experience in working with Python, SQL, and other scripting languages for data management and cataloging tasks.
- Knowledge of data governance and compliance regulations such as GDPR, CCPA, and HIPAA.
- Experience with Databricks notebooks for developing data pipelines and workflows.
- Strong problem-solving skills and experience troubleshooting data access issues, performance bottlenecks, and catalog management challenges.
Desirable Skills:
- Experience with other data cataloging and governance tools, such as AWS Glue, Apache Atlas, or Azure Purview.
- Certification in Databricks or relevant cloud certifications (e.g., AWS Certified Solutions Architect, Azure Data Engineer).
- Knowledge of machine learning workflows and experience in supporting data science teams with managed datasets.
- Experience in multi-cloud environments and data orchestration across Azure, AWS, and GCP.
Benefits:
Join a rapidly expanding start-up where personal growth is a part of our DNA.
Benefit from a flexible work environment focused on deliverable outcomes.
Receive private medical insurance through Aviva.
Enjoy the benefits of a company pension plan through Nest.
25 days of annual leave plus UK bank holidays.
Access Perkbox, a global employee rewards platform offering discounts, perks, and wellness resources.
Participate in a generous employee referral program.
A highly collaborative and collegial environment with opportunities for career advancement.
Be encouraged to take bold steps and embrace a mindset of experimentation.
Choose your preferred device, PC or Mac.
Diversity & Inclusion:
Here at Amber Labs, we are dedicated to fostering an inclusive and equitable workplace for all. Our commitment to diversity, equality, and inclusion includes:
Valuing the unique experiences, perspectives, and backgrounds of all employees and creating an environment where everyone feels welcomed, respected, and valued.
Prohibiting all forms of harassment, bullying, discrimination, and victimisation and promoting a culture of dignity and respect for all.
Educating all new hires on our Diversity and Inclusion policies and ensuring they are aware of their rights and responsibilities to create a safe and inclusive workplace.
By taking these steps, we are dedicated to building a workplace that reflects and celebrates the diversity of our employees and communities.
At Amber Labs, all employees are required to meet the Baseline Personnel Security Standard (BPSS). Please be advised that, at this time, we are unable to consider candidates who require sponsorship or hold a visa of any type.
What Happens Next?
Our Talent Acquisition Team will be in touch to advise you on the next steps. We have a two-stage interview process for most of our consultants. In certain cases, we may include a third and final stage, which is a conversation with the company Partners. This will only be considered if deemed necessary.
