Jobs

Data Engineer


Job details
  • Mirai Talent
  • London
  • 4 months ago

Data Engineer

We are on the lookout for a Data Engineer to join a dynamic, forward-thinking company. You’ll will work in various settings to build systems that collect, manage, and convert raw data into usable information for data scientists and business analysts to interpret. The ultimate goal is to make data accessible so that stakeholders can use it to evaluate and optimise their performance.

What you’ll be responsible for:

Support designing, developing, implementing, managing and supporting enterprise-level ELT/ELT processes and environment Technical and business processes are used to combine data from multiple sources to provide a unified, single view of the data. Be an architect making strategic decisions. Partly responsible for accessing, validating, and querying data from various repositories using available tools. Build and maintain data integration processes using SQL Services and other ETL / ELT processes and scripting tools as well as ongoing requests and projects related to the data warehouse, MI, or fast-moving financial data. Designing the infrastructure/architecture of the big data platform. Evaluating, comparing and improving the different approaches including design patterns innovation, data lifecycle design, data ontology alignment, annotated datasets, and elastic search approaches Developing, creating and maintaining a reliable data pipeline and schemas that feed other data processes; includes both technical processes and business logic to transform data from disparate sources into cohesive meaningful and valuable data with quality, governance and compliance considerations. Customising and managing integration tools, databases, warehouses, and analytical systems. Identifying and eliminating all non-value-adding activities through automation or outsourcing.

What you’ll be working on:

Design and implement the management, monitoring, security, and privacy of data using the full stack of Azure data services to satisfy business needs. Ensuring non-functional system characteristics such as such as security, maintainability, quality, performance, and reliability are captured, prioritized, and incorporated into products. Leverage Agile, CI / CD and DevOps methodologies to deliver high-quality products on time. Architecting, building, testing, and maintaining data platform. Develop and support a wide range of data transformations and migrations for the whole business. Construct custom ETL processes: Design and implement data pipelines, data marts and schemas, access versatile data sources and apply data quality measures. Monitoring the complete process and applying necessary infrastructure changes to speed up the query execution and analyse data more efficiently; this includes Database optimisation techniques (data partitioning, database indexing and denormalisation) & efficient data ingestion (data mining techniques and different data ingestion APIs). Responding to errors and alerts to correct and re-process data. Investigate data mismatches and apply solutions. Data scrubbing and analysis to troubleshoot data integration issues and determine root cause. Any additional duties as assigned.

What you can bring:

Ideally a Bachelor’s degree or equivalent in an engineering/numerate subject (e.g. Engineering, Stats, Maths, Computer Sciences) Experience in full-stack development and applying it to build science products (E.g. could include some or all Python/R, Linux scripting, SQL, Docker coupled with front ends such as Javascript) Some experience as a developer building data pipelines and schemas, with data WH implementation, with SQL database development Hands-on experience using Synapse or related tools with cloud-based resources (e.g. Stored Procedures, ADF, NoSQL Databases, JSON/XML data formats) Hands-on experience with Azure Functions, Azure service bus, Azure Data Factory data integration techniques Knowledge of Data Modelling concepts, monitoring, designs and techniques Knowledge of Data Warehouse project lifecycle, tools, technologies, and best practices Experience using Cloud Computing platforms (ADLS Gen2), Microsoft Stack (Synapse, DataBricks, Fabric, Profisee), Snowflake Data Integration, Azure Service Bus, Deltalake, BigQuery, Azure DevOps, Azure Monitor, Azure Data Factory, SQL Server, Azure DataLake Storage, Azure App Service, Apache Airflow, Apache Iceberg, Apache Spark, Apache Hudi, Apache Kafka, Power BI, BigQuery, Azure ML is a plus Experience with Azure SQL Database, Cosmos DB, NoSQL, MongoDB Experience with Agile, DevOps methodologies Awareness and knowledge of ELT/ETL, DWH, APIs (RESTful), Spark APIs, FTP protocols, SSL, SFTP, PKI (Public Key Infrastructure) and Integration testing Knowledge of Python, SQL, SSIS, and Spark languages. Demonstrative ability to develop complex SQL queries and Stored Procedures Relationship-building and stakeholder management

If this sounds like you, be sure to get in touch – we are shortlisting right away. If you like the sound of the opportunity, but don’t quite tick every box, we’d still like to hear from you ?

Mirai believes in the power of diversity and the importance of an inclusive culture. We welcome applications from individuals of all backgrounds, understanding that a range of perspectives strengthens both our team and our partners’ teams. This is just one of the ways that we’re taking positive action to shaping a collaborative and diverse future in the workplace.

Sign up for our newsletter

The latest news, articles, and resources, sent to your inbox weekly.

Similar Jobs

Data Engineer {Music Tech Start-up

Data Scientist/Data Engineer {Music Tech Start-up}Cape TownR60,000 - R100,000 p/m + Company BenefitsAre you a Data crunching Data Scientist/Engineer looking to join a super disruptive music tech start-up who are just about to scale up?Do you want to work in a company who have a solid backing and a product...

Cape Town

Data Engineer - London / Hybrid - £75,000

My client is based in the London area are currently looking to recruit for an experienced Data Engineer to join their team. They are one of the leaders within the Legal Industry, and are currently going through a period of growth and are looking for an experienced Data Engineer to...

City of London

Data Engineer - Azure - Altrincham - £70k

Data Engineer - Azure - Hybrid - Manchester - £70kThis is a fantastic opportunity for a data engineer to join an expanding IT consultancy based in Manchester! If you are an expert in the MS Azure tech stack, experienced overseeing a team of data professionals and building ETL pipelines, this...

Wilmslow

Data Engineer

Data EngineerPermanentLocation: Home / Crawley (3 days a week on site)Salary: £75,000 - £85,000 (+ benefits)Skills: SQL, Postgre SQL, CockroachDB, Docker, Flyway, SC ClearanceWe are looking to recruit an SC Cleared Data Engineer for a leading software & solutions organisation. The Data Engineer will rive data enablement that underpins cutting-edge...

Crawley

Data Engineer

Are you a Data Engineer who loves to experiment and brings empathy and creativity to your work?Then you’ll fit right in at this people-first, agile tech company. Where collaboration, creativity and a shared commitment to a continuous improvement drives it’s success.Guided by core values – Empathise, Enable, Empower, and Experiment...

Bishopsgate

Data Engineer or Data Strategy Consultant

Pontoon is an employment consultancy. We put expertise, energy, and enthusiasm into improving everyone's chance of being part of the workplace. We respect and appreciate people of all ethnicities, generations, religious beliefs, sexual orientations, gender identities, and more. We do this by showcasing their talents, skills, and unique experience in...

Bromley