Job Description
Our client, a leading global IT service provider,is recruiting for a Data Engineer - Streaming to join their business in the UK.
Key Responsibilities:
Participate in requirement gathering, analysis, solution design to build big data engineering applications on AWS Data services such as AWS EMR, Snowflake using Spark and AWS Glue as ETL framework.
Migrate and build Cloudera based hadoop, hive, impala, Kafka, Sqoop, Spark based data, jobs and security policies to AWS EMR , S3 , Snowflake and other AWS services such AWS Glue, DMS, IAM.
Hands on involvement in Low Level Design, Development and architecting of large Data projects leading developer and testing teams.
Job Scheduling and Automation.
Data Validation, Quality checks, profiling and data reconciliation testing.
Work as an individual contributor as well lead teams when required.
Mentor junior members in the team by improving their skills and knowledge and have the ability to get things delivered.
Work with both senior and junior team members like Project Manager, Hadoop Architect, other data engineers, data modelers, report developers, testers to complete the task.
Setting up security and governance policies on data, users, data pipelines on AWS data services.
Troubleshooting application errors and ensuring that they do not occur again.
Apply agile and CI/CD methodologies and tools for development and deployment in agile mode.
Must have:
Streaming Data Engineering and Analytics over it. Kafka, AWS Data Services, AWS, ETL, Spark/Scala, Java, Python, EMR, AWS Glue, AWS Athena
Good to have: Python, Spark, Hive, HDFS, Impala, Sqoop. Informatica
Qualifications:
Bachelor’s degree and 9+ years of experience in the IT industry.
5+ years’ experience as Big Data Engineer on Hadoop, AWS EMR and its ecosystems
Experience in Banking Domain plus.
Experience on AWS EMR, AWS Glue, Athena, S3, DMS, SCT and Cloudera CDH based hadoop.
Should have extensively worked on Big Data projects on AWS EMR based Hadoop, HDFS/S3, Spark, Hive, Impala leading teams and interacting with Architect roles and Client.
Work experience with SQL, RDBMS, complex queries.
Understanding and experienced in data warehousing data modeling concepts.
Has an understanding of large Batch and Stream processing.
Knowledge of Quality Assurance methodologies, exposure to all facets of Extract, Transform and Load (ETL) processes.
Proficiency on Linux / Unix command line.
Experience in applying agile and CI/CD tools and methodologies for deployment and related automations.
Experience in performance optimization techniques on Hadoop, HDFS, Hive, Spark, File Formats, and providing technical guidance to other application developers.
Banking domain experience is mandatory
English language fluent is required
The latest news, articles, and resources, sent to your inbox weekly.
Data Engineer4 Month ContractLocation: Home / Corsham (2 days a week)Rate: £550 - £600 per day (Inside IR35)Skills: Postgres SQL, SQL Stored Procedures, Power BI, SSIS, SC ClearanceWe are looking to recruit an SC Cleared Data Engineer for a global software and solutions organisation. This is an initial 4 month...
Corsham
Data Engineer - London - AWS - Up to 70,000 + BenefitsExciting opportunity to work with a forward thinking company who offer their employers the chance to work with cutting edge tech, grow their skills with fully-costed training programs, maintain a healthy work-life balance with modern working arrangements - up...
City of London
Job Title: Data EngineerLocation: LimerickSalary: €(phone number removed) (DOE)HybridReperio have partnered with a small, but high-performing MedTech company who are on the lookout for Data Engineers to join their growing Data department. As a Data Engineer, you will play a crucial role in designing, building, and maintaining scalable data pipelines...
Limerick
This is a fantastic opportunity to work as a Data Engineer, on a long term contract, inside IR35, requiring 2 days per month in Swansea.It's important to note that this Data Engineer role will require 2 days per month on site in Swansea.The key skills required for this Data Engineer...
Mynyddbach Community
Contract Data Engineer (6 Months) - Azure Azure, Azure Functions, ADF, Azure databricks, PowerBi We are seeking an experienced Data Engineer for a 6-month contract to design, develop, and deploy scalable, secure Azure solutions. This role offers the opportunity to work on cutting-edge cloud technologies and contribute to key business...
Northampton
Data EngineerPermanentLocation: Home / Brighton (on site 6 days a month)Salary: £50,000 - £60,000 (+ excellent benefits including 28% employer pension contributions, + bonus)Skills: Cloud Data Platforms (Azure preferred), SQL, Azure Data Factory, AgileWe are looking to recruit a Data Engineer for a technology driven public sector organisation based in...
Brighton
