Jobs

Senior Data Engineer


Job details
  • Printed.com
  • Cramlington
  • 1 month ago
Applications closed

You will play a crucial role in designing, building, and maintaining our data platform, with a strong emphasis on streaming data, cloud infrastructure, and machine learning operations.

Key Responsibilities:

  • Architect and Implement Data Pipelines:
    • Design, develop, and maintain scalable and efficient data pipelines.
    • Optimize ETL processes to ensure seamless data ingestion, processing, and integration across various systems.
  • Streaming Data Platform Development:
    • Lead the development and maintenance of a real-time data streaming platform using tools like Apache Kafka, Databricks, Kinesis.
    • Ensure the integration of streaming data with batch processing systems for comprehensive data management.
  • Cloud Infrastructure Management:
    • Utilize AWS data engineering services (including S3, Redshift, Glue, Kinesis, Lambda, etc.) to build and manage our data infrastructure.
    • Continuously optimize the platform for performance, scalability, and cost-effectiveness.
  • Communications:
    • Collaborate with cross-functional teams, including data scientists and BI developers, to understand data needs and deliver solutions.
    • Leverage the project management team to coordinate project, requirements, timelines and deliverables, allowing you to concentrate on technical excellence.
  • ML Ops and Advanced Data Engineering:
    • Establish ML Ops practices within the data engineering framework, focusing on automation, monitoring, and optimization of machine learning pipelines.
  • Data Quality and Governance:
    • Implement and maintain data quality frameworks, ensuring the accuracy, consistency, and reliability of data across the platform.
    • Drive data governance initiatives, including data cataloguing, lineage tracking, and adherence to security and compliance standards.

Requirements

Experience:

  • 3+ years of experience in data engineering, with a proven track record in building and maintaining data platforms, preferably on AWS.
  • Strong proficiency in Python, experience in SQL and PostgreSQL. PySpark, Scala or Java is a plus.
  • Familiarity with Databricks and the Delta Lakehouse concept.
  • Experience mentoring or leading junior engineers is highly desirable.

Skills:

  • Deep understanding of cloud-based data architectures and best practices.
  • Proficiency in designing, implementing, and optimizing ETL/ELT workflows.
  • Strong database and data lake management skills.
  • Familiarity with ML Ops practices and tools, with a desire to expand skills in this area.
  • Excellent problem-solving abilities and a collaborative mindset.

Nice to Have:

  • Familiarity with containerization and orchestration tools (e.g., Docker, Kubernetes).
  • Knowledge of machine learning pipelines and their integration with data platforms.

Benefits

  • 22 days holiday Plus 8 days bank holidays
  • Staff discounts & Friends and Family discounts
  • Cycle to work scheme and Tech Scheme
  • Breakfast and drinks provided
  • Charity day per annum supported
  • Summer and Christmas Parties
  • Street food days
  • Perkbox membership
  • Enhanced Maternity & Paternity leave
  • Employee referral scheme

Sign up for our newsletter

The latest news, articles, and resources, sent to your inbox weekly.

Similar Jobs

Senior Data Engineer

You will join the Service Performance team that monitors the entire suite of DWPs digital services a grand total of 700 software applications and systems. We use Splunk, ServiceNow and Power BI tooling to bring data together and develop quality dashboards which are relied upon and widely used by service...

Department for Work and Pensions Manchester

Senior Data Engineer

Senior Data Engineer needed for AI Startup Do you want to work on genuine greenfield projects? Want to use your skills to aid Education? Want to work with like minded individuals with the latest technologies? The Role Youll play a pivotal role in designing, building and optimising their data systems,...

Ocho Belfast

Senior Data Engineer

Senior/Lead Data EngineerA Senior/Lead Data Engineer is required by a rapidly growing Fintech based in London. The Senior/Lead Data Engineer will have extensive knowledge of modern technologies. This would ideally suit a Data Engineer with a passion for driving excellence and experience in a platform engineering or data engineering team....

Dragonfly People London

Senior Data Engineer web3

Hiring CriteriaMUST have experience setting up A/B testing pipelines & MUST have GCP experience.Experience working within the Crypto industry is a plus.Experience working at a fast moving impressive crypto company or FAANG level company or scale-up.Can work full time in London 5 days week. (West End)

TBC Holdings LLC London

Senior Data Engineer (start up experience)

Senior Data engineer- PermanentWant to help save lives?*Hybrid in Central London**2 stage interview*Did you know a frontline worker loses their life every 10 seconds?Imagine leading a team in developing a risk management platform to help save lives.Your role is crucial in creating and maintaining an innovative mobile application that will...

trg.recruitment London

Senior Data Engineer

Senior Data EngineerTech- Python, Airflow, Kafka, AWS/GCP/AzureSalary –£70,000 - £120,000Location– Full Remote (UK)A growing digital consultancy that specialises in transformation is looking for a highly skilled Senior Data Engineer to join a growing and highly experienced team solely focused on building modern data platforms. You will focus on exciting client...

Formula Recruitment Birmingham