Senior Networking Solution Test Engineer – AI Cluster Debugging

2 months ago
Seniority
Senior
Posted
3 Mar 2026 (2 months ago)

We are looking for a Senior Networking Test Engineer with strong system‑level debugging skills to join our End‑to‑End Verification team! You will work on pioneering NVLink, Ethernet and InfiniBand ‑ based AI clusters. Additionally, you will ow complex issues across hardware, system software and AI workloads.

What you’ll be doing:

  • Design and review test and product requirements across the NVLink, Ethernet and InfiniBand / NIC / DPU / Switch portfolio, focusing on large‑scale AI cluster behavior.

  • Build and maintain realistic customer‑like testbeds, including heterogeneous hardware, OS / driver combinations and complex network fabrics.

  • Own end‑to‑end cluster troubleshooting: reproduce customer scenarios, triage across the stack and drive issues to root cause and fix.

  • Read and understand relevant source code to identify defects, validate fixes and improve logging and instrumentation.

  • Collaborate closely with development teams to debug NCCL, RoCE/RDMA and related networking components using logs, code inspection and targeted experiments.

  • Define tests and guide the automation team to implement robust, debuggable suites that produce actionable logs, metrics and traces.

  • Run Regression, Performance, Functional and Scale testing, analyze results and provide clear, data‑driven reports to collaborators.

  • Profile and benchmark deep learning training and inference workloads, correlating model‑level metrics with system and network telemetry to uncover bottlenecks.

What we need to see:

  • B.A./B.Sc. in Computer Science, Electrical Engineering, or equivalent IT/Network/Systems experience.

  • 8+ years of hands‑on networking or system‑level testing and debugging on Linux.

  • Strong Linux networking and debugging skills (for example perf, tcpdump, ethtool, iproute2).

  • Proven production‑grade debugging experience: forming hypotheses, running experiments, and driving issues to root cause under pressure.

  • Expertise in host‑side NIC validation and tuning (offloads, queues, interrupts, firmware/driver interactions).

  • Strong knowledge of AI networking libraries (such as NCCL) and protocols (such as RoCE and RDMA), including performance and correctness debugging.

  • Ability to read and reason about source code (C/C++/Python or similar) and collaborate closely with developers on fixes.

  • Solid scripting and automation skills with Bash / Python / Ansible for setup, log collection, and experiment orchestration.

  • Fast learner, familiar with modern AI tools and workflows, able to adapt quickly.

  • Excellent analytical, problem‑solving and communication skills, with strong ownership and a collaborative approach.

Ways to stand out from the crowd:

  • Hands‑on debugging of collective communication libraries (for example NCCL) or large‑scale LLM training / inference clusters.

  • Experience with large cluster environments (tens to thousands of GPUs or nodes), including incident response and post‑mortem analysis.

  • Deep expertise in tuning and debugging congestion control and lossless Ethernet for AI workloads (for example DCQCN, ECN, PFC).

  • Familiarity with NVIDIA networking technologies (for example BlueField / BF3, ConnectX NICs) and their software stack and diagnostics.

  • Experience debugging issues that span multiple layers (L2/L3, transport, AI frameworks) or contributing to open‑source networking / AI systems.

At NVIDIA, we value diversity and are committed to creating an inclusive environment for all employees. We do not discriminate on the basis of race, religion, color, national origin, sex, gender, gender expression, sexual orientation, age, marital status, veteran status, or disability status. We provide reasonable accommodations to ensure all individuals can participate in the job application or interview process, perform essential job functions, and receive other benefits and privileges of employment. Join us and be part of a team that's pushing the boundaries of technology and making a real impact in the world.

Related Jobs

View all jobs
Spotlight

Senior ML Compiler Engineer

Fractile Bristol, United Kingdom
Spotlight

Machine Learning Engineer - National Security (Gloucestershire)

Mind Foundry Gloucester, Gloucestershire, United Kingdom
On-site Clearance Required

Senior Software QA Test Development Engineer

NVIDIA United Kingdom

Senior Solution Architect - Telco Network Automation

Senior Software and System Architect

NVIDIA United Kingdom

Senior Software and System Architect

Senior Software and System Architect

Senior Software and System Architect

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Where to Advertise Machine Learning Jobs in the UK (2026 Guide)

Advertising machine learning jobs in the UK requires a different approach to most technical hiring. The candidate pool is small, highly specialised and in demand across AI labs, financial services, healthcare, autonomous systems and consumer technology simultaneously. Machine learning engineers and researchers move between roles through professional networks, conference communities and specialist platforms — not general job boards where ML roles compete with unrelated software engineering positions for the same audience. This guide, published by MachineLearningJobs.co.uk, covers where to advertise machine learning roles in the UK in 2026, how the main platforms compare, what employers should expect to pay, and what the data says about hiring across different role types.

Machine Learning Jobs UK 2026: What to Expect Over the Next 3 Years

Machine learning has undergone a transformation that few technology disciplines can match. In the space of three years it has moved from a specialism sitting at the edges of most organisations' technology strategies to a capability that sits at the centre of them. The tools have changed, the expectations have shifted, and the range of industries treating machine learning as a core business function — rather than an experimental one — has expanded dramatically. For job seekers, this creates both opportunity and complexity in roughly equal measure. The machine learning jobs market of 2026 is significantly larger than it was three years ago, but it is also significantly more demanding. Employers have developed more sophisticated expectations, the technical bar for specialist roles has risen, and the landscape of tools, frameworks, and architectural patterns that practitioners are expected to know has broadened considerably. The candidates who will thrive over the next three years are those who understand where the discipline is heading — which specialisms are attracting the most investment, which technologies are reshaping what machine learning engineers and researchers are expected to build, and how the definition of a machine learning career is evolving beyond the model-building core toward a much wider range of roles across the full ML lifecycle. This article breaks down what the UK machine learning jobs market is likely to look like through to 2028 — covering the titles emerging right now, the technologies driving employer demand, the skills that will matter most, and how to position your career ahead of the curve.

New Machine Learning Employers to Watch in 2026: UK and Global Companies Driving ML Innovation

Machine learning (ML) has transitioned from a specialised field into a core business capability. In 2026, organisations across healthcare, finance, robotics, autonomous systems, natural language processing, and analytics are expanding their machine learning teams to build scalable intelligent products and services. For professionals exploring opportunities on www.MachineLearningJobs.co.uk , understanding the companies that are scaling, winning investment, or securing high‑impact contracts is crucial. This article highlights the new and high‑growth machine learning employers to watch in 2026, focusing on UK innovators, international firms with significant UK presence, and global platforms investing in machine learning talent locally.