High-performance computing with Ubuntu

Jason Nucciarone and Felipe Reyes gave back-to-back talks about high-performance computing (HPC) using Ubuntu at SCALE this year. Nucciarone talked about ongoing work packaging Open OnDemand — a web-based HPC cluster interface — to make high-performance-computing clusters more user friendly. Reyes presented on using OpenStack — a cloud-computing platform — to pass the performance benefits of one's hardware through to virtual machines (VMs) running on a cluster.

Open OnDemand

High-performance computing is, according to Nucciarone's slides: "A paradigm where you use supercomputers, computing clusters and grids to solve advanced scientific challenges." Nucciarone — a self-described "not so ancient elder of Ubuntu HPC" — has been involved with Ubuntu's HPC community team since it formed in the wake of the 2022 Ubuntu summit. At that event, participants from Canonical and Omnivector noticed that they were facing a lot of the same problems trying to package software for use on HPC clusters, and formed the community team to try and address that. Nucciarone called for people to join the group if they think it's interesting, even if they are not currently doing any HPC work.

He moved on to the main focus of the talk by asking why, even though HPC users are "bright and smart people", they often have trouble taking full advantage of an HPC cluster. He gave some examples of commonly touted reasons, such as the fact that HPC clusters often use different tech stacks, or that the system is "too advanced". None of those are the main problem he sees. "The main problem is this: the terminal."

HPC clusters are often organized with a pool of worker nodes where jobs are sent. Users do not connect to workers directly, but rather to a "head node" or "jump host" that provides them with tools to manage the cluster. When a new HPC user connects (invariably via SSH) to the head node of a cluster, they are presented with a shell prompt. Nucciarone pointed out four problems with this: affordances, learnability, consistency, and visibility. Firstly, the terminal doesn't tell the user what it can do. This makes discovering what the system is capable of difficult. Secondly, the languages involved in running an HPC system all have their own learning curve. Becoming simultaneously proficient in bash, Python, and any other programming languages in use on the cluster is a tall order. Thirdly, each tool has its own interface and command-line conventions. Learning the interface of one tool does not guarantee a user can transfer that knowledge to another tool. Finally, the shell just provides a user with a blank prompt, it doesn't guide them toward what to do next. "How do you learn what you don't know if you don't know what you don't know?"

Nucciarone's proposed solution to this is Open OnDemand, "an interactive portal for accessing HPC resources over the internet". He also described it as a "web-based application that moves beyond the terminal" and that "focuses on providing a visual interface for accessing your HPC resources". The benefits of using Open OnDemand include the ability to use interactive applications like notebooks, compatibility with many existing cluster scheduling systems, and even the capacity to "launch a desktop session directly on your cluster" (enabling users to run graphical applications on the cluster), he said.

The Ubuntu HPC team has been working to develop "Charmed HPC", a system to make deploying HPC clusters from scratch much easier using Juju. Open OnDemand could tie into that by providing a better user interface for users out of the box. For that to work, though, Open OnDemand needs to be packaged for Ubuntu, which is what Nucciarone is currently working on.

He has been writing a snap that runs all the services required for Open OnDemand. He gave a demo showing how this makes spinning up the Open OnDemand web interface as simple as installing the snap and then starting the service. Actually using the web interface for anything requires having a correctly configured HPC cluster already set up to connect to, of course.

Nucciarone is currently working on OpenID Connect integration and sorting out some internal problems with Open OnDemand. One particular difficulty has been packaging the custom versions of the Apache and NGINX web servers that Open OnDemand requires. When asked why both were needed simultaneously, Nucciarone replied: "To be honest, it's not quite clear to me myself". He went on to say that that was how the upstream project had it, and that a lot of their tooling is built around using Apache.

Once his work on fixing the last few problems is finished, he plans to make the snap available in the Snap Store. In the future, he wants to add built-in support for a base set of interactive web applications, and write a Juju operator for easy deployment. Nucciarone ended his talk by once again asking people to join the Ubuntu HPC team.

OpenStack

Reyes's talk was aimed at a slightly different audience. He talked about how to use OpenStack — an infrastructure-as-a-service platform which can be used to configure HPC clusters — to get the most performance out of a cluster. OpenStack allows users to run virtual machines on a cluster, transparently handling the setup of networking, migration of workloads, etc.

He described the relevant components of an OpenStack cluster, and pointed out that everyone has different metrics that they care about optimizing. Different people will care about throughput, CPU usage, latency, etc. He also showed a diagram from Brendan Gregg of the myriad performance-observation tools that we have to keep track of these things.

There are a number of different configuration options that users of OpenStack have available to fine-tune the performance of their clusters. Reyes explained how one can create "flavors" — particular types of node available on a cluster, fine-tuned to a particular workload. "You may have a fleet of nodes, and those nodes may have different properties", he explained. Segmenting nodes with a particular property (for example, solid-state drives) into a separate flavor can help the scheduler to decide where a VM running on the cluster ought to be assigned.

Once nodes have been split into flavors, the rest of the settings he discussed can then be applied per-flavor. The general theme of the options is making the virtualized hardware that is available to the VM map as closely as possible to the underlying physical hardware, to avoid overhead. Reyes summed this up as trying to get a "good representation so the hardware can be taken advantage of".

In pursuit of this, flavors can specify the number of sockets, cores, and threads made available for each VM. Those virtual CPUs can then be pinned, to ensure that specific physical CPUs are used for each virtual CPU. If no other VMs have access to those CPUs, this will ensure that only the configured VMs can run there. This isn't sufficient to give full control of the CPUs to the VM, though. The host kernel could still choose to run kernel tasks on those CPUs. To solve this, Reyes recommended adding the isolcpus command line option, to instruct the Linux kernel not to schedule tasks on those CPUs. "This gives you the guarantee that any VMs [...] are not competing for resources."

He also noted that OpenStack supported configuring Non-Uniform Memory Access (NUMA) for each flavor. Usually, the kernel picks up information about the NUMA configuration of a machine from the hardware. By configuring the virtual hardware of the VM to match the real NUMA properties of one's cluster, one can let the kernel take advantage of its NUMA optimizations. By default, VMs have only one NUMA node, but one can configure a more-specific setup, including the ability to "create asymmetric definitions".

During the Q&A session, Nucciarone asked what could be done to improve the documentation of these options, so that people running private clusters know what they need to do. Reyes responded that the documentation was indeed a problem, and "makes a lot of assumptions", ultimately suggesting that it could be improved by adding more pointers to other places where people can learn about concepts like NUMA.

Another member of the audience asked whether the documentation should include example use cases. Reyes agreed with the spirit of the request, but said that it was tricky because people sometimes take examples in documentation as golden rules that they can copy and paste without understanding. "If you don't understand the characteristics of the things you're running, it's better to stay in the defaults". This led to some discussion, with a different member of the audience saying: "No default works well for everyone".

Ultimately, the configurations Reyes discussed are highly specialized, and probably best employed by people who already know how details like NUMA and CPU-pinning work. He was clear that while these options offered fine control over the performance characteristics of hosted VMs, there is no substitute for actually measuring the performance characteristics one is interested in improving.

Conclusion

High-performance computing is a challenging industry. Despite that, people are putting significant effort into making Ubuntu more useful as a platform for building an HPC cluster. Future developments are likely to be reported on Ubuntu's HPC Blog or discussed on the Ubuntu Discourse forum.

[I would like to thank LWN's travel sponsor, the Linux Foundation, for travel assistance to Pasadena for SCALE.]