|
|
Subscribe / Log in / New account

What we need to take away from the XZ Backdoor (openSUSE News)

Dirk Mueller has posted a lengthy analysis of the XZ backdoor on the openSUSE News site, with a focus on openSUSE's response.

Debian, as well as the other affected distributions like openSUSE are carrying a significant amount of downstream-only patches to essential open-source projects, like in this case OpenSSH. With hindsight, that should be another Heartbleed-level learning for the work of the distributions. These patches built the essential steps to embed the backdoor, and do not have the scrutiny that they likely would have received by the respective upstream maintainers. Whether you trust Linus Law or not, it was not even given a chance to chime in here. Upstream did not fail on the users, distributions failed on upstream and their users here.


(Log in to post comments)

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 12, 2024 15:24 UTC (Fri) by gwolf (subscriber, #14632) [Link]

I only partially agree here. While distributions do have the responsability to check what they package and distribute, we also work trusting the upstream developers. Finding a hidden *and* obfuscated backdoor like this, likely designed by a national security organization, is not within what I'd expect a regular package maintainer to check.
It's good to have a rough understanding of the diff between previous and new version, but this was not an easy to spot backdoor. And I cannot blame the distributions for not finding it *earlier*.
And, I must add: This is a case where the staging mechanisms used by the different affected distributions (Debian with the unstable/testing/stable process, and the RedHat ecosystem with the progression between RawHide, Fedora, CentOS and RHEL) worked *exactly* as it should. The backdoor was found, yes, by mere chance, by an over-zealous developer worried about milliseconds worth of difference, and not by formal auditing processes. But it worked: It effectively prevented the backdoor from being deployed in live, production servers.
Of course, if people choose to run unstable / bleeding edge distributions... Expect bugs and warts!

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 12, 2024 16:12 UTC (Fri) by JoeBuck (subscriber, #2330) [Link]

Ah, but the lesson here for future attackers is that they must make sure that their back door doesn't introduce milliseconds worth of difference in timing. I'm sure that they or their rivals are doing a postmortem so they can do better next time.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 12, 2024 17:13 UTC (Fri) by Heretic_Blacksheep (subscriber, #169992) [Link]

I think the failure point with your argument is the traditional trickle down system *didn't* work. The backdoor was found on a fluke that can't be counted on to produce similar results in the future. If that stroke of OCD about a few hundred milliseconds in performance hadn't pricked Freund's curiosity, the malicious code would still be there. Jian Tan would still be working its tendrils into XZ, Debian, other distributions, and projects. From the perspective of the end user, it doesn't matter who's to blame, be it the overworked-overstressed upstream maintainer(s) whom I do sympathize with, or the distro package maintainer. The system failed. It took someone outside of the upstream - distro maintainer pipeline to spot the problem. FOSS merely made it easier to spot once someone was alerted enough to look. If both FOSS and proprietary software projects aren't at this very minute combing through third party previously unvetted code right now they're responsible for, they're fools. Jian Tan is definitely not the only group to attempt this. They're just the ones that got caught! It's definitely going to end up a Darwinian process. It's well known in security circles China and NK have been working diligently to get sleepers into both tech corporations and popularly used hobby projects alike for years with this very aim.

The fact of the matter is that *both* upstream projects and distro package maintainers will have to be more diligent who and what they gate into their code bases and repositories in the future. It's the responsibility of both groups, not just one or the other, to make sure what's packaged is not just reasonably bug free, but also reasonably free of malicious code. The tools to do this at scale will have to evolve. If people don't like that, perhaps its time to re-evaluate career or hobby choices, because TLAs aren't going to play nice just because someone's an unpaid volunteer. If this results in less software out there, but higher quality review processes and slower releases, then that's just going to need to be done. Corporations and governments with their security on the line are going to have to stop freeloading. Both are extremely bad at investing in necessary infrastructure till it's crashing down around their ears. Individual programmers can't realistically go up against nation states without greatly expanded resources, including education about secure coding practices, and in the case of the general public, better technological education and accountability.

At some point heightened diligence isn't going to just be a good idea. It's likely to become the law in many, if not most, countries. Software won't be usable by organizational or government entities without proper vetting. Those vetting processes will also have to evolve because the current methods of certifications are far too arthritic to keep up with modern security landscapes.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 12, 2024 20:47 UTC (Fri) by kleptog (subscriber, #1183) [Link]

Honestly, this feels a little too much "glass half empty" to me.

Attacking open-source this way is a high-risk, high-reward strategy. Sure, if you could backdoor ssh you'd be all set, but the evidence would be in public view for all time. All it takes is one person to stumble across it. There's no way to ever remove the backdoor from history. If it were really that common, we'd see the evidence all over the place. And we don't. This attack was found precisely because the source was publically available.

Add to the fact that the TLAs and their respective governments also use open-source software everywhere, so they'd be building backdoors into their own systems. (I'm guessing the targetting of DEB and RPM was to avoid custom government distributions being affected.)

On the other hand, backdooring closed-source software is much lower risk. The number of people who would ever see the code is much lower. When a product reaches end of life the code simply vanishes. Insert a backdoor in an NVidia/Broadcom/Dell driver, who would ever find out?

So sure, open-source developers and maintainers need to get better at vetting, but it's the corporations where the risk is. And I guess the tools to discover these kinds of attacks are also going to be developed as open-source because there's no direct benefit to any individual corporation. And how could you trust a closed source vetting tool not to be backdoored itself?

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 0:01 UTC (Sat) by LtWorf (subscriber, #124958) [Link]

USA government distributions can and are based on debian, ubuntu and red hat. They would have been hit if the vulnerability hadn't been discovered.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 2:14 UTC (Sat) by alp (subscriber, #136414) [Link]

Depending to whom you speak, a NOBUS that is protected by a single ed448 key, assuming ed448 and the rest of the code is secure, would have been acceptable.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 20, 2024 9:34 UTC (Sat) by sammythesnake (guest, #17693) [Link]

I think it's important to remember that this backdoor was designed to only respond to requests cryptographically signed by a private key held by the author - all the TLA(s) involved would need to do is not actively attack their own infrastructure(!)

As to what the real motivation might be for limiting it to debs and rpms - who knows‽ My guess is that it's about shrinking the chances of detection. Some installations (and particularly highly specialised and therefore possibly more closely scrutinised ones) would be clean and not be a means of detection, but I reckon the larger reason is simple compatibility concerns.

With unbounded numbers of non-standard build systems, packaging systems etc. there's a much greater chance that something more conspicuously wrong would happen and lead to detection. The number of systems *not* built on RPMS/debs is small enough that skipping them is probably a sensible trade-off...

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 12, 2024 21:52 UTC (Fri) by AdamW (subscriber, #48457) [Link]

"If that stroke of OCD about a few hundred milliseconds in performance hadn't pricked Freund's curiosity, the malicious code would still be there."

It's important to note you cannot know this. It's a counterfactual. We don't live in the world where Andres didn't find the issue, so we don't know what would have happened next. Perhaps what would have happened next is somebody else would have found it half an hour later. Perhaps not. We don't *know*. You can't say you know that "the malicious code would still be there. Jian Tan would still be working its tendrils..." etc.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 12, 2024 18:20 UTC (Fri) by willy (subscriber, #9762) [Link]

I think the conclusion is solid though. The distro packager *should* be part of the upstream community. They bring unique insight that is useful to both the package upstream and the distro downstream.

I'm always disappointed by things like Debian bug 897426 where the packager washes their hands of the patch and wants me to talk to upstream directly.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 12, 2024 20:36 UTC (Fri) by WolfWings (subscriber, #56790) [Link]

It's an issue for all hexchat not specific to the Debian distro, so why carry that extra patch downstream when it's not specific to that downstream?

Like I see nothing at all done wrong here by Debian, you just came barking up the wrong tree.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 12, 2024 21:17 UTC (Fri) by willy (subscriber, #9762) [Link]

The point of using a distro is to have a single contact point. One bug tracker to use. It's the job of the maintainer to coordinate patches with upstream. That's what I did when I was a Debian developer. It got a lot of nice patches into pciutils.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 12, 2024 22:32 UTC (Fri) by zeha (subscriber, #61580) [Link]

> The point of using a distro is to have a single contact point. One bug tracker to use.

This was very useful in a time when most software was developed by single entities, without public bug trackers, mailing lists, etc.

> It's the job of the maintainer to coordinate patches with upstream.

If the job is mostly to copy-paste data from one bug tracker into another, its pointless bureaucracy. Please just talk to upstream directly when it's easy.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 13:57 UTC (Sat) by elw (subscriber, #86388) [Link]

An extra layer it is but pointless it is not. There is one upstream maintainer of a given piece of software but there are multiple packagers of that software; one per distribution in theory. By distributing front line contact to the maintainers within each distro you are reducing the work on the single upstream maintainer, allowing that person to focus on actually fixing upstream bugs.

If you combine this with a system where distro maintainers are the ones working directly with upstream, or even a part of the upstream community, it allows a strong rapport to form between the upstream maintainers and those submitting issues upstream. This not only decreases the quantity of reported issues but increases the quality of what is reported, further improving upstream’s ability to focus on what matters.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 15, 2024 7:13 UTC (Mon) by calumapplepie (subscriber, #143655) [Link]

In general? Yes. Having a distro maintainer filtering and fixing bugs is a good thing. However, if said maintainer can't reproduce a problem, or see the neccesity for a patch, then it's bad. What if upstream asks for more details on a bug? Does the distro maintainer email the reporter? And then forward their response, back and forth forever? What if upstream requests a patch from someone else be reworked to try and fit their goals? Should the distro maintainer try to rewrite the patch they didn't write which solves a problem they don't have for a reason they didn't understand?

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 16, 2024 16:10 UTC (Tue) by NYKevin (subscriber, #129325) [Link]

My 2¢: Don't report anything you can't repro yourself. You can tell the user where to report it, explain how to test it on a "clean" upstream build, provide any relevant distro-specific info, etc, but you should not be reporting bugs that you have not observed, except in extraordinary cases where the user who observed the bug is somehow unable to report it on their own (e.g. in a closed bug tracker or the like).

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 20, 2024 9:49 UTC (Sat) by sammythesnake (guest, #17693) [Link]

"Closed bug tracker" and "FOSS" are, while not *incompatible*, at least a very *odd* combination.

On the other hand, any Linux-based distro that packages software developed without the benefits of it being not only FOSS but also having all the usual ancillary gubbins like in-the-open development processes (including bug trackers available to the general population) probably *should* consider themselves the primary contract point for those who are perhaps better described as "customers" than "users"...

For my tuppen'th, I think I'd generally reach out to upstream and downstream in parallel, providing each with a link to the bug in the other's tracker.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 12, 2024 22:09 UTC (Fri) by mathstuf (subscriber, #69389) [Link]

I've found that it depends. Most of the time it's better for a direct stakeholder to be involved in the review of patches sent upstream. I really don't like playing telephone between upstream reviews and distro patch submitters. Even as a maintainer for a project, if someone brings up an issue on the discussion forum, I think it is better to ask for an interested party to file the issue for tracking so that there's a stakeholder in the loop from the start. Filing on their behalf is prone to leading away from the desired outcome of the original requester making maintainers unhappy that there's an unwanted feature merged and a user without a solution.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 10:42 UTC (Sat) by smcv (subscriber, #53363) [Link]

> Most of the time it's better for a direct stakeholder to be involved in the review of patches sent upstream

Right - as a downstream maintainer, if I forward someone else's patch, I will not necessarily be able to defend its design decisions when upstream queries them, because they weren't my design decisions and I don't necessarily know why they were made.

For relatively obvious bug fixes ("there's a double-free here, solution: don't free it the second time") that's less of a concern, but for bug fixes that involve tricky changes that could have been done a different way, it's better for the author to be the one talking to upstream, and doubly so for feature requests where there isn't yet any specific "correct" behaviour.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 17:51 UTC (Sat) by dilinger (subscriber, #2867) [Link]

That's not the packager washing their hands of it, that's the packager kindly asking if you can send it upstream. Maybe they're busy with other things, and it's your patch; you can answer upstream's questions about it, you're in a better position to make adjustments/fixes to it, and you deserve the credit for it. Having the packager as an intermediary often just wastes everyone's time, *unless* there's circumstances in which upstream is difficult to work with or some special reason why the packager would need to massage the fix to get it upstream. And if for some reason you can't or don't want to send the patch upstream, you can simply respond asking the maintainer to do it for you. You apparently didn't respond, so the maintainer even went and showed upstream the patch to see if they'd be open to accepting it.

I regularly do the same thing with chromium. I package it, but it's a complex piece of software with lots of things I've never used/touched. When someone reports a bug about their screenreader not working properly with it, I request they submit it upstream. If they can't, I'll do it for them, but I've never even used a screenreader. If upstream asks me any questions about the desktop environment, or how to reproduce the bug, I can't tell them. Instead, I'll just pass the message along to the bug reporter. That's silly, and introduces lag (and maybe I'm busy filing taxes and forget to forward the email). Similarly, I've never used widevine or done chromecasting, so I'm going to send those reporters upstream as well.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 15, 2024 15:38 UTC (Mon) by nim-nim (subscriber, #34454) [Link]

Actually, the article describes very well the value of a distribution (which is not what the author thinks it is).

The value is common auditable build system that enabled to identify all the bits the malicious code touched and rebuild them in a few hours.

Just try to do the same without a distribution, I dare you.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 8:10 UTC (Sat) by tilt12345678 (subscriber, #126336) [Link]

Do not patch OpenSSH unless you can be sure that the changes make it upstream. Do not patch OpenSSH without monitoring the consequences; if such a monitoring is not feasible, then do not patch OpenSSH. Do not patch OpenSSH just because others do it. Do not patch OpenSSH.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 8:16 UTC (Sat) by mjg59 (subscriber, #23239) [Link]

OpenSSH isn't an idol. The OpenSSH developers aren't magically better than everyone else. It's entirely legitimate to patch it to meet requirements that upstream is not currently invested in, as long as there's appropriate review of those patches.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 8:22 UTC (Sat) by tilt12345678 (subscriber, #126336) [Link]

> It's entirely legitimate to patch it to meet requirements that upstream is not currently invested in, as long as there's appropriate review of those patches.

No, it is not.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 8:36 UTC (Sat) by mjg59 (subscriber, #23239) [Link]

No upstream project is sacrosanct. No upstream project *should* be expected to care about every requirement that a downstream has. Different people have different needs, and a core tenet of free software is that you're able to deal with that regardless of upstream's desires.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 10:55 UTC (Sat) by tilt12345678 (subscriber, #126336) [Link]

What essentially is created by such a downstream decision are forks of OpenSSH. They then have to be scrutinized with the same effort that the upstream project is. Responsibility is not a given, it has to be taken.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 11:24 UTC (Sat) by pizza (subscriber, #46) [Link]

> What essentially is created by such a downstream decision are forks of OpenSSH

Yeah, so? What else are you supposed to do when upstream explicitly rejects your use case?

> They then have to be scrutinized with the same effort that the upstream project is.

What makes you say the various downstreams didn't carefully scrutinize this particular patch? This wasn't a matter of "different effort" so much as "different priorities". Indeed, the path of least effort was to abandon the use case that led to that patch to begin with.

> Responsibility is not a given, it has to be taken.

...What does this mean in the presence of an *actively hostile upstream*?

...Do *you* personally review every line of code of every software component in your system, prior to compiling and installing it yourself?

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 13:57 UTC (Sat) by tilt12345678 (subscriber, #126336) [Link]

All you are saying, in somewhat heated terms, is: "this ist fine".

For the Sake of toning this debate down a bit: on behalf of my employer, my colleagues and our customers, we truly value the work the distributors are performing, in many specific cases and in general, and by "truly" i also mean in the economic sense.

However, nothing you can say will convince me that patching critically exposed software is fine. It's one of the most ugly technological debts, and if done properly, consequentially incurs massive project cost. I speak from experience.

In this particular instance, if openssh does not integrate with systemd-notify upstream, *and* if another integration is not possible, *and* if system service management is a must-have, then we have a very critical situation. It must be resolved, clearly and openly, and not be obscured by downstream patches.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 14:19 UTC (Sat) by bluca (subscriber, #118303) [Link]

And how much is your employer going to pay to "resolve it cleanly and openly"? Where do we send the invoice(s)?

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 14:32 UTC (Sat) by tilt12345678 (subscriber, #126336) [Link]

Depends. Can you deliver, or do you just want to win an internet argument?

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 16:37 UTC (Sat) by bluca (subscriber, #118303) [Link]

So "nothing" then, got it

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 14:57 UTC (Sat) by Wol (subscriber, #4433) [Link]

> All you are saying, in somewhat heated terms, is: "this is fine".

No. They are saying this (bad) state of affairs is the only option available to them.

> However, nothing you can say will convince me that patching critically exposed software is fine. It's one of the most ugly technological debts, and if done properly, consequentially incurs massive project cost. I speak from experience.

Tough titty.

What you seem to be missing is that "patching critically exposed software" may be a NECESSITY. And if upstream *WON'T* talk (as seems to be the case here), then you are between the rock of "running security software with known vulnerabilities" and the hard place of forking.

Both of which, in your words, "incur massive project cost".

But hey, if it's fine in your world to order other people to talk to brick walls, expect them to assume it's fine for you to talk to a brick wall as well :-)

Cheers,
Wol

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 19:55 UTC (Sat) by tilt12345678 (subscriber, #126336) [Link]

> What you seem to be missing is that "patching critically exposed software" may be a NECESSITY. And if upstream *WON'T* talk (as seems to be the case here), then you are between the rock of "running security software with known vulnerabilities" and the hard place of forking.

In this particular example, i see no such a necessity. At least the following alternatives come to mind:

1. The team developing the system service manager could have opted to provide a less intrusive, more compatible notification mechanism.
2. The switch of the distribution to said system service manager could have been postponed until a time where such apparently critical security concerns would have been resolved.
3. The patches could have been applied, but this should have lead to the conclusion that a re-implementation of the a service for the secure shell and secure file transfer protocols was required that would meet the demand of Linux environments using systemd service management.
3a. This could also have lead to an evaluation of the secure shell protocol being an adequate remoting utilitiy for such environments.

I am convinced that with enough expertise and good will, a different solution for notification could have been found that does not require shoehorning it into the service using a downstream patch to an upstream project that obviously considered this an unsupported scenario.

My main point of critique about the entirety of decision-making that lead up to this incident is that there is indication that reasonable security practices were neglected in favor of personal ambition, and if it just was about gaining the upper hand in factional disputes (systemd vs. systemd-is-evil, bsd vs. linux and others).

Here's a theory about it: We work in a very competitive industry (IT), we are constantly exposed to the internet with all its trolls and an unfiltered melange of patterns and anti-patterns, *and* we do F/LOSS, which makes us feel like in a hamster wheel, just that it's not about running, it's about arguing with morons. On top of that, many of us are badly funded and feel exploited, be it by megaclouds, tech billionaires or even our own users. It makes perfect sense that we are susceptible to factional infighting and zealotry. Sometimes, that mindset even might be adequate. But in IT security it is not, and while problems like stress and underfunding are all understandable, the rules of the game change into a direction where such practices will not be tolerable anymore.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 20:47 UTC (Sat) by kleptog (subscriber, #1183) [Link]

> 1. The team developing the system service manager could have opted to provide a less intrusive, more compatible notification mechanism.

There are no other notification mechanisms, this is the only game in town. It's not even systemd specific. Why would we need a different notification method for OpenSSH than literally every other daemon? It's one line of code, hardly intrusive. Most other system daemons on Linux support notification out of the box.

> 2. The switch of the distribution to said system service manager could have been postponed until a time where such apparently critical security concerns would have been resolved.

The patch was one line of code. There was no indication there was any security problem with it. The reason it was rejected wasn't (AFAICT) security related. You're not going to halt a migration for something fixed by a one line of code.

> 3. The patches could have been applied, but this should have lead to the conclusion that a re-implementation of the a service for the secure shell and secure file transfer protocols was required that would meet the demand of Linux environments using systemd service management.

The patch is one line of code [1]. There is no world in which a reimplementation is cheaper than adding one line of code.

> 3a. This could also have lead to an evaluation of the secure shell protocol being an adequate remoting utilitiy for such environments.

OpenSSH is a standard. Why would you switch to something else when adding one line of code solves the problem?

Your alternatives might be reasonable in a world were everyone has loads of time to go around finding optimal solutions to problems. In reality it's all just volunteers who do this for fun. After the OpenSSH maintainers rejected the patch, no-one else was going to spend more than 5 minutes looking for an alternative for a one line patch.

> My main point of critique about the entirety of decision-making that lead up to this incident is that there is indication that reasonable security practices were neglected in favor of personal ambition,

Honestly, given the choice between the one-liner we used to have and the patch that has now been merged to OpenSSH, the former seems more obviously correct. There was no obvious security issue here, that was never brought up as an issue. I don't really know what was going through the minds of the OpenSSH maintainers, but if >90% of your users are using a patched version you disapprove of, that should trigger some scratching behind the ears.

Even now, most people using OpenSSH on Linux have it patched for socket activation which upstream doesn't like. Yet only actually running the daemon on demand is just obviously superior.

[1] https://git.centos.org/rpms/openssh/blame/SOURCES/openssh...

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 14, 2024 0:21 UTC (Sun) by wahern (subscriber, #37304) [Link]

> Honestly, given the choice between the one-liner we used to have and the patch that has now been merged to OpenSSH, the former seems more obviously correct. There was no obvious security issue here, that was never brought up as an issue. I don't really know what was going through the minds of the OpenSSH maintainers, but if >90% of your users are using a patched version you disapprove of, that should trigger some scratching behind the ears.

We can partly tell what was going through the maintainer's mind from the original thread in August 2018, "openssh 7.6 and 7.7 on Oracle Linux 7 (compiled from source) doesn't start correctly with systemd", https://lists.mindrot.org/pipermail/openssh-unix-dev/2018... The most relevent participants in that thread were Damien Miller (maintainer) and Peter Stuge (long-time contributor, IIUC).

The initial one-line approach depending on libsystemd was quickly rejected. It seems like Damien's express reasoning was not wanting a functional dependency, especially one that's also a literal dependency on an LGPL library, but also for other architectural reasons between how OpenSSH behaves and the expectations of systemd.

Peter Stuge also rejected the one-line approach outright because, "At the very least it introduces a dependency on libsystemd into sshd, which is undesirable for reasons of security and convenience."[1] I think Damien would have been of a similar mind regarding security, except in this case his expressed focus seemed to lie elsewhere.

Most relevant messages in that thread:

[1] Peter Stuge's in-depth analysis of systemd integration describing fundamental mismatches between OpenSSH process management requirements and systemd expectations, but also including a patch implementing sd_notify in-tree: https://lists.mindrot.org/pipermail/openssh-unix-dev/2018...

[2] Damien Miller agreeing with the analysis, but coming to a different conclusion than Peter re the prudence of any systemd integration: https://lists.mindrot.org/pipermail/openssh-unix-dev/2018...

[3] Peter Stuge trying to change Damien's mind: https://lists.mindrot.org/pipermail/openssh-unix-dev/2018...

[4] Damien Miller replying to another participant that an optional dependency (literal, functional?) is a dependency like any other: https://lists.mindrot.org/pipermail/openssh-unix-dev/2018...

[5] Peter Stuge replying to several other points made else thread: https://lists.mindrot.org/pipermail/openssh-unix-dev/2018...

> Even now, most people using OpenSSH on Linux have it patched for socket activation which upstream doesn't like. Yet only actually running the daemon on demand is just obviously superior.

OpenSSH implements very careful and deliberate semantics on reload which don't work well with any of systemd's service types. See Peter's messages above. Fundamentally, and similar to the case of sandboxing, there's a limit to what behaviors and features systemd can provide *outside* the process(es) it starts, even with partial solutions like sd_notify. systemd integration can introduce unnecessary complexity and confusion, if not outright barriers, to implementing the best solution. Usually this won't matter because systemd's options are usually a "step forward" (to quote Peter) in relation to the *average* quality of process management in service daemons. But OpenSSH isn't your average project. Reasonable people can disagree on the prudence of systemd integration, but many of the issues under contention aren't obvious.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 14, 2024 11:21 UTC (Sun) by bluca (subscriber, #118303) [Link]

> OpenSSH implements very careful and deliberate semantics on reload which don't work well with any of systemd's service types.

That is self-evidently not true, given the fact that all distributions have shipped openssh with the notification + socket activation out-of-tree patches for a decade or so now, and they work well. And the notification patch is now upstream, and supports reloading too - I made sure of that.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 15, 2024 9:44 UTC (Mon) by rnestler (subscriber, #160299) [Link]

> That is self-evidently not true, given the fact that all distributions have shipped openssh with the notification + socket activation out-of-tree patches for a decade or so now, and they work well.

ArchLinux apparently didn't patch openssh to link to libsystemd/lzma according to https://archlinux.org/news/the-xz-package-has-been-backdo...

> Arch does not directly link openssh to liblzma, and thus this attack vector is not possible. You can confirm this by issuing the following command: ldd "$(command -v sshd)"

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 15, 2024 10:24 UTC (Mon) by bluca (subscriber, #118303) [Link]

Sorry, meant to say "all important distributions" </flame>

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 20, 2024 10:13 UTC (Sat) by sammythesnake (guest, #17693) [Link]

I don't personally use Arch, but even if only for their awesome wiki (which I certainly *do* use) I think I'd include them in "important distributions"...

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 14, 2024 3:06 UTC (Sun) by tilt12345678 (subscriber, #126336) [Link]

Thank you very much for the detailed analysis.

I am under the impression that the "rock and the hard place" really were the OpenSSH and the systemd developers. On the one hand, "this is undesirable" is not a sufficiently scientific explanation for why a project refuses to implement a feature that is required for supporting 90% of its use-cases. On the other hand, a service manager that requires a service software to link to a huge library that does who knows what is a questionable architecture to say the least.

Next time something like this happens, do not hesitate to halt a release over a single line of code. Both sides have made their point. A resolution could not be achieved. Time to escalate. Take the problem upstairs. Community projects also have an "upstairs" - the public. Debate it there, solve it there. This really has nothing to do with fun, release-pressure, professional courtesy or anything. It affects all.

Anyhow, i understand why this was a difficult situation.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 14, 2024 3:13 UTC (Sun) by pizza (subscriber, #46) [Link]

> Time to escalate. Take the problem upstairs. Community projects also have an "upstairs" - the public. Debate it there, solve it there.

Uh, no. Until "the public" has packaging and/or commit rights, so they are literally incapable of "solving it".

And "the public" will, 99.999% of the time, chose "I don't give a F- about security, I just want it to work, the faster the better."

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 14, 2024 8:08 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link]

> On the other hand, a service manager that requires a service software to link to a huge library that does who knows what is a questionable architecture to say the least.

OpenSSH could have just merged that small patch to manually do the completion notification, without requiring libsystemd.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 14, 2024 14:33 UTC (Sun) by kleptog (subscriber, #1183) [Link]

> A resolution could not be achieved.

But a resolution *was* reached: the upstream developers rejected the patch, and all the Linux distributions included it anyway. The systemd developers have nothing to do with this, they created the notification protocol and the library but are not responsible for getting it included into daemons like OpenSSH. That's purely between the OpenSSH developers, the Linux distributions that ship it and their users.

That's the beauty of open-source: there doesn't need to be one solution. Everyone can choose their own solution. Forking a project is not a problem of itself. It's often a solution in difficult situations, like this one.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 15, 2024 17:43 UTC (Mon) by ibukanov (subscriber, #3942) [Link]

I think the point the grandparent was making was that a change to support one group of users who needed tight systemd notification support made things much less secure for all users of OpenSSH on Linux distros including those who do not benefit from the feature.

Moreover, given that OpenSSH uses the model of first starting to listen for the connection and then forking the systemd notification support can be implemented without touching OpenSSH using other notification mechanisms that systemd supported and that were used prior notify changes.

So refusion of OpenSSH to include the changes were rather reasonable and was strong indication that the changes were problematic.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 15, 2024 18:43 UTC (Mon) by pizza (subscriber, #46) [Link]

> I think the point the grandparent was making was that a change to support one group of users who needed tight systemd notification support made things much less secure for all users of OpenSSH on Linux distros including those who do not benefit from the feature.

Um, no. That (compile-time) change had no effect on distros that don't utilize systemd.

Given that every user of systemd+openssh benefited from that feature (whether or not they recognize or accept it), it had no effect on anyone else.

> Moreover, given that OpenSSH uses the model of first starting to listen for the connection and then forking the systemd notification support can be implemented without touching OpenSSH using other notification mechanisms that systemd supported and that were used prior notify changes.

Um, no. The prior mechanism was not reliable, hence the entire reason for the patch (and its nearly universal use) to begin with.

> So refusion of OpenSSH to include the changes were rather reasonable and was strong indication that the changes were problematic.

Except they've since accepted the change, which is a strong indication that this change was indeed beneficial after all.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 16, 2024 10:04 UTC (Tue) by kleptog (subscriber, #1183) [Link]

> Moreover, given that OpenSSH uses the model of first starting to listen for the connection and then forking the systemd notification support can be implemented without touching OpenSSH using other notification mechanisms that systemd supported and that were used prior notify changes.

But what about reload notification (I've received your signal to reload the config and am done) and stopping notification (I've received your shutdown signal and am on it)?

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 9:25 UTC (Sat) by ballombe (subscriber, #9523) [Link]

You know, you can always install the upstream version of openssh instead of the distro version if you trust it more. Distros just offer you more options. No need to enter an infinite loop over this.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 13:15 UTC (Sat) by mcatanzaro (subscriber, #93033) [Link]

I'm going to be blunt: your opinion is bad. If you don't patch OpenSSH, then `systemctl status` cannot reliably report whether sshd is running or not, which is a dire security and usability issue. The code released by upstream is not fit for purpose without patching, and distros have to do so to meet basic user expectations.

It's *nice* and *ideal* to have no downstream patches, but when upstream is intransigent it is sometimes essential.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 13:41 UTC (Sat) by bluca (subscriber, #118303) [Link]

> If you don't patch OpenSSH, then `systemctl status` cannot reliably report whether sshd is running or not

Until the next upstream release, which will finally include the functionality natively, to be fair. It only took a world-ending supply chain attack to get that 8-years-old patch merged. Now if we just could get Jia Tan to compromise another dependency of openssh, maybe we could get the socket activation patch merged too :-)

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 15:17 UTC (Sat) by tilt12345678 (subscriber, #126336) [Link]

> If you don't patch OpenSSH, then `systemctl status` cannot reliably report whether sshd is running or not, which is a dire security and usability issue.

Sounds to me more like a deficiency of systemd than one of OpenSSH ... SCNR! Notwithstanding this, if OpenSSH now has decided that they will perform the integration and take care of it in the log run, Kudos to them. It needs to be done. If i would have to guess, i'd say Linux with systemd is the most frequent environment OpenSSH is run in.

I hope that other such controversies do not require (let me quote another post here) "a world-ending supply-chain attack" to be resolved. Please, everyone, be more cooperative.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 17:45 UTC (Sat) by flussence (subscriber, #85566) [Link]

Can you provide an example of these “dire security and usability issues” that apparently every other supervision system suffers from?

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 18:46 UTC (Sat) by Wol (subscriber, #4433) [Link]

Well, I don't know if it's been fixed, but I think it was Bind.

As part of its shutdown sequence, it shut down networking. Then it had a habit of hanging.

When it's headless, in a lights-out co-lo, that is an EXPENSIVE problem, unless you can log in to the UPS, kill power to the server, and then send a "wake on wan" command.

Systemd, on the other hand, can be configured to say "this system is going down, and if something hangs it WILL NOT interrupt the shutdown.

Whether you like systemd or not, a *functioning* pid1 is much simpler, smaller, and easier to understand than most (all?) *functional* alternatives. A bare systemd is much bigger than a bare SysVInit, but as soon as you throw in all the service scripts, SysV grows like Topsy. With massive amounts of cargo-culted bash code.

Systemd, if you tell it to go down, IT GOES DOWN.

Cheers,
Wol

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 21:16 UTC (Sat) by flussence (subscriber, #85566) [Link]

Right... but what does any of that have to do with startup notification? We're talking about that here. The whole exploit vector thing. You don't need libsystemd to shut something down cleanly, you use a cgroup for that.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 21:44 UTC (Sat) by Wol (subscriber, #4433) [Link]

Sorry. But you mentioned "dire usability issues". Okay you were concentrating on "securilty" but I picked up on "usability" and - you have to admit - a system that fails to shut down in a lights-out co-lo IS a dire usability issue.

Cheers,
Wol

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 14, 2024 0:00 UTC (Sun) by cesarb (subscriber, #6266) [Link]

> and - you have to admit - a system that fails to shut down in a lights-out co-lo IS a dire usability issue.

Even a system which fails to shut down in a normal office can be a problem.

Story time.

On a former company, for some reason one particular desktop failed to shut down correctly, and when that happened, some stuck networking daemon in it caused a broadcast flood. Since the office network was flat, the broadcast flood also hit the embedded device which controlled the keycard entry to the office, which after some time slowed down and then failed. Since the managers had left the backup device (a remote control keyfob which could also cut power to the magnetic locks) locked within their office, everyone was stuck outside.

As luck would have it, that security system was badly designed (by the former occupier of that office space), and if you knew where to push, you could open a hidden cover on the *outside* of one of the doors, which gave access to the wiring (and the power supply and battery) for that door. A few borrowed tools later, and one of the wires was cut (I chose to cut the red one, because it's tradition, but any of them would have worked), permanently disabling the lock so everyone could get in.

The problem was then properly solved by enabling broadcast storm control in the switch, isolating the keycard control device in a separate "things" VLAN, and convincing the managers that they really need to take that keyfob home with them. But another thing that could have prevented the issue? This was pre-systemd (i.e. upstart) Ubuntu; if it were modern Ubuntu, systemd would have timed out, force killed the stuck daemon, and force shut down the desktop, long before the embedded device could get too overloaded to recover on its own.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 14, 2024 17:51 UTC (Sun) by flussence (subscriber, #85566) [Link]

I'm not questioning that. But asserting that a system where daemons aren't linked against systemd is insecure requires extraordinary evidence, and that has not been forthcoming. I mean, the opposite is obviously true. It's right there in the headline.

And — assuming I'm not dealing with a load of hor-^WFUD here, which I'm really starting to think this is — I would really like to know details of how the init system I'm using is supposedly insecure in a dire way for lack of startup micro-optimisations so I can _fix_ it. I'm sure everyone else hacking on non-systemd systems wants to know what the deal is too.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 14, 2024 14:34 UTC (Sun) by jejb (subscriber, #6654) [Link]

> Do not patch OpenSSH unless you can be sure that the changes make it upstream. Do not patch OpenSSH without monitoring the consequences; if such a monitoring is not feasible, then do not patch OpenSSH. Do not patch OpenSSH just because others do it. Do not patch OpenSSH.

This is completely the wrong takeaway: If you imagine a world where openssh had accepted the patch and systemd could be a configured dependency in upstream it would have done nothing to disrupt the attack ... everything would still have happened the way it did. More theoretical scrutiny on libsystemd linked to openssh wouldn't have flagged the xz dependency as a vulnerability.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 14, 2024 15:33 UTC (Sun) by ovitters (guest, #27950) [Link]

To me it that poster lacks packaging experience and is pretending things are either black or white. Further, awfully easy to tell other people what to do, say things after the fact, plus say things while standing from the sideline. Or like the Dutch saying: https://en.wiktionary.org/wiki/de_beste_stuurlui_staan_aa...

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 14, 2024 18:43 UTC (Sun) by ballombe (subscriber, #9523) [Link]

Also given the amount of resource expended to mount this attack, the particular are irrelevant,
the attackers could have performed it in a lot of different ways.
From their point of view it must be an abject failure, unless they were trying to make a point.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 14, 2024 20:54 UTC (Sun) by smoogen (subscriber, #97) [Link]

xz is used by a LOT of utilities. Openssh was one path, but this attack could have gone for rpm, dpkg, or a myriad of other root level tools which could have allowed for a massive compromise of systems. I think this was the reason this utility was chosen because it could have allowed a myriad of compromises in different places which might take years to untangle.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 15, 2024 7:38 UTC (Mon) by calumapplepie (subscriber, #143655) [Link]

The patch to OpenSSH was not a problem. It did not malfunction, it did not blow up or burn or die. The specific exploit path taken by this vulnerability involved this patch, yes. But it did not need to. The reason the patch was valid was because it coupled something already pretty security critical (systemd) into openssh. The only way linking in this extra library can be harmful is if systemd is itself compromised. If systemd is fundamentally compromised, then there are many other paths to exploitation.

Perhaps many of those, done correctly, would be more noticeable. Overwriting a critical function in a secure program by linking in a compromise library has a lot more potential to go unnoticed than, say, an extra file in authorized-keys. But if you're looking for it, this can be checked for statically fairly easy; and I wouldn't be surprised if tools popped up to do just that in the near future. Heck, I might try my hand at writing one; a little script over objdump -T, wiring it into debian maintainer tools... etc.

The patch to OpenSSH did not enable the vulnerability, and thus was not flawed. In fact, I understand it has been accepted upstream. Don't rail against patches you don't fully understand.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 15, 2024 10:39 UTC (Mon) by kleptog (subscriber, #1183) [Link]

> The patch to OpenSSH did not enable the vulnerability, and thus was not flawed. In fact, I understand it has been accepted upstream. Don't rail against patches you don't fully understand.

I agree with you, but just wanted to clarify some points. AIUI the patch included by upstream is not the patch distributions used. Two different patches for the same problem.

The distribution patch was adding a single line of code plus some autoconf magic to configure it. For packagers this is the obvious choice since they don't necessarily understand all the code and it's a minimal "obviously correct" patch.

The patch applied by upstream [1] avoids linking an external library and vendors the sd_notify() function, tweaking it for the OpenSSH code base. Both the distribution patch and a version of this patch were offered to the OpenSSH developers in 2018, and rejected. This patch is much larger and not as "obviously correct" so given a choice, packagers are going to go for the short patch if they get a choice.

[1] https://github.com/openssh/openssh-portable/commit/08f579...

It's interesting to compare this to the Linux kernel. There there is social pressure to not have external patches and to upstream everything. There are all sorts of good reasons for this. But the flip-side is that the kernel developers have to not randomly stand in the way of feature requests just because they don't care about the use-cases. The OpenSSH developers have opinions about the use-cases (e.g. socket activation) but also want to discourage downstream users from patching. You can't have it both ways.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 15:54 UTC (Sat) by DOT (guest, #58786) [Link]

Something that came to my mind recently: where is the FBI in this whole adventure? I haven't heard any talk about tracking down the criminals who tried to break into the world's computer systems.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 17:00 UTC (Sat) by NYKevin (subscriber, #129325) [Link]

I imagine they are currently going through the entire Intelligence Community trying to figure out if "we" did it (for some value of "we"). But it ultimately doesn't matter - if it's a US intelligence agency, there's not much the FBI is going to do about that, and if it's some other country's intelligence agency, then there's also not much the FBI is going to do about that. I suppose it's possible that it's some large NGO, which the FBI could prosecute, but that strikes me as unlikely.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 13, 2024 18:39 UTC (Sat) by branden (guest, #7029) [Link]

Dirk Mueller isn't wrong, but he's...spitting into the wind.

Early in the piece he goes to a familiar well. To his credit, he critiques it.

“Given enough eyeballs, all bugs are shallow.” -- "Linus's Law", glibly popularized by Eric Raymond

But there's a bigger problem. Let's call it the engineering manager's corollary to Linus's Law

"Eyeballs cost money. How are eyeballs going to deliver shareholder value THIS QUARTER? I've got a promotion or lateral move lined up that depends on shipping on time. Are these eyeballs going to get product shipping faster? Then f*** eyeballs."

Silicon Valley, Redmond, the Grand Duchy of Luxembourg (site of SuSE HQ these days)--these places will not solve the problem. They are constitutionally unable to connect what needs to be done to their bottom line.

I predict that Andreas Freund's next performance review at Microsoft will go spectacularly well for him. He has my thanks and congratulations!

After that, if he continues in his recent vein, he's going to have to go to work elsewhere if he desires advancement.

That won't be his fault or even his boss's fault. Tech firms are structurally incapable of incentivizing robust software quality.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 15, 2024 10:48 UTC (Mon) by walex (subscriber, #69836) [Link]

"Tech firms are structurally incapable of incentivizing robust software quality."

That is not really different from any other business: the incentive is always to dump (externalize) costs on others and to defer costs as much as possible into the future. Whether it is toxic pollution of air, water, soil, or pollution of software with security and reliability issues.

The outcome is that we have toi learn two things: #1 survive unreliable (including extensively backdoored) products and processes #2 find some mechanisms to incentivice "good enough" quality. Neither are easy tasks.

As to software backdoors I would expect that nothing has changed from the past: in the past a significant percentage of every important organization have been moles or informers for management or for rivals; if the spy services of the major players are doing their job right I would expect that 5-10% of technical employees at Google, Microsoft, Facebook, NVIDIA, ... and most significant open source projects to be moles or informers. It is both very easy to arrange and very cheap, and much less risky than planting moles and informers into military or government organisations. Not for nothing both the RF and PRC governments have been imposing that any critical hw or sw be entirely designed and manufactured within their countries, and the UK government only accepted Huawei products where each line of sw in them had been vetted by government appointed auditors.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 15, 2024 11:37 UTC (Mon) by paulj (subscriber, #341) [Link]

And it's not theoretical. We know of the Greek Vodafone foreign espionage case, used to spy on the Greek government. It was conducted via a backdoor and rootkits into the Ericsson phone exchange machines, allowing surreptitious and unlogged intercepts. The evidence available suggests it was conducted by the USA.

As a result of this illegal wiretapping, a Greek manager in Vodafone died. Either by suicide or murder.

What we need to take away from the XZ Backdoor (openSUSE News)

Posted Apr 15, 2024 17:03 UTC (Mon) by hkario (subscriber, #94864) [Link]

> That is not really different from any other business: the incentive is always to dump (externalize) costs on others and to defer costs as much as possible into the future. Whether it is toxic pollution of air, water, soil, or pollution of software with security and reliability issues.

And the solution is the same as with all those other industries. Is your widget safety critical? Then it needs to follow verified 3rd party certified testing. That's done through legislation.

That's also the reason why Americans don't like "legislation" as a solution: they have been under 40 years of constant propaganda telling them how "the gubbement" doesn't work, paid for by the very companies that want to make stuff as cheaply as possible because next quarter financial results are the only thing that matters. Boeing is the poster child of that approach.


Copyright © 2024, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds