|
|
Subscribe / Log in / New account

2023 PSF annual impact report

The Python Software Foundation (PSF) has announced its annual impact report for 2023. The report includes updates from PSF staff as well as summaries of the foundation's activities, financials, and infrastructure. The PSF celebrated the 20th anniversary of PyCon US, distributed more than $370,000 in grants, and enjoyed impressive traffic on PyPI:

In 2023 PyPI saw a 45% growth in download counts and bandwidth alike, serving 603,378,275 downloads for the 516,402 projects hosted there requiring 747.4 Petabytes of data transfer, or 189.6 Gbps of bandwidth 24x7x365.

See the full report for a breakdown of grant disbursements and trends, PSF expenses, and high-level plans for the rest of 2024.



(Log in to post comments)

2023 PSF annual impact report

Posted May 6, 2024 22:48 UTC (Mon) by flussence (subscriber, #85566) [Link]

And likely 99.5% of that traffic comes from a tiny number of users of containers, proprietary OSes and CI services, boiling the oceans with cut-n-paste automation for the sake of one of those repo status badges nobody cares about. Like the AOL era's web page hit counters but they each set an entire barrel of oil on fire.

PyPI are doing a heroic service in the face of whatever causes all that traffic. There are other language ecosystems (not naming names, but their owners have several orders of magnitude more resources than the PSF and most other mainstream languages combined) that dump that burden directly on their contributors' upstream git servers to save a penny. I imagine Python will still be around and popular long after those others have crumbled.

2023 PSF annual impact report

Posted May 7, 2024 6:30 UTC (Tue) by LtWorf (subscriber, #124958) [Link]

They could just start to return 429 Too Many Requests and "encourage" people to do caching.

But my suspicion is that google is happy to spend that money to obtain the data of what are companies using.

But fun fact, if you do "pip install xxx" it will connect to check the version and might use a local cache. That won't count for the package download counter.

I think websites like snyk, that decide how good a python module is also based on how many pip install downloads it has are a problem. It means that modules that have knowledgeable users will show up as "unpopular, don't use".

2023 PSF annual impact report

Posted May 7, 2024 22:52 UTC (Tue) by NYKevin (subscriber, #129325) [Link]

Hot take: If you're hitting an endpoint that is not intended to be hit by browsers, you should get 429'd very aggressively unless you pass If-Modified-Since or If-None-Match (or e.g. an API key that proves you're paying for the service, or something along those lines, in which case you should get rate limited according to whatever ToS you agreed to when you provided your payment information). Apparently, there is also a common pattern of making up dates in the distant past and passing them as If-Modified-Since, or making up invalid ETags and passing them as If-None-Match, which should both be treated identically to passing no header (i.e. you still get rate-limited).

Rationale: If you are going to use somebody else's web service, and you haven't paid for the privilege of doing so, you can go to the bother of implementing a sane caching strategy.

Unfortunately, I suspect some people would resort to using BeautifulSoup or something to parse the human-readable web page, rather than knocking out a basic HTTP(S) cache in an hour or two. This is why we can't have nice things.

2023 PSF annual impact report

Posted May 7, 2024 23:07 UTC (Tue) by LtWorf (subscriber, #124958) [Link]

The entirety of github CI doesn't cache. I don't believe that microsoft doesn't have the resources to implement caching, or that it's cheaper for them to re-download everything.

openssf and their scorecard stuff impose people to use github and their CI among other things.

If there was a will to not be so wasteful there, I think it would have been done.

2023 PSF annual impact report

Posted May 7, 2024 23:31 UTC (Tue) by NYKevin (subscriber, #129325) [Link]

Why can't PyPI and other endpoints that get called into from GitHub simply implement 429 errors unilaterally? It's not as if Microsoft is just going to tell everyone to stop using pip from their CI flows.

Yes, in the short run, some builds would break, but if *every* Python-related build on GitHub breaks at the same time, I don't think Microsoft is just going to ignore that.

2023 PSF annual impact report

Posted May 8, 2024 5:55 UTC (Wed) by LtWorf (subscriber, #124958) [Link]

I think your assumption that microsoft and google are not colluding is incorrect.

Both companies are heavily involved and cooperate a lot.

For example google paid the development of "trusted publishers" for pypi. Which only works from github. Github's CI is in a way the preferred way to upload on pypi.

I wouldn't be surprised if in 1-2 years the "important" (defined by download probably) projects would be forced to be uploaded on pypi just via github.

This would signify no more signatures.

But the reason why I think google would want this, is because github can mass-ban russians or whomever the USA doesn't like next time, so that modules uploaded this way have a guarantee of having been written by "the good guys™" and can be used by government contractors.

A google employee (I think it was one of the creators of go) has stated as a goal to eliminate anonymous open source contributors.

I think that microsoft and google are cooperating towards slowly reaching that goal, because it interests both of them.

Of course they can't just put the requirements all at once, because people would just move. But they can sloooooowly add more and more requirements.

2023 PSF annual impact report

Posted May 8, 2024 6:36 UTC (Wed) by NYKevin (subscriber, #129325) [Link]

Disclaimer: I work for Google, not on PyPI or any of this stuff. Some of my work does relate to backends of "social-ish" services like Photos and Drive, and it was previously a backend of Plus.

Frankly, this sounds like a conspiracy theory to me. Not because I believe that Google is benign, but because:

1. I'm skeptical that Google's upper management has the attention span for it.
2. I'm skeptical that they can generate significant amounts of alpha[1] from it.
3. The "real names" thing was definitely... a thing, ~5-10 years ago, but these days, Plus no longer exists. I imagine there's probably some people on the security side of the fence who really wish pseudonymous FOSS contributors would go away (especially after the whole xz fiasco), but I don't believe they have the kind of top-down all-hands-on-deck pull within the company that you're suggesting.
4. GitHub is already required to comply with sanctions laws regardless of what you or I might think about them. This obligation attaches *now.* They have to stop doing business with such people *today.* Collusion with Google simply has no bearing on whether GitHub, as a US company, has to follow US laws.
5. Uh, we were talking about rate limiting? What does Google have to do with the price of rice in China?

[1]: https://en.wikipedia.org/wiki/Alpha_%28finance%29

2023 PSF annual impact report

Posted May 8, 2024 10:02 UTC (Wed) by LtWorf (subscriber, #124958) [Link]

It is a conspiracy theory :) Just remember that conspiracies do exist, so conspiracy theories can be right from time to time. I don't think it's particularly useful to lump together "Obama is actually an alien shapeshifter" with "huge company is doing something bad".

Why is google financing pypi? Why is google gently pushing for python projects to be on github (and github only, no other git website)? What is the economic return of that for google?

> GitHub is already required to comply with sanctions laws regardless of what you or I might think about them

Sure. And google benefits from it if more and more packages are coming from github instead of somewhere else, precisely because they know that microsoft will comply with the sanctions. Isn't that an advantage for google to not have to individually make sure that that the dependencies they use don't come from russian developers?

And isn't it easier to apply sanctions if you require a valid identification?

> What does Google have to do with the price of rice in China?

I have no idea what you're talking about.

2023 PSF annual impact report

Posted May 10, 2024 2:06 UTC (Fri) by NYKevin (subscriber, #129325) [Link]

> Sure. And google benefits from it if more and more packages are coming from github instead of somewhere else, precisely because they know that microsoft will comply with the sanctions. Isn't that an advantage for google to not have to individually make sure that that the dependencies they use don't come from russian developers?

I am not going to comment on what is or is not easier for Google - that's well above my paygrade and probably confidential anyway.

Having said that... I'm concerned by the attitude reflected in this line of thinking. US sanctions laws do not merely extend to large corporations like Microsoft or Google. They apply to every person or corporation that lives, works, or does business in the US. OFAC is not shy about asserting extraterritorial jurisdiction over foreign people or businesses, either.[1]

I'm not going to comment on whether this is a good thing or a bad thing. My point is, it is a thing. Anyone operating a forge, FOSS repository, or anything else that in any way touches the US economy is probably required to comply with US sanctions. If anyone has been getting away with not doing that, then they need to be aware that OFAC could discover and crack down on them at any time, and should probably consult an attorney for specific legal advice (hint: do not post publicly-visible comments wondering about the legality of specific arrangements - that's called "discoverable evidence of intent," and could be used against you).

And yes, I know that many people are doing this as a private hobby and have absolutely no desire (and in many cases, no money) to talk to a lawyer. I don't make the rules, I'm just trying to make sure everybody is aware of them.

> I have no idea what you're talking about.

The phrase "What does [x] have to do with the price of rice in China?" is a common idiom for "[x] is not relevant to the subject at hand."

[1]: https://www.justice.gov/opa/media/1341411/dl?inline

2023 PSF annual impact report

Posted May 10, 2024 17:09 UTC (Fri) by LtWorf (subscriber, #124958) [Link]

It seems to me that despite the fact that you work for google and I do not, you aren't in possession of any inside information and are just making hypothesis, same as me. Am I correct?

> They apply to every person or corporation that lives, works, or does business in the US

I don't live in the USA. So I think for me it's better to use codeberg, which is also not in USA. And since it's not a business I doubt it will ever do business with USA, and if your government wants to block it and stop accessing my software I really wouldn't care. I care if your government decides to block me from accessing my own software :)

2023 PSF annual impact report

Posted May 10, 2024 17:29 UTC (Fri) by farnz (subscriber, #17727) [Link]

The US tends to take an expansive view of "does business in the US" for the purposes of sanctions regulations; you do "do business" in the US, since you have an account on a US website (LWN.net), and from the US point of view, that's enough of a link.

Question is whether the US could enforce penalties against you.

2023 PSF annual impact report

Posted May 10, 2024 23:20 UTC (Fri) by LtWorf (subscriber, #124958) [Link]

From my point of view the loss of my lwn account would be less dramatic than access to my own projects.

Not all accounts have the same value.

2023 PSF annual impact report

Posted May 12, 2024 14:29 UTC (Sun) by farnz (subscriber, #17727) [Link]

It's not loss of your LWN account alone, though - it's also loss of access to all financial instruments that you've ever connected with this LWN account.

Now, maybe you pay your subscription purely with gift cards bought through a broker for cash; but if you use a credit or debit card, the US takes the view that it's entitled to drain those accounts to pay your penalties for being a "terrorism sponsor". It also believes that it's entitled to demand your identity from your bank, and arrange to have you extradited for your role in international terrorism.

So, unless you go to some quite challenging lengths to avoid creating a link between your domestic identity and your LWN account, just paying for an LWN subscription is enough to cause you to be at risk of the US's sanctions laws.

2023 PSF annual impact report

Posted May 12, 2024 2:36 UTC (Sun) by NYKevin (subscriber, #129325) [Link]

> you aren't in possession of any inside information and are just making hypothesis, same as me. Am I correct?

If I was in possession of inside information, I would not post it here. My disclaimer is a disclaimer, not a claim to greater knowledge.

2023 PSF annual impact report

Posted May 12, 2024 15:35 UTC (Sun) by marcH (subscriber, #57642) [Link]

> It is a conspiracy theory :) Just remember that conspiracies do exist, so conspiracy theories can be right from time to time. I don't think it's particularly useful to lump together "Obama is actually an alien shapeshifter" with "huge company is doing something bad".

Once a "conspiracy" is proven and admitted (= most of the time because you can't keep a secret with more than 2 people) and it becomes a scandal, then it ceases to be interesting. The most striking example is Big Pharma: they've been caught countless times, most recently with opioids: 100,000s of estimated casualties!

https://en.wikipedia.org/wiki/List_of_largest_pharmaceuti...

But conspirationists never talk about these countless, actual scandals because it does not fit their narrative. They need to make something up: only secrets kept by 1,000s of people (which is obviously impossible) are "exciting"!

2023 PSF annual impact report

Posted May 12, 2024 15:54 UTC (Sun) by marcH (subscriber, #57642) [Link]

> But conspirationists never talk about these countless, actual scandals because it does not fit their narrative

... or because they don't actually believe their own "conspiracy" and are just trolls paid to distract attention away from real and proven hence "boring" issues.

2023 PSF annual impact report

Posted May 13, 2024 5:42 UTC (Mon) by LtWorf (subscriber, #124958) [Link]

It can take several years for a conspiracy to be revealed. Or it could never happen. After all we only know the ones that were revealed.

https://en.wikipedia.org/wiki/Tuskegee_Syphilis_Study

https://en.wikipedia.org/wiki/Operation_Gladio

Both of them involved more than 2 people and for more than just a few days.

it seems to me that the assumption that conspiracies are impossible collides with reality.

2023 PSF annual impact report

Posted May 13, 2024 14:13 UTC (Mon) by marcH (subscriber, #57642) [Link]

It depends but the main factor is really how many people are involved. I agree you can keep a secret with more than two people. Not for long if thousands of people are involved, it will statically leak fairly fast.

Also depends "how bad" it is and how guilty people feel and how scared people are: the Mafia is obviously much better at keeping "large" secrets secret than Boeing (although some recent deaths look a bit worrying, I hope they're being investigated)

2023 PSF annual impact report

Posted May 8, 2024 19:47 UTC (Wed) by rgmoore (✭ supporter ✭, #75) [Link]

I imagine there's probably some people on the security side of the fence who really wish pseudonymous FOSS contributors would go away (especially after the whole xz fiasco),
I sincerely doubt requiring real names would have helped with the xz fiasco. It might help with ordinary criminals trying to scam a quick buck, but a state-level attacker- and the xz attack seems to have been the work of a state-level attacker- can create a fake persona that would pass even very rigorous tests. That's not to say stopping ordinary criminals is worthless, just that any kind of identity check a place like GitHub is likely to implement won't stop a dedicated, well-resourced attacker.

2023 PSF annual impact report

Posted May 8, 2024 20:12 UTC (Wed) by pizza (subscriber, #46) [Link]

> That's not to say stopping ordinary criminals is worthless, just that any kind of identity check a place like GitHub is likely to implement won't stop a dedicated, well-resourced attacker.

It's not enough to _present_ credentials, there has to be a way to _validate_ those credentials.

Otherwise "dedicated, well-resourced attacker" just means "Anyone with photoshop"

2023 PSF annual impact report

Posted May 8, 2024 20:19 UTC (Wed) by mb (subscriber, #50428) [Link]

What does validation even mean, in this context?
It means exactly nothing.

Some random person presents a random passport from a random country? Useless.
Why would this make *any* statement about the trustworthiness of the person?

People used to be married to people who betrayed them. So, why would "validating credentials" help in any way?
It doesn't.

2023 PSF annual impact report

Posted May 8, 2024 22:15 UTC (Wed) by pizza (subscriber, #46) [Link]

> What does validation even mean, in this context?

It means asking the issuing agency if the credentials are valid, and taking appropriate steps to make sure the credentials match the person presenting them.

> Why would this make *any* statement about the trustworthiness of the person?

It only tells you that the person with the credentials is who they claim to be, not that they are trustworthy, The latter is a largely impossible task, as past performance does not guarantee future results.

But more importantly than trustworthiness, this gives you legal recourse [1] should they "betray" that trust by doing BadThings(tm).

[1] ie some sort of enforceable penalty or punishment... after the fact, of course. Which is the basis of every legal system.

2023 PSF annual impact report

Posted May 8, 2024 22:41 UTC (Wed) by mb (subscriber, #50428) [Link]

>It only tells you that the person with the credentials is who they claim to be

No, it does not. Not at all.
It only tells, that another "authority" says so. You still have to trust the "authority".
That's the TLS certificate "authority" BS game.

2023 PSF annual impact report

Posted May 8, 2024 23:05 UTC (Wed) by pizza (subscriber, #46) [Link]

> It only tells, that another "authority" says so. You still have to trust the "authority".

Well, duh -- Except that this "authority" has the full force of law (and a very literal army) standing behind it.

> That's the TLS certificate "authority" BS game.

No. This is more akin to DNSSEC or DANE, where there is precisely one entity allowed to/capable of issuing (and more importantly, validating) these *state-issued* credentials.

Meanwhile, CAs can freely issue anything to anyone.

Another key difference is that, unlike the CA mess, there are major (ie involving jail time) for falsifying these credentials. And you have to jump through some non-trivial hoops to get them.

Anyway. my point, again, is that without a mechanism to validate/authenticate said credentials, they're not worth the electrons they're made up of.

2023 PSF annual impact report

Posted May 8, 2024 23:16 UTC (Wed) by mb (subscriber, #50428) [Link]

>Except that this "authority" has the full force of law (and a very literal army) standing behind it.

Useless.
There are over a hundred countries, laws and armies that I don't trust.
"Authoritative" documents from them are useless.

2023 PSF annual impact report

Posted May 9, 2024 13:51 UTC (Thu) by pizza (subscriber, #46) [Link]

> There are over a hundred countries, laws and armies that I don't trust.

Congratulations, you're doomed to only ever to work with people you personally know.

And even then, you've already established you can't trust them. Or anyone.

Enjoy your life of total isolation.

2023 PSF annual impact report

Posted May 9, 2024 14:02 UTC (Thu) by mb (subscriber, #50428) [Link]

Well, I said the exact opposite.
What counts is personal trust, not documents from "authorities".
You can't establish personal trust with certification from "authorities".

2023 PSF annual impact report

Posted May 9, 2024 14:30 UTC (Thu) by pizza (subscriber, #46) [Link]

> You can't establish personal trust with certification from "authorities".

According to you, personal trust isn't um, trustworthy either:

"People used to be married to people who betrayed them"

Trust is, until it isn't.

The best you can _ever_ do is have some recourse after the fact, and hope it either acts as a sufficient deterrent or can compensate you for your damages/loss. Decry this principle all you like, but it is the basis [1] of every legal system out there.

[1] Granted, the true basis of _every_ system is the explicit threat of force against those that don't comply with (or otherwise violate) the rules.

2023 PSF annual impact report

Posted May 9, 2024 14:57 UTC (Thu) by mb (subscriber, #50428) [Link]

>Trust is, until it isn't.

Yes. Trust is never absolute.

I just wanted to say that an "authorization" from government X (place your favorite distrust country here) is useless for me.
I would not trust such authorization from most countries in the world.

Improving trust by asking an untrusted third party is not going to work.

Requiring such authorization just introduces huge barriers into projects for no good reason.

I would probably also have fallen for Jia Tan, if I she had attacked me. But no level of government authorization could have prevented it.
Requireing such things makes the situation *worse*. Here is my certificate from government X. How dare you don't trust me! It's written *here* that I am trustworthy.

That is not how trust works. At all.

2023 PSF annual impact report

Posted May 10, 2024 15:50 UTC (Fri) by kleptog (subscriber, #1183) [Link]

> Requireing such things makes the situation *worse*. Here is my certificate from government X. How dare you don't trust me! It's written *here* that I am trustworthy.

That makes no sense. Governments do not issue statements of trustworthiness. They issue proofs of identity, which you can do with what you like. You not trusting the proof of identity is orthogonal to whether you trust someone. Trust also relative: I trust most people not to want to kill me, but the people I would trust to pay back a €1000 loan is much smaller.

There are places where the fact you have passport X makes you (somehow) more trustworthy than someone with passport Y, but on an individual level that makes no sense whatsoever. Context matters.

2023 PSF annual impact report

Posted May 10, 2024 16:08 UTC (Fri) by mb (subscriber, #50428) [Link]

>Governments do not issue statements of trustworthiness.

Next time please read the full text and not only the last sarcastic sentence. Ok? :)

2023 PSF annual impact report

Posted May 10, 2024 20:01 UTC (Fri) by pizza (subscriber, #46) [Link]

> That makes no sense. Governments do not issue statements of trustworthiness.

Yes they do; it's called a security clearance.

But that's another matter entirely.

2023 PSF annual impact report

Posted May 10, 2024 21:55 UTC (Fri) by kleptog (subscriber, #1183) [Link]

> > That makes no sense. Governments do not issue statements of trustworthiness.

> Yes they do; it's called a security clearance.

I don't know about all jurisdictions, but at least here what such a clearance means is "we did a bunch of research on someone and didn't find any red flags". And then there are laws that say certain information can be shared with such people. That doesn't mean those people are actually trustworthy, just that from a risk management perspective the risk is low.

So guess you could say they issue "this person is 99% chance trustworthy, and we can lock them up if they break trust" certificates. Which from a government's point of view is good enough for their purposes. It's of no use whatsoever for open-source projects though.

2023 PSF annual impact report

Posted May 11, 2024 14:28 UTC (Sat) by pizza (subscriber, #46) [Link]

> So guess you could say they issue "this person is 99% chance trustworthy, and we can lock them up if they break trust" certificates. Which from a government's point of view is good enough for their purposes. It's of no use whatsoever for open-source projects though.

Why is that of "no use whatsoever" for open source projects? I mean, that's the same principle F/OSS licences and all other legal constructs (and I'd argue nearly all human interactions) are based on -- folks who violate the rules get punished (either directly or otherwise enforced) by the state. (And, I mioght add, this is the ultimate goal of all "Real names" policies. If its proponents say otherwise, they're either lying or blithering idiots)

I mean, that "99% trustworthy, we'll lock them up if they break trust" is good enough for folks that deal with actual life-and-death situations -- Are you seriously saying that F/OSS development should be held to a _higher_ standard of trust than a doctor or military general?

2023 PSF annual impact report

Posted May 11, 2024 14:50 UTC (Sat) by mb (subscriber, #50428) [Link]

>Why is that of "no use whatsoever" for open source projects?

Because it's a ridiculous process.
If an Open Source project requires any sort of state based authorization, then I'd rather not contribute than go through this nonsense. And I bet I'm not the only one.
So you are effectively reducing people working on the things and you are making things worse by adding this process.

Not even my employer, for whom I develop safety critical software, requires such nonsense. I have not shown any state authorization document to them. I could have sent anybody under my name.

2023 PSF annual impact report

Posted May 11, 2024 16:26 UTC (Sat) by pizza (subscriber, #46) [Link]

> Not even my employer, for whom I develop safety critical software, requires such nonsense. I have not shown any state authorization document to them. I could have sent anybody under my name.

That is, IMO, completely fair. And I also completely agree with you.

It's a ridiculous amount process that _still_ won't guarantee that someone can be "trusted" even in the short term.

...Which is why any proposal along the lines of "developer trustworthiness" should be jettisoned with extreme prejudice -- Frankly, even entirely trustworthy well-intentioned people still make mistakes with potentially disastrous consequences (see: log4j debacle) so we have to be able to deal with those messes regardless.

Instead, we need to focus on (early) detection, containment, and (*always* after-the-fact) cleanup.

...But keep in mind that one facet of post-facto cleanup is using the legal system to punish ne'er-do-wells, which isn't possible without tying psuedonyms to real-world identities, which in turn currently requires a _lot_ of work so is only done for particularly egregious acts (eg where death, serious injury, or very large monetary losses occurred). Having some sort of cross-jurisdiction-verifiable [1] identification requirement would make that much easier, and thus make it possible to go after lower-level offenders (and the resulting deterrent effects[2]). Again, this sort of thing is a core precept of both civil and criminal law.

Of course, when the same entity that carries out the punishment also gets to define what is and isn't a punishable offence, there is a significant (and oft-demonstrated) potential for abuse. So there are clearly pros and cons, but ultimately each society has to debate those and determine for themselves how they will balance those opposing principles.

[1] And by that I mean actually *verifiable*, not "send us an easily-photoshopped image of a physical ID card"
[2] A good example of this is how Hollywood has evolved its efforts to combat "piracy"; I personally know several folks who stopped routinely pirating everything once their ISP sent them "do this again and you'll get disconnected, and oh, there's no competition so good luck getting online with a different provider" letters,

2023 PSF annual impact report

Posted May 11, 2024 17:05 UTC (Sat) by mb (subscriber, #50428) [Link]

>But keep in mind that one facet of post-facto cleanup is using the legal system to punish ne'er-do-wells

Punishment gets us nowhere.
Does it reduce the effects of the attack?
No.
Does it ensure such crimes happen less?
No. There is no deterrence for crimes above a certain steal-bubblegum-threshold.
Does it reduce the possibility of the perpetrator doing it again?
No. In some countries criminals in prisons even get *more* criminal.

Punishment is hard and expensive to do. Especially, if you don't even live in the country of the perpetrator.
And then, what do you get? Nothing.

Some countries criminal laws are not even based on punishment as such.

Yes, I would personally also like to know who Jia Tan really is. But what would we do with this information? I can't think of anything good. If he was Chinese, I could immediately see how stupid people would start to generalize and make stupid conclusions. That would be bad. Especially, as we have such people in governments these days.

It would not improve things to know who Jia Tan is. Except for me personally knowing and having a "good" feeling about my prejudices being "right".
But it could have serious drawbacks for a society to know it.

2023 PSF annual impact report

Posted May 11, 2024 17:32 UTC (Sat) by pizza (subscriber, #46) [Link]

> No. There is no deterrence for crimes above a certain steal-bubblegum-threshold.

You are correct -- except that we're currently nowhere near that bare minimal threshold.

2023 PSF annual impact report

Posted May 11, 2024 17:42 UTC (Sat) by mb (subscriber, #50428) [Link]

Yes. We are nowhere near it. We are well above it.
There is no deterrence *above* the threshold. That sounds counter intuitive at first. But it actually isn't. People don't think about the possible law consequences before committing a big crime, because they expect not to be caught in the first place.

2023 PSF annual impact report

Posted May 12, 2024 14:18 UTC (Sun) by pizza (subscriber, #46) [Link]

> Yes. We are nowhere near it. We are well above it.

We are saying the same thing, from opposite perspectives.

You can't have punishment without first getting *caught*, and since the odds of getting caught are so small, any potential pumishment has no deterrent effect.

However, it's been repeatedly demonstrated that requiring "real names" [1] considerably increases the odds of getting caught and therefore punished.

[1] Even minimally verified

2023 PSF annual impact report

Posted May 13, 2024 5:15 UTC (Mon) by LtWorf (subscriber, #124958) [Link]

It also increases the chance of someone else using your name and getting you punished.

2023 PSF annual impact report

Posted May 13, 2024 14:17 UTC (Mon) by pizza (subscriber, #46) [Link]

> It also increases the chance of someone else using your name and getting you punished.

As I've repeatedly said (in other threads, in this thread, and even in the message you're replying to) "real names" have to be at least "minimally verified" to have even the possibility of a positive outcome.

(I've also said that you need a much stronger standard -- ie a way to (1) authenticate the credentials themselves, and (2) ensure the credentials match the person presenting them. These are inherently political/jurisductional issues, not technical)

2023 PSF annual impact report

Posted May 12, 2024 17:52 UTC (Sun) by farnz (subscriber, #17727) [Link]

Punishment gets us nowhere. Does it reduce the effects of the attack? No. Does it ensure such crimes happen less? No. There is no deterrence for crimes above a certain steal-bubblegum-threshold.

That last line is contradictory to what I know of criminology; increasing the punishment does increase deterrent effect, as long as the chances of getting caught are high enough. The problem comes in when you're not increasing the chances of getting caught, and attempting to deter purely by high penalties if caught.

First, you have people who, for some reason, do not have the ability to engage in causal reasoning. These people are rare, but they do exist.

More significantly, the punishment's effect on deterrence scales with the perceived chance of getting caught to begin with. If you consider your chances of getting caught to be near-zero, no amount of punishment will have a deterrent effect; what's the difference between a loud "NO!" and life in prison if you don't think either will happen?

To put it differently, when they're considering breaking the rules, people multiply their perceived cost of punishment by the perceived chance of being caught; if the resulting number is small enough compared to the perceived benefit of breaking the rules, then they'll break the rules. And there's a mental "clamp" on the range for everything "perceived", so you can't just increase the punishment further to get a bigger deterrent; the only option once the cost of punishment reaches people's "basically too big to get bigger" is to increase the chance of being caught, or reduce the benefit of breaking the rules.

2023 PSF annual impact report

Posted May 12, 2024 23:17 UTC (Sun) by Wol (subscriber, #4433) [Link]

> First, you have people who, for some reason, do not have the ability to engage in causal reasoning. These people are rare, but they do exist.

You've clearly not watched all these programs about the police :-)

I think what you say is true of the older generation, but so many kids these days seem to have brains addled by drugs (or drink) that they don't have a clue what they're doing ...

And for big crimes, people don't seem to think about the consequences of getting caught at all. Many crimes are "spur of the moment" things - and the bigger ones are often fuelled by anger (as I said, driven by drink or drugs ...).

Cheers,
Wol

2023 PSF annual impact report

Posted May 11, 2024 22:01 UTC (Sat) by mpr22 (subscriber, #60784) [Link]

"Is considered trustworthy enough by Leviathan to be allowed access to some of Leviathan's secret documents" is not, in fact, a useful proxy for "should be considered trustworthy-enough by an entity not directly backed by Leviathan".

2023 PSF annual impact report

Posted May 9, 2024 14:27 UTC (Thu) by farnz (subscriber, #17727) [Link]

The problem is that I only trust some countries and not others. Unless you present me a document that I can verify through an authority I trust, then it's precisely as useful as proving that you control a GitHub or Apple account with the same e-mail.

And that's what makes this problem so damn hard; you need a chain of trust from somewhere I trust (or am forced to trust, like my national government), otherwise your proof of identity is low-value. Geopolitics being what they are, that trust chain inevitably limits me to under 20% of the world population with trustworthy ID.

2023 PSF annual impact report

Posted May 9, 2024 14:41 UTC (Thu) by pizza (subscriber, #46) [Link]

> And that's what makes this problem so damn hard; you need a chain of trust from somewhere I trust (or am forced to trust, like my national government), otherwise your proof of identity is low-value.

No, that part's still relatively easy. What's hard is the same problem we've always had -- Even if you have incontrovertible proof of a BadPerson(tm)'s legal identity, if they are in a jurisdiction other than your own, it is quite difficult (ie expensive and/or time consuming) to enforce any judgement against them, even for nominally friendly jurisdictions. An actively hostile jurisdiction will fart in your general direction. If even that.

2023 PSF annual impact report

Posted May 9, 2024 14:59 UTC (Thu) by farnz (subscriber, #17727) [Link]

I'd count that as part of "somewhere I trust"; if I have no effective recourse should you abuse my trust, then your identity is not chained from "somewhere I trust".

2023 PSF annual impact report

Posted May 9, 2024 15:14 UTC (Thu) by Wol (subscriber, #4433) [Link]

> if they are in a jurisdiction other than your own, it is quite difficult (ie expensive and/or time consuming) to enforce any judgement against them, even for nominally friendly jurisdictions

ALMOST ALL jurisdictions protect their own citizens in their home country.

To stand any chance of success, you need to go to their jurisdiction, and sue them there. (a) your chances of success are just damn low to start with, and (b) seeing as you are the foreigner, you'll probably lose because you mis-understand (or most likely don't know) the rules.

Cheers,
Wol

2023 PSF annual impact report

Posted May 9, 2024 17:14 UTC (Thu) by rgmoore (✭ supporter ✭, #75) [Link]

The problem is that I only trust some countries and not others.
Even that degree of trust may be too much. I trust my own government a fair bit, but I also know there are lots of fallible individuals working for that government. If even one person with the power to enter data into the system is bribed, blackmailed, or tricked, it can result in a false ID in the system. It may not be quite as easy as they show it being in spy movies, but it's definitely possible.

2023 PSF annual impact report

Posted May 10, 2024 10:28 UTC (Fri) by paulj (subscriber, #341) [Link]

I wouldn't say it's possible, rather that it is common. Maybe less so in developed western nations, but still happens there.

2023 PSF annual impact report

Posted May 8, 2024 12:18 UTC (Wed) by kleptog (subscriber, #1183) [Link]

> Why can't PyPI and other endpoints that get called into from GitHub simply implement 429 errors unilaterally? It's not as if Microsoft is just going to tell everyone to stop using pip from their CI flows.

Just today I received an email from Azure that, because Docker Hub is implementing anonymous rate limiting soon, that they've created a new feature that acts as a pull-through cache for external docker registries. So if PyPI announced that they were rate-limiting anonymous downloads per IP on (some date in 2025) you have a good chance that something will be done.

It's not even hard. We just installed devpi as a container and added some pip configuration to the workers. Only slightly more complicated than getting Debian packages cached (which Azure DevOps *does* cache from a local mirror).

2023 PSF annual impact report

Posted May 8, 2024 1:20 UTC (Wed) by intelfx (subscriber, #130118) [Link]

> This is why we can't have nice things.

I see a certain irony here, recalling the story of a certain company's Go proxy bots being responsible for an uncomfortable amount of traffic to Drew DeVault's SourceHut. And no amount of 429ing could get that company to do the right thing in a timely fashion, because it was just too big to be bothered.

This is also why we can't have nice things.

2023 PSF annual impact report

Posted May 8, 2024 7:17 UTC (Wed) by NYKevin (subscriber, #129325) [Link]

*sigh*

I can't control what other Google employees do, or speak on their behalf. I also probably should not go around openly advocating for external users to deliberately cause Google outages.

What I will say is that any large corporation is mainly controlled by a series of middle managers, who speak only in terms of "business problems." Caching is not a business problem. Drew's blog post might be a business problem, if the press picks it up, but otherwise it is not. A total outage of the service is definitely a business problem (but I'm not going to say that Drew was wrong to avoid this option). "We're wasting somebody else's resources" is, unfortunately, not a business problem (for "us," anyway).

I don't like it any more than you do.

2023 PSF annual impact report

Posted May 8, 2024 4:08 UTC (Wed) by flussence (subscriber, #85566) [Link]

I would add to that: if something is hitting a public API or webpage frequently, at a bare minimum it should be required to implement Accept-Encoding too. Preferably brotli, but I hardly ever see that in the wild even though it's trivial (just use curl!)

2023 PSF annual impact report

Posted May 7, 2024 7:43 UTC (Tue) by kleptog (subscriber, #1183) [Link]

When we built our local build farm we including a caching proxy for PyPI and debian packages. This was mainly a performance optimisation: not downloading is always faster than downloading. But now with everyone pushing us to Azure DevOps/hosted GitLabs, all these runners don't have caching so they're pulling from the actual sites. Which bugs me, but it's not something that I can easily change (AFAICT).

These CI/CD sites should be hosting local mirrors of all this stuff so that at most they're sending requests with If-Modified-Since headers. Though I suspect the real problem is that HTTPS is not compatible with caching without explicit client support. And with everyone (correctly) pushing HTTPS everywhere, no longer caching packages just seems to be collateral damage.

2023 PSF annual impact report

Posted May 7, 2024 10:20 UTC (Tue) by hkario (subscriber, #94864) [Link]

and proxy support in software is spotty at best...

2023 PSF annual impact report

Posted May 10, 2024 0:17 UTC (Fri) by Hello71 (subscriber, #103412) [Link]

AWS has people so anchored to massive price gouging that nobody realizes how cheap IP transit actually is. A 10G connection is on the order of $2k/month, depending on utilization and location, so 190 Gbps would cost on the order of $38k/month without discounts. Or, if we look at the total global bandwidth usage, various sources claim it's on the order of 1 Pbps, so 190 Gbps is 0.019% of global internet usage. That seems high but within reason for such an important and widely-used internet function.


Copyright © 2024, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds