This is what I came to say. We pre cache dependencies into an approved baseline image. And we cache approved and scanned dependencies locally with Nexus and Lifecycle.
A TPM is a form of HSM (Hardware Security Module).
HSMs come in all sizes, from a chip in your phone (secure element) or even a dedicated part of a SoC chip, to a big box in a datacenter that can handle tons of requests per second.
The idea is having dedicated hardware to protect the private key material. This hardware can execute signing operations, so it can use the key but it can't share the key material itself. It is usually also physically hardened with techniques to extract said keys, like sidechannel attacks based on power draw, X-ray inspection, decapping etc.
The story implies that these are signing keys, so there is no reason for the private halves to be present in the product's silicon in any form. If these were encryption keys stored in a TPM, they'd have been extracted not leaked.
> capsule (?) for humans riding to the ISS looks reminiscent of the 1960s too
Dragon 2 [1] looks like the Apollo and Gemini craft for the same reason it resembles Soyuz 3 [2]. Crewed disposable atmospheric reëntry vehicles launched in cylinders and soft landed or splashed down under parachutes are going to look similar.
“The previous people who did a similar analysis
did not have a direct pipeline to the wisdom
of the ages. There is therefore no reason to
believe their analysis over yours. There is
especially no reason to present their analysis
as yours.”
Absolutely. What sound pretty cool, and different, here is CalPrivacy would be required to build a request mechanism that's one request sent to every data broker.
They open themselves up to a lot of risk, but more likely they only comply when CA residents are concerned or stop collecting for CA residents. Good question about outside the USA. Makes me wonder if there may end up being some sort of data broker safe havens setup, like we've seen with banking.
California will take them to court and/or block them from doing business in the state, have various ways to penalize them, etc. California is big enough that many will want to play game with them and having a state as powerful as California on board will get other states to jump on board and pass their own legislation and take up the same tactics with non-complying companies. Once it gets enough traction at the state level, the fed will step in because this will affect interstate commerce and that is federal jurisdiction. This is how state sovereignty works, it is not that states can do as they please, they can only do it up until the point it affects other states or crosses the line with federal law.
> Sounds an awful like The Right to be Forgotten under GPDR Article 17
Does DROP let you censor search records?
I’d encourage anyone in Europe to compare California’s CCPA to the EU’s GDPR. It was inspired by the latter, and fixes a lot of its problem. (The Swiss referendum system was based on learning from and improving on California’s.)
Pypi has fewer than one million projects. The searchable content for each package is what? 300 bytes? That's a 200mb index. You don't even need fancy full text search, you could literally split the query by word and do a grep over a text file. No need for elasticsearch or anything fancy.
And anyway, hit rates are going to be pretty good. You're not taking arbitrary queries, the domain is pretty narrow. Half the queries are going to be for requests, pytorch, numpy, httpx, and the other usual suspects.
2. apt repositories are cryptographically signed, centrally controlled, and legally accountable.
3. apt search is understood to be approximate, distro-scoped, and slow-moving. Results change slowly and rarely break scripts. PyPI search rankings change frequently by necessity
4. Turning PyPI search into an apt-like experience would require distributing a signed, periodically refreshed global metadata corpus to every client. At PyPI’s scale, that is nontrivial in bandwidth, storage, and governance terms
5. apt search works because the repository is curated, finite, and opinionated
The install side is basically Merkle-friendly (immutable artifacts, append-only metadata, hashes, mirrors).
Search isn’t. Search results are derived, subjective, and frequently rewritten (ranking tweaks, spam/malware takedowns, popularity signals). That’s more like constantly rebasing than appending commits.
You can Merklize “what files exist”; you can’t realistically Merklize “what should rank for this query today” without freezing semantics and turning CLI search into a hard API contract.
The searchable context for a distribution on PyPI is unbounded in the general case, assuming the goal is to allow search over READMEs, distribution metadata, etc.
(Which isn’t to say I disagree with you about scale not being the main issue, just to offer some nuance. Another piece of nuance is the fact that distributions are the source of metadata but users think in terms of projects/releases.)
> assuming the goal is to allow search over READMEs, distribution metadata, etc.
Why would you build a dedicated tool for this instead of just using a search engine? If I'm looking for a specific keyword in some project's very long README I'm searching kagi, not npm.
I'd expect that the most you should be indexing is the data in the project metadata (setup.py). That could be unbounded but I can't think of a compelling reason not to truncate it beyond a reasonable length.
You would definitely use a search engine. I was just responding to a specific design constraint.
(Note PyPI can’t index metadata from a `setup.py` however, since that would involve running arbitrary code. PyPI needs to be given structured metadata, and not all distributions provide that.)
>The searchable context for a distribution on PyPI is unbounded in the general case, assuming the goal is to allow search over READMEs, distribution metadata, etc.
How does the big white search box at https://pypi.org/ work? Why couldn’t the same technology be used to power the CLI? If there’s an issue with abuse, I don’t think many people would mind rate limiting or mandatory authentication before search can be used.
The PyPI website search is implemented using a real search backend (historically Elasticsearch/OpenSearch–style infrastructure) layered behind application logic on Python Package Index. Queries are tokenized, ranked, filtered, logged, and throttled. That works fine for humans interacting through a browser.
The moment you expose that same service to a ubiquitous CLI like pip, the workload changes qualitatively.
PyPI has the /simple endpoint that the CDN can handle.
It’s PyPI philosophy that search happens on the website and pip has aligned to that. Pip doesn’t want to make a web scraper understandably so the function of searching remains disabled
reply