[HN Gopher] PyPI in a Box
___________________________________________________________________
PyPI in a Box
Author : luu
Score : 45 points
Date : 2022-02-25 20:35 UTC (2 hours ago)
(HTM) web link (vuyisile.com)
(TXT) w3m dump (vuyisile.com)
| digisign wrote:
| We had mirrors on every continent for linux distributions (and
| other things) way back before the turn of the century! _cough_
| _cough_
|
| Mildly surprised a service as big as PyPI doesn't.
| woodruffw wrote:
| > Mildly surprised a service as big as PyPI doesn't.
|
| PyPI uses Fastly as a CDN, which does indeed have presence on
| every continent (except the big, cold one). The problem here
| isn't presence, but connectivity to the outside Internet
| itself.
|
| Source: I'm an active developer on PyPI.
| bool3max wrote:
| PyPI is more complex and probably serves way more traffic.
| mjw1007 wrote:
| I think PyPI is mostly served via Fastly.
|
| https://dustingram.com/articles/2021/04/14/powering-the-pyth...
| heavyset_go wrote:
| PyPI needs more funding. PyPI even disabled their search API
| for the pip CLI because of infrastructure overload. It would be
| nice if more sponsors stepped up to fund their infrastructure.
| db65edfc7996 wrote:
| I think companies could choose to be more responsible with
| their usage. Looking at PyPi utilization, I have to imagine
| the bulk of it comes from CI/CD tooling hammering the servers
| without any intermediate caching.
| melissalobos wrote:
| I think the idea here isn't that there isn't a PyPI mirror in
| Africa. It is that not everyone has internet, so this person
| wants to have a tiny computer output a local WiFi network
| people can connect to and download pip packages. Imagine a
| small town or village with some power and a classroom, but no
| internet. A teacher could setup a network and have students
| connect to this device and download packages so they can
| complete some assignment/make the next Facebook.
| bscphil wrote:
| Or that the Internet access they do have is often metered. A
| friend tells me 100MB costs about 1 USD where he is, which is
| not an insignificant amount of money. Really puts the whole
| 300 MB electron app thing in perspective; at any rate it's
| understandable why having a PyPI mirror in the classroom
| would be preferable to having each student download the
| packages over and over.
| qbasic_forever wrote:
| I remember at pycon ~2016 one of the maintainers of pypi did a
| short talk on it and the entire pypi service at the time only
| ran on one or two boxes. It was surprisingly scrappy for such a
| critical service.
| quietbritishjim wrote:
| devpi acts as a caching proxy for PyPI and takes a bit less setup
| than this. Plus, you can use it for storing your own packages in
| a separate index.
|
| https://github.com/devpi/devpi
| woodruffw wrote:
| This is great work!
|
| Python packaging is complicated for many reasons (both good and
| bad), but PyPI's index format is delightfully simple. Projects
| like this reinforce my opinion that keeping it simple has been a
| great decision by the Python community and PyPI admins.
| simonw wrote:
| The developer experience in regions that don't have fast internet
| access is hard to imagine, especially with bandwidth-hogs like
| the npm ecosystem.
|
| See also https://meyerweb.com/eric/thoughts/2018/08/07/securing-
| sites... - which points out that when every site moved to https
| it broke local caching proxies, which had a big negative impact
| on people in countries with slower internet.
| dralley wrote:
| You can also do this with Pulp, and have it act as a caching
| proxy that lazily caches the packages only when they first get
| downloaded.
|
| It's a lot more heavyweight though, so maybe it's not the best
| choice for a Raspberry Pi.
___________________________________________________________________
(page generated 2022-02-25 23:00 UTC)