[HN Gopher] Show HN: Fully-searchable Library Genesis on IPFS
___________________________________________________________________
Show HN: Fully-searchable Library Genesis on IPFS
Author : sixtyfourbits
Score : 280 points
Date : 2021-09-19 15:42 UTC (7 hours ago)
(HTM) web link (libgen.fun)
(TXT) w3m dump (libgen.fun)
| isoprophlex wrote:
| And it's fast as hell too!
|
| This is fantastic yo, mad props.
|
| Thanks for liberating our collective knowledge, ipfs style. Keep
| it up!
| dr_dshiv wrote:
| https://libgen-crypto.ipns.dweb.link/
|
| Wow, this is way more user friendly than normal! Nice work.
| severine wrote:
| You could add a link to IPFS Lite (" _IPFS Lite node with modern
| UI to support standard use cases of IPFS_ ") on F-Droid. Seems
| actively updated, can anone vouch for its quality?
|
| Source: https://gitlab.com/remmer.wilts/ipfs-lite
|
| F-Droid: https://f-droid.org/en/packages/threads.server/
| 8K832d7tNmiQ wrote:
| It's probably just a few steps away to finally build an IPFS
| version of sci-hub. Sadly, I'm not a fan of the site's method of
| searching the libgen index by using sqlite's partial load feature
| [1] mainly because of the possible limited available storage
| issue.
|
| [1]:https://phiresky.github.io/blog/2021/hosting-sqlite-
| database...
| morsch wrote:
| I remember the sqlite via static host discussion, but I don't
| understand what you mean by "the possible limited available
| storage issue". Can you explain?
| betwixthewires wrote:
| Bravo. This is big stuff.
|
| There are some good critiques and ideas in this thread I won't go
| over since it's been done, I have been putting off building
| something akin to this (not the same thing but somewhat similar)
| hoping someone more motivated would do it, and I'm very excited
| to see it happen.
| mbStavola wrote:
| Immutability is a blessing and a curse for IPFS.
|
| It's cool for preventing things like censorship. Something like
| SciHub would really benefit from it.
|
| However, for "real world" use cases, many people want to be able
| to remove or modify what they've uploaded. With IPFS, as far as
| I'm aware, doing either doesn't really change the underlying data
| but just creates a new object in IPFS instead which you'd point
| to via IPNS. Anyone who still wanted to view the old content
| still could, provided they had the right content id.
|
| God forbid you accidentally upload a "personal" photo, your only
| hope is that someone never comes across the content id of that
| image. There is no way to undo it!
| eric__cartman wrote:
| From my understanding if you accidentally upload a personal
| file, as long as no one downloaded it in the time you took to
| realize your mistake taking down the only node that has the
| file (your computer in this case) should effectively "erase it"
| in the sense that unless the node comes back up, even if
| someone has the id of that file they are SOL.
| unknownOrigin wrote:
| Ok, I have to ask, what is an actual difference between
| torrents and ipfs? I don't care for technical details, I mean
| the business logic, so to speak.
|
| - Both use DHTs to search for sources of fingerprinted
| content. - Both use nodes (seeds in BT terminology) that
| actuallu store the content. - Both don't have an "archive"
| system, and so if at least one node doesn't have the file, it
| may as well not exist at all. - Both can have content
| censored by going after the node operators.
|
| Am I getting any of these wrong?
| easrng wrote:
| BitTorrent is easier to understand (IMO) and can sometimes
| be faster at lookups than IPFS. IPFS dedups content,
| whereas if you have an identical file in 2 torrents but
| only one is seeded, you can't download it from the other
| one. As far as I know those are the only differences.
|
| Edit: Actually BitTorrent v2 dedups files so it seems like
| IPFS and BitTorrent are now functionally identical.
| grumbel wrote:
| The difference is the granularity. A torrent is like a tar
| file, it's a big blob of static data that can't be updated.
| IPFS in contrast works more like a file system, you have a
| top level directory that points to the content within it.
| If you want to change something, you just update the top
| level directory, while all the content within it can stay
| the same. Each file on IPFS has it's own checksum and can
| be addressed individually.
|
| IPFS doesn't help much with censorship, as it has all the
| same issues as torrent in that area. It doesn't help much
| with privacy either, as it's all rather public. It's really
| for legitimate uses, not outside-the-law kind of stuff.
|
| The benefit of IPFS is that its granularity makes it much
| more useful for smaller tasks. For example you can host Git
| repositories or source trees on there. And since IPFS on
| Linux can be mounted as a file system, you can just access
| them with a simple `cd` command, no manual download or
| extraction needed.
| madars wrote:
| Tech details from the Getting Started guide:
|
| > How does this work?
|
| > SQLite compiled into WebAssembly fetches pages of the database
| hosted on IPFS through HTTP range requests using sql.js-httpvfs
| layer, and then evaluates your query in your browser.
|
| The same guide, https://libgen-crypto.ipns.dweb.link/, also
| explains how you can also download the page to search locally
| without constant internet access.
|
| sql.js-httpvs was previously discussed on HN here: Hosting SQLite
| databases on GitHub Pages or any static file hoster (1812 points)
| https://news.ycombinator.com/item?id=27016630
| hugoroussel wrote:
| The website looks down :(
|
| I love libgen will switch to your version for sure :)
| fabianhjr wrote:
| It is available over IPFS. Do you have a client installed and
| running?
|
| EDIT: if you do have IPFS and the companion extension then
| libgen.crypto should resolve to something like
| `http://libgen.crypto.ipns.localhost:8080/` which currently
| works as advertised.
| ajvs wrote:
| Awesome project, now all we need is the SciHub version of this.
| sayonaraman wrote:
| there is an open source project scihub-p2p (written in Go)
| indexing scihub articles on libgen torrents, but it also seems
| to work with IPFS
|
| https://www.reddit.com/r/scihub/comments/ovp5c0/make_scihub_...
| antegamisou wrote:
| Most of SH papers are accessible from libgen too.
| cmeacham98 wrote:
| > Put http:// as the protocol prefix instead of various https://,
| ipfs://, or ipns://. The most universal format is http://, since
| transport-level security (TLS) is not yet fully adopted in dWeb
| systems (neither it is essential for our applications where a
| domain name resides on a blockchain and SSL traffic encryption is
| employed further on)
|
| I understand that this isn't really their fault because they'd
| have to get a CA to issue a cert for the non-standard .crypto
| TLD, but has to be untrue, assuming I understand correctly that
| the HTTP version is just hosting a JS IPFS client? And therefore
| the non-IPFS link is suceptible to MITM attacks if I understand
| correctly.
| ducktective wrote:
| > decentralized Web
|
| I thought the web and internet was decentralized already?
|
| Interesting project, btw! Many thanks to the devs.
| superkuh wrote:
| I agree. IPFS is more centralized in that 99% of the people
| that attempt to view data on the IPFS network go through the
| actual web proxies not IPFS. And when that happens they're easy
| to take down with legal or traffic attacks.
|
| Additionally, IPFS's devs have already stated on their
| community forums that content like sci-hub is not welcome
| there.
| jancsika wrote:
| > Additionally, IPFS's devs have already stated on their
| community forums that content like sci-hub is not welcome
| there.
|
| Do you have a link?
| TacticalCoder wrote:
| > Additionally, IPFS's devs have already stated on their
| community forums that content like sci-hub is not welcome
| there.
|
| I'm not familiar with IPFS: is content on IPFS actually
| moderated somehow?
| IceWreck wrote:
| No, the discussion of "sci-hub on IPFS" is not allowed in
| their forums.
|
| Theres nothing stopping you frm actually doing that, just
| talk about it somewhere other than their official forums.
| jazzyjackson wrote:
| I think people don't notice that IPFS is not built for
| privacy- you are broadcasting that you have possession of a
| particular hash, not a smart place to be a pirate.
| mnd999 wrote:
| Isn't this a whole load of pirate ebooks though?
| chrisco255 wrote:
| I mean, maybe? It's trivial to add noise to a file to
| produce a different hash. It's a neutral routing protocol
| that doesn't make prescriptions one way or the other.
| zozbot234 wrote:
| Doesn't torrent have the same issue?
| sixtyfourbits wrote:
| The gateways are indeed centralized (though there are several
| different public gateways). But anyone who installs IPFS on
| their computer can access the content directly. Also some
| browsers support IPFS natively, e.g. https://brave.com/ipfs-
| support/
|
| From what I understand, the notion of sci-hub/libgen "not
| being welcome" was only about discussion on the official
| forums. See https://discuss.ipfs.io/t/mirror-of-sci-hub-in-
| ipfs/1613 and https://news.ycombinator.com/item?id=25209246.
| But IPFS is a protocol just like bittorrent or HTTP, and the
| software is open source; it doesn't and can't enforce
| copyright restrictions.
| superkuh wrote:
| >But IPFS is a protocol just like bittorrent or HTTP,
|
| Yes, but it's a protocol with a centralized single group
| doing development who can change whatever they want without
| the users' consent. Take a look at what is happening to the
| Tor ecosystem on Oct 15th this year: all tor v2 routing
| support is being dropped from the main client and
| infrastructure (for security reasons). Entire communities
| built on onionland and other tor v2 features, as well as
| all URLs/links, search engine databases, etc, will just go
| poof when the devs drop support.
|
| Unfortunately being a protocol isn't enough. It has to be a
| community protocol, not a proprietary one where everyone
| follows one group's code. HTTP and bittorrent are safe from
| these kinds of attacks. IPFS isn't (yet) and that's why
| their butt-covering anti-sci-hub/libgen stance is worrying.
| gzer0 wrote:
| That's being quite unfair to the developers. The entire
| process has been public and announced well in advance.
|
| You are welcome to download the Tor source code and add
| v2 functionality back in, and you'll be able to visit
| sites hosted by people who have done the same. No one is
| stopping you.
|
| To very quickly summarize why we are deprecating, in one
| word: Safety. Onion service v2 uses RSA1024 and 80 bit
| SHA1 (truncated) addresses [1]. It also still uses the
| TAP [2] handshake which has been entirely removed from
| Tor for many years now _except_ v2 services. Its
| simplistic directory system exposes it to a variety of
| enumeration and location-prediction attacks that give
| HSDir relays too much power to enumerate or even block v2
| services. Finally, v2 services are not being developed
| nor maintained anymore. Only the most severe security
| issues are being addressed.
|
| That being said, the deprecation timeline is now quite
| simple because v3 has reached a good maturity level:
| * v3 has been the default since Tor 0.3.5.1-alpha.
| * v3 is feature parity with v2. * v3 now has Onion
| Balance support [3] * Entire network supports v3
| since the End-of-Life of 0.2.9.x series earlier
| this year.
| a1369209993 wrote:
| > [1] [2] [3]
|
| Citation (literally) needed.
| easrng wrote:
| This is cool, but more centralized than it needs to be. The
| update check resolves libgen.crypto using
| @unstoppabledomains/resolution[0] with its default Ethereum
| provider, Infura[1]. That means that if Infura disables the
| default API key, goes down, or starts censoring responses, the
| update check will fail and users will be stuck on an older
| version of the site. Using the .crypto domain for updates is
| unnecessary, a simple IPNS[2] lookup (Not to be confused with
| DNSLink[3]) would've sufficed.
|
| [0]: https://www.npmjs.com/package/@unstoppabledomains/resolution
|
| [1]:
| https://github.com/unstoppabledomains/resolution/blob/HEAD/R...
|
| [2]: https://docs.ipfs.io/concepts/ipns/
|
| [3]: https://docs.ipfs.io/concepts/dnslink/
| nynx wrote:
| Pretty neat. Too bad searches take such a long time.
| c54 wrote:
| This is great! I've been half-barely-following IPFS development
| for a few years now but I think this is a salient use case that I
| could actually see myself using.
|
| I think also with IPFS i can share files with peers pretty
| easily? It's nicer than uploading to a filesharing site, and
| easier than setting up a torrent.
|
| So, what's next @sixtyfourbits? Is there a read-only wikipedia on
| ipfs yet?
|
| edit: found it, but I think it's not searchable
| https://en.wikipedia-on-ipfs.org/wiki/
| sixtyfourbits wrote:
| Next up is the scimag collection (also maintained by library
| genesis), which is a backup of all articles on sci-hub.
|
| The torrents are already widely replicated, but whether or not
| IPFS is going to be able to scale to the level required for 85M
| articles is still an unknown.
| ramon wrote:
| Nice, is it open source? Can we learn from this implementation?
| Is there any documentation?
| sixtyfourbits wrote:
| https://libgen-crypto.ipns.dweb.link/source.tar.gz
|
| It makes use of the sql.js-httpvfs library, previously
| discussed on HN here:
| https://news.ycombinator.com/item?id=27016630
| [deleted]
| mijailt wrote:
| It's not working for me on Firefox 92.0. It hangs on
| "Initializing...", with a single error on the console which says
|
| Loading Worker from
| "http://libgen.crypto.ipns.localhost:8080/dist/257fb50677e116...
| was blocked because of a disallowed MIME type ("text/plain")
___________________________________________________________________
(page generated 2021-09-19 23:00 UTC)