[HN Gopher] Libgen Storage Decentralization on IPFS ___________________________________________________________________ Libgen Storage Decentralization on IPFS Author : jerheinze Score : 214 points Date : 2020-11-25 13:58 UTC (9 hours ago) (HTM) web link (freeread.org) (TXT) w3m dump (freeread.org) | Vespasian wrote: | I'm hesitant to participate in IPFS in the context of libgen. | | Torrenting copyrighted materials (especially if you are | distributing) can be quite expensive in some jurisdictions and I | do not believe that IPFS is treated different than regular | BitTorrent. | whimsicalism wrote: | > Torrenting copyrighted materials (especially if you are | distributing) can be quite expensive in some jurisdictions | | Why? Your ISP charges for BitTorrent traffic? | dewey wrote: | In case you are serious: Lawyer agencies are joining the | swarm, recording IPs, write letters to ISPs and send you | fines. That's very common for example in Germany. | | Edit: I didn't downvote you if it looks like that :) | whimsicalism wrote: | > write letters to ISPs and send you fines. That's very | common for example in Germany. | | And you have to pay the fines? That's nuts. | | In the US, you are pretty much as likely to get fined for | torrenting as you are to get arrested for jaywalking | (although without the racial disparity). | dewey wrote: | > And you have to pay the fines? That's nuts. | | Yes, conveniently there's a second group of lawyers that | focuses on getting people out of these fines...for a fee | that's not so much lower. Unfortunately it's a bit of a | business model in some countries. | nemo wrote: | In the US the (very) common scenario is to get a couple | warnings from your ISP and if you continue torrenting | copyrighted material you will be dropped as a customer | and banned. | bboygravity wrote: | France was pressured by the US diplomatically to do the | same in France. It's called HADOPI. | | Source: wikileaks | whimsicalism wrote: | > the (very) common scenario is to get a couple warnings | from your ISP and if you continue torrenting copyrighted | material you will be dropped as a customer and banned. | | This is not "very" common - the ISPs don't really ban | you, they just threaten you. I no longer do this, but I | extensively torrented over multiple home ISPs and they | would send me warnings but they would never actually do | anything. | nemo wrote: | AT&T has banned a friend permanently so I would be | careful about over-broad generalizations. Folks in my my | household were torrenting and we got our service cut with | the second offense until we went through some rigmarole | promising we'd never do it again at which point they | stopped. The cost to an ISP's lawyers to field these | things isn't free and at least some really do respond | with action. | whimsicalism wrote: | I'm curious if the action taken depends on broadband | competition in the area - do you live in a city? | djeiasbsbo wrote: | Check out https://gnunet.org/ | | It works well for file sharing as well. Don't know about | copyrighted material though, have only shared my own files. | andrius4669 wrote: | Does it actually work well? I tried to play with it a while | back and while sharing files appeared to work, actually | downloading them from other node didn't go that well.. | djeiasbsbo wrote: | The thing that doesn't work well yet in my experience is | the NAT traversal and finding peers in general. But I | tested the same thing with two peers and I could download | with no problems. It also seems to get updated quite often, | I tested very recently. | Cactus2018 wrote: | Wikipedia | | > The InterPlanetary File System (IPFS) is a protocol and peer- | to-peer network for storing and sharing data in a distributed | file system. IPFS uses content-addressing to uniquely identify | each file in a global namespace connecting all computing devices. | fsflover wrote: | I think such materials should better be placed to darknets | protecting the privacy of users and seeders, e.g., to I2P. | LockAndLol wrote: | It'd be nice for IPFS to support an I2P proxy. That would be | quite an important change and would save anybody downloading | from LibGen over IPFS. | bboygravity wrote: | I agree. Actually in this day and age I think all information | sharing should better be placed on darknets. I don't see the | net advantage to anybody of being profiled constantly on | clearnet. | | Naive question: is there such a thing as Bittorrent over I2P or | otherwise anonymized decentralized file sharing? | [deleted] | RealityVoid wrote: | I see it as 2 separate problems: data availability and | anonymous transfer. IMO, you need 2 separate solutions | because they are 2 different problems. | | IPFS solved the data availability problem because you can | incentivize storage miners to make your data available. This | is something new, IMO. | | Accessing it anonymously is possible with an extra protocol | layer. | apples_oranges wrote: | For instance, using TOR to download books from libgen | should be pretty safe, no? | segfaultbuserr wrote: | > _Bittorrent over I2P_ | | Yes. I2P is designed with P2P in mind. The default Java I2P | client already comes with a bundled BitTorrent client, there | is also a BitTorrent client called "Vuze" and is used by some | non-anonymous users to help seeding a torrent across the | clearnet and I2P simultaneously. Because of the nature of | I2P, don't expect great speed. No seeder is "usual", 10 KiB/s | is "normal" and 50 KiB/s with only 3 or 5 seeders is "good", | overall, it feels as if it's still the early 2000s, but when | you get used to it and adopt a "download and forget, come | back in 3 days" mentality, it's not actually that bad. The | fastest download I've ever seen was the movie _Snowden_ , 20+ | seeders and over 200 KiB/s+... | derefr wrote: | If you take IPFS's model and add inherent anonymity, that's | exactly what Freenet is. (To this day I'm confused about why | IPFS is popular while Freenet isn't.) | chriswarbo wrote: | I've not tried Freenet for a few years, but every time I did | it was _very_ slow; e.g. browsing basic HTML sites hosted on | Freenet was quite painful. IPFS can serve streaming video, | and host massive archives like Wikipedia. | | Also, Freenet actively seeks out data from the network to | store locally, to give plausible deniability to users | (there's no way to tell if something is locally cached | because the user requested it, or whether the client fetched | it automatically). IIRC this can be disabled by limiting | oneself to a whitelist of peers (preventing network chatter | that might reveal Freenet's presence). IPFS will only cache | what's requested, so it's more predictable and useful for | building infrastructure (e.g. hosting assets for a normal Web | site); and presumably leaner on data requirements too. | | IPFS isn't really competing with Freenet IMHO; it's competing | with HTTP and BitTorrent. | eMGm4D0zgUAVXc7 wrote: | > I've not tried Freenet for a few years, but every time I | did it was very slow | | It has become a lot faster recently, try again :) When I | look at its statistics during active usage it can easily | show 1 MiB/s traffic. | | You still can't expect HTML to load in sub-second delays, | for large multi-MiB sites it may take a low single-digit | amount of seconds, but there is an inherent cost to | anonymity which probably induces a boundary on how fast | things can be: | | To achieve anonymity, data needs to be _redirected_ across | multiple people so the sender and recipient can 't | determine who each other of them is. | | Redirecting stuff across a longer path than necessary slows | things down. | bawolff wrote: | > To achieve anonymity, data needs to be redirected | across multiple people so the sender and recipient can't | determine who each other of them is. | | > | | > Redirecting stuff across a longer path than necessary | slows things down. | | That's skipping past a lot of details. There are still | choices there. Freenet traded speed, efficiency and | scalsbility for a specific security model. There are | anonoyminity systems (e.g. tor) that are much more | efficient but have different trade-offs. | Mediterraneo10 wrote: | Freenet didn't take off in part because there one hosts all | kinds of distributed files you don't know the content of, | because they are encrypted, but if someday that encryption | algorithm is broken, there is a strong chance many people | were (unwittingly, but still) hosting child pornography. | fsflover wrote: | What is the point of Freenet when we have I2P, which allows | any protocol on top of it, not just file sharing? | dooglius wrote: | A strange choice given that distributing copyrighted materials | violates the IPFS Code of Conduct [0]. Why not use BitTorrent | whose attitude toward copyright is much more in sync with | libgen's? | | [0] https://github.com/ipfs/community/blob/master/code-of- | conduc... | toomuchtodo wrote: | > given that distributing copyrighted materials violates the | IPFS Code of Conduct | | Everyone has wishes, but wishes aren't enforceable. It's the | equivalent of "Stop! Or I'll say stop again!" | josu wrote: | Yeah, and the fact that they've already decided to blacklist | sci-hub. | | https://discuss.ipfs.io/t/mirror-of-sci-hub-in-ipfs/1613 | arsome wrote: | Am I missing something? I don't see any mention of a | blacklist in that thread. They did close it because of | course, they don't want to be in Limewire's position, where | they get held responsible for the tech they built because | they supported, even in a minor way, its use in piracy. | m-p-3 wrote: | > distributing copyrighted materials violates the IPFS Code of | Conduct | | Even if so, what can they (the IPFS devs) even do about it? | segfaultbuserr wrote: | LibGen already has "torrent per 1000 files" for a long time, | some books also have their individual torrents but it's not | common. I'm not sure why, but I think the problem is that files | on BitTorrent cannot be directly addressed by hashes, it had to | go through an intermediate step called a "torrent", which does | not uniquely identify individual files directly. You can always | select the one file you need from a huge torrent, but it's not | as convenient as directly addressing files by their hashes. | dooglius wrote: | That's fair, BitTorrent v2 does solve this problem, but I | can't say I've seen any v2 torrents around (not that I'm | specifically looking) | elvis70 wrote: | Ah, thanks for mentioning that. If I understand well, it's a | distributed filesystem but there is still an authority able to | ban users or contents? | diggan wrote: | The answer for both BitTorrent and IPFS is: No. | m-p-3 wrote: | They can blacklist hashes on the ipfs.io reverse proxy, but | they cannot block you from accessing those through a | different node like your own. | diggan wrote: | Protocols don't have attitudes, everything you're using IPFS | for you could also use BitTorrent for (almost). The protocols | don't really care about what data you send, and no one can stop | you from using them, BT or IPFS. | | The CoC you've linked covers the IPFS Community and other | Protocol Labs initiatives, it doesn't cover usage of IPFS | itself. freeread.org is not bound by that CoC. | josu wrote: | The protocol can't, but the network can. | | https://discuss.ipfs.io/t/mirror-of-sci-hub-in-ipfs/1613 | diggan wrote: | Your link is to a forum hosted by Protocol Labs for the | IPFS Community. Of course they are trying to follow their | own CoC on their own properties, especially when it comes | to copyright infringement. As a US company, they're bound | to follow US law. Hardly surprising. | | What does that have to do with the network? Again, the | network nor the protocol currently has nothing in place | that describes a "attitude" of being against copyright | infringement. | | I remember that we (I'm a ex-Protocol Labs employee) used | to have plenty of discussions about adding allow/blocklists | to the protocol that people could opt-in to, but don't | think that was never added (yet?). If it was, I'd | understand where you're coming from. | georgyo wrote: | They are specifically asking for help and guidance to break | the law. Any public community has to avoid those questions | like the plague. | | Remember youtube-dl got taken down because it's README file | referenced copyrighted material. | | It doesn't matter if a tool _could_ be used distribiting | copyright material. The tool (and communities) cannot | promote anything related to that use case at all. | | The same is true with bittorrent. Bittorrent _could_ be | used to download copyrighted music and movies, but you | won't find a single reference to anything like that use | case on their web page because that is illegal: | https://www.bittorrent.com/ | | In fact the bittorrent terms of service says you may not | use the tool for any illegal purposes: | https://www.bittorrent.com/legal/terms-of-use/ | dooglius wrote: | It's more that just the protocol though. TFA links directly | to the IPFS Desktop Client, which certainly can have an | attitude e.g. if a future update blocks this project. That | risk doesn't really exist for any mainstream torrent client. | donpdonp wrote: | The linked article does not introduce the project, nor does the | project site have an about page. wikipedia to the rescue! | | https://en.wikipedia.org/wiki/Library_Genesis | MichaelMoser123 wrote: | does anyone know why libgen is still running over http? (But | sincerely thanks for the site! it saved me on several occasions!) | generationP wrote: | They don't want the extra weak spot of certificate revocation | being abused against them? | | Though I'm not sure how big of an issue this is; sci-hub is | using https. | jdu9 wrote: | Unless they activate HSTS, it shouldn't be a problem. They | can always go back to HTTP if they want. | | Also, Let's Encrypt doesn't even revoke certificates for | malware and phishing sites. [1] | | Maybe they're doing this for better caching performance? It | makes SSL flood attacks impossible, but is that really worth | it? | | [1] https://community.letsencrypt.org/t/how-to-report- | abuse/4110... | warmwaffles wrote: | Probably because they don't care enough to add https. | dublinben wrote: | One of the new IPFS-backed mirror sites (https://libgen.fun/) | is on HTTPS. | segfaultbuserr wrote: | The IPFS mirror really gives a substantial performance boost. In | the past, the files had to be fetched from a tired server far | away in Russia, the speed can be as low as 0.5 Mbps in my | experience (depends on your ISP and connectivity), now it can be | lightning fast thanks to P2P, 2 Mbps+, more than adequate to | download a 100 MiB+ file. You don't even have to install IPFS | (but you probably should, to maintain decentralization), books | can be downloaded straight via HTTPS from Cloudflare's IPFS | reverse proxy (called a "gateway"), cloudflare-ipfs.com. | | The IPFS network itself cannot be taken down, but I wonder when | will the publishers discover IPFS web gateways, and start filing | DMCA takedown requests against them, and what are the gateways | gonna do? Will Cloudflare implement a huge copyright blocklist or | even terminate its service (or worse, send a list of IP addresses | to Prentice Hall or Elsevier)? What about the semi-official | ipfs.io gateway? And the smaller independent IPFS gateways (there | are currently 10+ on the public web)? Will they be DMCAed out of | existence? | an_opabinia wrote: | > send a list of IP addresses to Prentice Hall or Elsevier | | I don't know, do you think nobody has noticed LibGen by now? Do | you think the reason they're still running is because of some | jurisdiction issue? Even the lawyers would be wrong on that | one. | segfaultbuserr wrote: | > _do you think nobody has noticed LibGen by now? Do you | think the reason they're still running is because of some | jurisdiction issue?_ | | I'm not sure about the implication of your words. But, | clearly, Elsevier has already tried to take legal actions | against LibGen previously, and once in a while, LibGen's | domain names are still being blocked and routinely replaced, | just like Sci-Hub. I'd still say the reason it's still | running is a jurisdiction issue, or let's say it's a | geopolitical issue, it's using Russia to shield itself away | from US influences. | | But an IPFS gateway hosted in the United States by Cloudflare | does not enjoy these legal and political benefits. | tutfbhuf wrote: | IPFS gateways are similar to Tor exit relays, they just route | the traffic to the public internet. Tor exit relays still | exist, and haven't been DCMAed out of existence so far. | | One could also imagine IPFS over Tor. | georgyo wrote: | You're almost right. The IPFS gateway nodes also cache the | content. IE, they are both storing, hosting, and sharing to | other IPFS nodes. | mpfundstein wrote: | how can one validate that the cached version is indeed bit | equal to the one stored within ipfs? | openfuture wrote: | You hash it, everything is content addressed. | georgyo wrote: | The address of the file is the hash of the file. (Really | the hash of the DAG of the file) | | So you could validate it if you wanted to. Because the | URL _is_ the hash, if you did validate it you could be | very sure that it is correct. | | If a webserver hosts a file, how can you be sure that | file is correct? Sometimes the ship a sums file next to | it. At best this can find corruption, but it would not | find malicious files. | mintplant wrote: | IPFS-over-Tor has been trapped in development hell for 5 | years. | | https://github.com/ipfs/notes/issues/37 | segfaultbuserr wrote: | If IPFS-over-Tor is in the development hell, IPFS-over-I2P | is in the development limbo. | | https://github.com/ipfs/notes/issues/124 | segfaultbuserr wrote: | Tor exit nodes are forward proxies, they are used by the | client-side to initiate connections to the public Internet | but do not accept connections from the public Internet. You | cannot host an Internet-facing webserver on a Tor exit (Onion | Service exists solely within the Tor network), DMCA is not an | issue, they do not act as a point of content distribution, | there is nothing to take down. The main issue is the abusive | outgoing traffic, which is seen as unavoidable and outweighed | by Tor's greater benefits, so the existence of Tor exits is | justified. | | IPFS gateways (and Tor-to-Web) are reverse proxies, they are | configured at the server-side to accept connections from the | public Internet and route traffic to IPFS (and Tor), and they | distribute content to the public Web. As far as I can see, | this is a problem, someone can send DMCA notices, asking the | gateway operators to take files down from the Web. The only | protection I see is DMCA Section 512 (a.k.a Safe Harbor | Provision), the operators are not liable for things solely | hosted on IPFS, which is good, but gateways must comply with | the takedown notices. So I think it's entirely possible for | Cloudflare to put a huge blocklist on its IPFS gateway, and | for smaller gateways that don't have the necessary resources | to handle the requests, be DMCAed out of existence entirely. | In the 2000s, RIAA launched campaigns against eD2k servers | and BT trackers, including the use of honeypot servers to | collect information. RIAA was not ultimately successful, but | it created a major short-term disruption (eventually new | servers would always appear in a different jurisdiction). A | similar campaign against IPFS gateways sound possible if the | publishers decided to launch an aggressive crackdown like the | RIAA. The question is only a matter of whether the decision | is made. | | I'm not saying that IPFS requires the use of IPFS gateways, | they don't, and in a sense, gateways are counterproductive | for the decentralization goal. But they do currently provide | an useful service for the Web, and I'm just speculating | whether a major disruption is possible. | traverseda wrote: | In my experience IPFS is _only_ fast when you 're using it | through something like Cloudflare's IPFS proxy (which is | basically a caching proxy). I haven't found IPFS to be actually | usable any time I've tried running a node myself. | | Is that unusual? | magila wrote: | This has been the state of IPFS for years now. The protocol | and client haven't been developed to where they can work | reliably in a real P2P scenario. IPFS only appears to work | because most requests go strait to one of the big gateways | which already has the content cached. | | Development of IPFS slowed to a crawl when most of Protocol | Labs switched to working on Filecoin. It remains to be seen | if either Filecoin or IPFS will ever be viable for their | intended purpose. | dsun179 wrote: | Yes, this is still the case. For a file which is on exactly | one other computer, you have about 45 seconds lookup time | before the download can start. | georgyo wrote: | I'm not sure you are right about this. The following link | will stream big buck bunny directly from the network with | full P2P support. It is not using a gateway node for the | video content. | | https://bbb.fu.io/ | | You can run the following in the console to see all your | peers for await (const peer of await | node.swarm.peers()) { console.log(peer.addr.toString()) } | | The console currently get's spammed pretty hard because | they haven't implemented filtering out plain ws (non-wss) | peers if you are connecting https. But it will use webrtc | to find peers, and find all wss nodes in the network. | magila wrote: | For me at least the vast majority of the data is being | downloaded from various servers under bootstrap.libp2p.io | or preload.ipfs.io. These are clearly dedicated servers | not true peers. | | The contact info for these servers appears to be | retrieved via the DNS rather than the DHT. This isn't | surprising because browser clients can't participate in | the DHT, but the DHT is currently one of the biggest weak | points of IPFS. IPFS's DHT is so chatty that it consumes | over a megabit of throughput just to maintain an idle | node. Lookups are excruciatingly slow and often fail | completely. | georgyo wrote: | Sorta, a peer can advertise dns addresses, and the | browser can participate in the dht as a client. The | example above uses webrtc-star for browser to browser | discovery. $ ipfs dht findpeer | 12D3KooWJxNHY6zE1KnzFdBJrTMCbhCVNtSLvjp7qR5Wsp49DnUC | /dnsaddr/ipfs.scalable.io | /dns4/ipfs.scalable.io/udp/4001/quic | /dns6/ipfs.scalable.io/tcp/443/wss | /dns6/ipfs.scalable.io/udp/4001/quic | /dns4/ipfs.scalable.io/tcp/443/wss | derefr wrote: | IPFS is the engine of Filecoin, though. Filecoin just | incentivizes IPFS nodes to store things. So further IPFS | development _should_ still be necessary from the | perspective of making Filecoin development /adoption | happen. | | (I say this, and yet I know it isn't true in practice. But | _why_ isn 't it true?) | nwah1 wrote: | Pump up Filecoin price. Sell Filecoin. Move to Fiji. | Don't do any coding while in Fiji | the_duke wrote: | No. | | The last time I tried it (maybe 9-12 months ago) it was a | real resource hog (including choking the network) and really | slow. | swills wrote: | Are you opening the required ports (4001, 8080)? I found it | was like that too until I did opened the ports. | segfaultbuserr wrote: | Interesting, didn't notice that before. I believed my | downloads are too obscure to be cached by Cloudflare, but I | may be wrong. I'll do a comparison once I fixed my server at | home... | mathnmusic wrote: | And what about when browsers (like Brave) start supporting IPFS | natively, and gateways are not needed any more? | jakobdabo wrote: | Am I understanding it right that DMCA takedown requests apply | to a URL? | | Maybe IPFS can introduce a feature to generate one-time URLs, | e.g. `oneTimeURL = f(realURL);`. | | Websites then can generate new URLs for example every hour or | even for each client request. Copyright bots will file a DMCA | request to a URL which only they have. | judge2020 wrote: | In general DMCA takedown requests reference a URL to locate | infringing content, but the actual requests ask that the | content itself be removed and not just the URL. | frrad wrote: | I wrote this https://github.com/frrad/skyhub the other day to | provide a nicer interface on top of the scihub torrents. | kristianpaul wrote: | "- Recommended at least 16GB RAM and Intel i5 or equivalent | processor - Recommended at least 100 mbps, gigabit connection | preferred" | | And I was going to try this one of my embedded computers :( | petra wrote: | What's missing from this system(and P2P systems in general), is | automatic pinning - i.e. the system automatically choses which | files to pin on which user, to ensure every file is available on | the net. | | It's a bit of tricky problem, because people may not like | unknown, random files on their pc, and probably, because such | algorithm is distributed and will need to resist attacks. | | But i think that the payoff is big: what Bitorrent did to popular | files, such an algorithm could do to rare files. | rudolph9 wrote: | Ipfs has something called "bitswap" which solves some of what | you're describing but is lacking in areas such as long term | file availability. That's where filecoin sort of finds sort | fills in the gap. | | https://docs.ipfs.io/concepts/bitswap/ | dboreham wrote: | We built a simple pinning system for a project I'm involved | with: each node queries a blockchain-based database for a set | of resources to pin locally. Resource publishers pay blockchain | fees to advertise their data for pinning. | | We also found it necessary to build a kind of "overlay" to the | p2p network (how each peer selects its peers) because the | default algorithm produced too sparse a network. The | probability of one node ever being connected to another node | that holds content it would like to sync approaches zero. The | topology for the overlay network is also fetched from said | blockchain. | corey_moncure wrote: | If the pinning came with encryption where the key is not known | to the pinned system, I think most would be OK with it. There's | no way to know, or demonstrate that anyone could know, the | contents of any given file. Furthermore, the file could be | broken into encrypted chunks and distributed redundantly to | many users. Kind of like BT turned inside out. | arsome wrote: | This is pretty much exactly how Freenet works. | derefr wrote: | https://en.wikipedia.org/wiki/Perfect_Dark_(P2P) is this kind | of system. You allocate a slice of your disk for the program | to use as a global cache, and random encrypted chunks from | the network end up downloaded into it. | chriswarbo wrote: | See the comment above about Freenet, which does something like | this; although according to | https://freenetproject.org/pages/documentation.html#content it | seems to prioritise popular files rather than rare ones. | | IIRC eDonkey prioritised rare files, although I don't know if | this was a property of the network or just a common choice in | clients. | eMGm4D0zgUAVXc7 wrote: | There is a Freenet "KeepAlive" plugin which can be used to | selectively pin files which you want to preserve, even if | they are completely unpopular. | crest wrote: | https://cluster.ipfs.io/ | rudolph9 wrote: | This doesn't automatically pin, rather distributes a pin set | across a set of trusted nodes. | | It's good stuff nonetheless. | fareesh wrote: | Is book piracy a major concern for the publishing industry? Many | avid readers that I know are happier buying the physical book | because they like the feel of reading from a book. Have others | found this to be true as well? I wonder if younger folks are less | susceptible to this feeling. | Synaesthesia wrote: | Most people don't know how to pirate books but yeah it is. I | got a lot of university textbooks this way. | contravariant wrote: | Many avid readers I know are happy with physical books but also | happy to use an e-reader for purely pragmatic reasons (easier | to carry, custom font settings). | | As for piracy if the places I see libgen and it's kind | referenced are any indication then this isn't about just any | readers but about students, professors etc. who want to access | scientific publications. | justarandomq wrote: | Is LibGen safe? I remember testing some of its mirror links on | virustotal and saw some red flags, so I decided not to risk | downloading. | Synaesthesia wrote: | Yes it's just ebooks. | mpfundstein wrote: | and scientific papers | mpfundstein wrote: | what red flags did you see? | justarandomq wrote: | It was some book, and when submitting the download link, 3 of | the antivirus results were red. Perhaps they were false | positives. ___________________________________________________________________ (page generated 2020-11-25 23:00 UTC)