[HN Gopher] Scuttlebot: Peer-to-peer database, identity provider...
___________________________________________________________________
Scuttlebot: Peer-to-peer database, identity provider, and messaging
system
Author : pcr910303
Score : 225 points
Date : 2020-04-18 18:35 UTC (4 hours ago)
(HTM) web link (scuttlebot.io)
(TXT) w3m dump (scuttlebot.io)
| Vendan wrote:
| Scuttlebutt is a neat concept, burdened by a bad protocol.
| Signing a message involves serializing a json object, signing it,
| adding the signature as a field on that json object, and then
| serializing it again. To verify, you deserialize the message into
| an object, remove the signature field, and then reserialize it,
| and verify the signature against that new serialization. This
| means that all the clients have to have a json serialization that
| 100% matches the node.js serialization that the majority
| (totality?) of current clients use. Any other implementation (I
| wrote one in Go) becomes a nightmare of chasing rarer and rarer
| differences that prevent verification, and if you miss one before
| you serialize your own message with it, suddenly you've poisoned
| your own journal...
|
| All in all, it's something to study and learn from, but I
| strongly recommend not becoming involved unless you are 100%
| happy with being tied into node.js.
| crispyporkbites wrote:
| I've had similar, albeit easy to solve, problems with key
| formats when signing emails.
|
| But that isn't a reason why scuttlebutt isn't more popular. It
| only takes one Go implementation and the problem is solved
| permanently.
| az77 wrote:
| Ju. G
| alexh1 wrote:
| This is how Cosmos (https://cosmos.network/) deals with signing
| transactions, caused _major_ headaches for me trying to figure
| out why my signatures were invalid when sending them from
| Node.js
| goliatone wrote:
| I did something similar for a project that had clients in go
| and node. I solved this by flattening the object key paths to
| strings and sorting the keys, basically. You need to build
| that into the client serialization/deserialization, it feels
| clunky but I had 0 issues and has been working smoothly for a
| long while now.
| masukomi wrote:
| there is work on C, Go, and Rust versions. It runs in iOS and
| Android. So, yeah, not so much "tied into node.js". Yes, that's
| where the majority of the work is, but...
| dboreham wrote:
| Any signing done over structured data has this problem. You
| always need a canonical representation.
| mouldysammich wrote:
| could it use a HTTPHeader style system instead to avoid the
| json serializing back and forth?
| Vendan wrote:
| Just serialize, sign, pack the signature and the raw bytes
| of the serialization. Doesn't matter how, just gotta pack
| the raw bytes, not futz with it
| monocasa wrote:
| You can even have a JSON object containing base64 fields
| for message and signature if you feel it really needs to
| be JSON all the way down for whatever reason.
| Skinney wrote:
| This might actually improve in the near future:
| https://github.com/ssbc/ssb-spec-drafts/blob/master/drafts/d...
| hinkley wrote:
| I spent quite a bit of time with XML digital signatures, which
| is a similar situation, but with even more surface area to get
| wrong. Months to implement, over a year to harden.
| TimJRobinson wrote:
| Once someone has written a library to do this correctly in
| Go/Rust/Whatever isn't this problem solved? Everyone building
| scuttlebutt apps with that language can use that library. It
| didn't seem like this protocol is changing.
| Vendan wrote:
| I tried, and tried, and tried to make that library for Go. I
| failed, cause of serialization issues too involved for it to
| be worth it to me (got to the point where I'd have to write
| my own json implementation)
|
| Just giving people the heads-up that JS seemingly is the only
| blessed language for scuttlebutt
| sneak wrote:
| When I was faced with this (signing a structure), I serialized
| the json into base64, then put that base64 string as a value
| (along with the MAC) into a new json document. It of course
| increases deserialization overhead (json, verify, unbase64,
| inner json) but sidesteps this issue.
|
| I thought about sorting keys and other things like that, and
| the dozen edge cases and potential malleability issues
| dissuaded me for the compatibility issues mentioned above.
|
| How have others solved it?
| VWWHFSfQ wrote:
| I just use JWTs whenever I have to pass signed messages.
| flir wrote:
| Binary protocols. Or out-of-band signing.
| elviejo wrote:
| Do you mind going deeper here?
|
| Which protocols? Can you point to examples?
|
| Because I was about to design a signing Json solution but
| based on the comments here it is a bad idea.
| SyrupThinker wrote:
| After reading about the protocol I came to a similar conclusion
| as you. Although it needs to be noted that the JSON
| serialization is defined as JSON.stringify as defined in
| ECMA-262 6th Ed. plus some more.
|
| To me it's worse that key order must be preserved, which this
| standard does not specify the way I understand it.
|
| Source: https://ssbc.github.io/scuttlebutt-protocol-
| guide/#message-f...
| SupriseAnxiety wrote:
| THE SCUTTLEBUG HELPED MARIO BEAT FOX
| milansuk wrote:
| So the homepage of this project is basically documentation. Where
| others introduce use-cases, you actually teach us how it's done.
| I love it.
| jariel wrote:
| Except for those of us who have no idea really what it is, how
| it would be used, or what the features really do.
| SyrupThinker wrote:
| Then you probably want https://scuttlebutt.nz/ instead, which
| is more of a user introduction.
| Boss123123 wrote:
| Thanks for this post really great to read same here I wrote at
| https://www.triveditech.com/
| audiodude wrote:
| I've used patchwork before, but can someone comment on the other
| applications of the scuttlebut network and where this scuttlebot
| fits in?
| TimJRobinson wrote:
| Scuttlebot is the core server and plugins for Scuttlebutt. The
| Github repo is now called ssb-server. Patchwork uses it for
| replication and adds a gui and new plugins on top of it.
| fabianhjr wrote:
| Yes, there is a decentralized git:
| https://git.scuttlebot.io/%25n92DiQh7ietE%2BR%2BX%2FI403LQoy...
|
| A decentralized book review system and a decentralized chess.
| dang wrote:
| A thread from 2017: https://news.ycombinator.com/item?id=14409187
|
| Points to https://scuttlebutt.nz/ which is related
| masukomi wrote:
| This HN post is linking to the protocol docs. Scuttlebutt.nz is
| about the social network built on top of them. More importantly
| it's the most significant thing that uses the protocol. The
| "apps" that use the protocol are, in practice, just plugins
| that work in one particular app (somewhat geeky app) and that
| particular app is primarily a client for the social network.
| z0mbie42 wrote:
| I recently discovered SSB and was really intrigued.
|
| I would love to hear an experience report from actual users like
| what is working great, and what is bad.
| vector_spaces wrote:
| I love SSB, in principle. The protocol itself is very well
| documented[1]. The community tends to center lofty ideals
| around accessibility, anti-authoritarianism, and social
| responsibility[2][3] which I'm all about.
|
| Unfortunately, I've found the software implementations
| maintained by the SSBC to be "barely working" at best, with
| pretty scant and out of date documentation (to the point where
| code in "Getting Started" sections doesn't actually work) for
| most libraries/tools, PRs and issues lounge for many months
| without a response, and I've noticed a disappointing tendency
| among the SSB community and maintainers to be a bit
| condescending to newcomers and less technical users (not to
| mention cliqueish) in a way that seems in tension with some of
| the ideals that they pay lip service to.
|
| That said, I'm aware that we're all human, and my experience
| here is as more of an observer and tinkerer than active
| participant, so should be taken with a grain of salt
|
| [1] https://ssbc.github.io/scuttlebutt-protocol-guide/
|
| [2]
| https://www.theatlantic.com/technology/archive/2017/05/meet-...
|
| [3] https://www.zdnet.com/article/manyverse-and-scuttlebutt-a-
| hu...
| kseistrup wrote:
| I've been on SSB for some 3 years (with some breaks when I had
| enough of npm). Once you're onboarded it works like a charm:
| exchange of data between peers works swiftly and efficiently,
| to the extend that you can even use it for realtime chat the
| way IRC works. The community is colourful and friendly, and the
| signal to noise ratio is high. I've learned a lot about
| fermentation and growing mushrooms and living off-grid while
| reading posts on SSB.
|
| My biggest frustration has been that all usable clients are
| written in nodejs. I recently took a seven months break to cool
| off from rage over npm (and yarn, for that matter), but now I'm
| back again.
|
| Onboarding can be tricky because there are no central servers
| -- it's 100% p2p -- but I guess it's easier these days than it
| was in the beginning. And if you know somebody who is already
| onboard it shouldn't pose a problem at all.
|
| A better site to start with is probably https://scuttlebutt.nz/
| z0mbie42 wrote:
| Thank you!
|
| The lack of multi-device support isn't too much cumbersome?
|
| Also, do you not fear, due to the gossip protocol, that your
| private messages may be stored forever by peers and one day,
| your private key leaks, and all your conversations are
| publicly exposed?
| dduane wrote:
| Not as much as I fear this from Facebook
| kseistrup wrote:
| It would be sweet if we had multi-device support, but it
| doesn't bother me too much: All my IDs are mutually
| following each other, and I have my "hops" set to 3, so I
| see virtually the same timeline on each device. What can be
| frustrating, though, is that notifications and private
| messages to a given ID are only visible from the device
| with that ID.
|
| I am not overly concerned about leaks of my private key,
| although the risk is certainly there.
|
| One think many people have to get used to, though, is that
| the log is append-only: once you've published a message and
| is no way to delete or undo the message -- is it there for
| "all eternity". The positive side is that you're more
| conscious about what you post and why, because there is no
| way of taking it back.
| jekrb wrote:
| It's been a while, but I used to be an active SSB user.
|
| I hosted SSB pubs and used to post on patchwork semi-regularly.
|
| I thought it worked pretty well as a social network. I
| discovered new and interesting ideas from folks that I don't
| see much on mainstream social media.
|
| I haven't followed the space much recently, and I'm curious
| about how it has evolved over the last year or so.
|
| My favorite memories on SSB:
|
| Someone promoted a book that they had written, and we arranged
| for a sales transaction by talking purely over the network. I
| sent them some amount of Bitcoin, and they sent me the PDF of
| their book. It felt very personal to work with the author
| directly, and side-step payment processors.
|
| I loved taking my laptop out on the train or to a coffee shop,
| and replying to threads and publishing a post to SSB while
| offline. Something about reading other peoples ideas while
| disconnected, and then writing my thoughts, and having them
| automatically sync to the network when I got back on my WiFi at
| home, gave me a different perspective on ways to use
| technology.
| z0mbie42 wrote:
| Thank you!
|
| The lack of multi-device support wasn't too much cumbersome?
| jekrb wrote:
| The lack of multi-device support was a constraint that gave
| me a different perspective of interactions on the web.
|
| My ssb keypair was on a work laptop, so when I changed jobs
| and had to give my laptop back, I lost my keypair. Now, I
| could have exported the keypair and continued to use my
| "account" on my new laptop. The network would have synced
| on my new device, and I'd get all my posts and pictures
| back. But decided to embrace the constraint instead.
|
| When I rejoin the network, I'll have a new keypair, and no
| post history. I think this can have an interesting effect
| on how we view our attachment to data.
| zozbot234 wrote:
| > I loved taking my laptop out on the train or to a coffee
| shop, and replying to threads and publishing a post to SSB
| while offline. Something about reading other peoples ideas
| while disconnected, and then writing my thoughts, and having
| them automatically sync to the network when I got back on my
| WiFi at home, gave me a different perspective on ways to use
| technology.
|
| You can do this with Usenet and most BBS's. Most native IM
| apps will also do this for you, plus of course there's email.
| oosh0Jai wrote:
| I've been using ssb for about a year and a half, and love it.
| The ideas behind the dweb do not get the credit they deserve,
| IMO : p2p data sharing where your data is sent directly to your
| friends rather than centralized on a know-it-all server is what
| the world needs, for anyone concerned about privacy.
|
| If I had to say what I don't like about ssb, it would probably
| be that it's not easy to write an application for it. After
| trying for a while, I turned to Dat and the Beaker browser,
| which let me write frontend applications the usual way, with
| just an api to manage the p2p archives.
|
| Both dat and ssb are javascript projects, so if you want to
| write your dweb applications using an other language, you're
| out of luck. I heard there is a rust implementation of dat
| being worked on that will allow to access an ABI, and thus
| write bindings for other languages, if it gets to conclusion.
|
| Also, for all dweb stuff, the main current problem IMO is
| privacy. You can make private discussions on ssb, you can add
| an encryption layer on top of dat/beaker (I made myself a
| library using libsodium, that's good enough), but the main
| focus is about publishing things for the world, like you
| publish blog posts.
|
| All in all, on my free time, I'm now more interested to work on
| the dweb than on the web, so I can only encourage people to toy
| with it.
| z0mbie42 wrote:
| Thank you!
|
| The lack of multi-device support isn't too much cumbersome?
|
| Also, you do not fear, due to the gossip protocol, that your
| private messages may be stored forever by peers and one day,
| your private key leak, and all your conversations are
| publicly exposed?
| oosh0Jai wrote:
| You're welcome :)
|
| > The lack of multi-device support isn't too much
| cumbersome?
|
| For ssb, it's not a problem for me, because I only use it
| on my laptop. There's a mobile app that exists, Manyverse,
| but you have to make a separate account for it, so usually
| ssb users will have a "john_doe" account and a
| "john_doe_mobile" one. I guess that's good enough.
|
| For dat, yes, it's been a major problem for me for a while,
| because I mainly use it to make my own "p2p cloud", so I
| want my data on mobile as well. There is the Bunsen browser
| on android, quite experimental while able to load dat urls.
| Sadly for me, localStorage doesn't work in it, and it's
| what I use to store my encryption keys. I thought it was
| hopeless for a while, until I started using Termux (which
| basically provided a POSIX environment on android). From
| there, I start dat processes to replicate my data, and I
| wrote a small server to serve them on 127.0.0.1, which then
| allow me to use the app in any browser on mobile.
| Completely hackish, and I can't recommend any sane person
| to do that, of course:)
|
| > Also, you do not fear, due to the gossip protocol, that
| your private messages may be stored forever by peers and
| one day, your private key leak, and all your conversations
| are publicly exposed?
|
| Yes indeed, this is a real risk for any p2p data. I _think_
| it's still better than having them unencrypted on big
| database known for snooping, but we'll have to deal with
| that at some point. I guess the best would be to have some
| sort of encryption capable of autodestructing past a given
| age, I guess? That's a challenge for cryptographers,
| especially given it needs to not allow to just set the
| clock in the past to bypass it. Well, I hope the world will
| surprise me once again :)
|
| On the other hand, when I thought about that, I also
| considered it may actually be a good thing, depending on
| how many years it takes to break the encryption or find the
| keys. If I'm long dead, I'm fine with my data being
| decrypted, because otherwise we'll make the work of future
| historians impossible, if data is sparse and severely
| encrypted so they can't access it.
| justusthane wrote:
| How do you get connected to the scuttleverse? Did you already
| know people who were also using it?
| oosh0Jai wrote:
| Nope, I used a "pub", which are bot accounts that give you
| an invitation and auto follow you to boot the network. The
| way to use them is described here :
| https://scuttlebutt.nz/get-started/
|
| It's a bit cumbersome, but the purpose is to avoid having
| any central authority, if I understood correctly (well,
| except the server running the webpage on which pubs are
| listed :] ).
| thulecitizen wrote:
| Awesome similar DWeb projects for those interested/inspired:
|
| Ceptr/Holochain: http://developer.holochain.org/
|
| DAT protocol: https://datprotocol.github.io/how-dat-works/
| asdkhadsj wrote:
| Warning: A bit of a ramble about my experience trying to use SSB
| for my "application".
|
| I really enjoy SSB, conceptually. In reality I've had a fair
| amount of trouble getting started writing an application in it.
| Notably, I have difficulty knowing what patterns are good or bad
| in SSB.
|
| As far as I can tell SSB utilizes effectively a commit ledger for
| user data. This is fine, but some of the specifics of my
| application mean that I feel the need to create many sub-
| identities so that revisions to a piece of content don't bloat
| the parent identity when syncing to mobile/etc.
|
| Strangely, I've got a fair bit of experience in these types of
| designs - as I've written my own distributed storage for the
| application in question in many different forms (various ledgers,
| CRDTs, etc). Yet my issue is that I'm unsure what is considered
| "bad form" for the SSB crowd. I don't want to write an
| application for SSB but written in some form of SSB anti-pattern.
|
| Combine this with SSB not having well established[1]
| implementations in my language of choice (Rust), and all around
| as much as I want to use SSB as the foundation for my app the
| protocol was just a bit of a heavy burden currently. I care what
| the SSB community thinks because I don't want to just "use" the
| SSB protocol, I want my application to join the SSB hubs/networks
| and share data.
|
| I've looked a lot into this, but I still struggle to conceive
| exactly how my application can fit into the SSB ecosystem, if at
| all. This isn't a complaint post, just sharing some personal
| thoughts and my confusing history with SSB.
|
| Love the project, and I especially love the design of gossip,
| person to person based applications in this day and age.
|
| [1]: Rust does have a handful of SSB libs, but I think I'd be
| looking for a full, batteries included library. Everything I've
| seen is quite low level SSB.
| moojd wrote:
| I've been tinkering with SSB recently as well and I'm also
| finding it difficult to figure out how to interface with the
| wider SSB network in the 'correct' way. The communication
| protocol is well documented and easy to implement if you have
| libsodium available. Building applications on it should be
| straight forward considering it's just an eventually consistent
| event store.
|
| I am having a few issues though. The 'api' that patchwork and
| other parts of the network use on top of the communication
| protocol seems to not be documented at all outside of the
| 'post' message. I'm not sure if it's not documented or if I
| just haven't discovered it yet. For example, I'm having to dive
| in to the code to see how a 'vote' message is formatted. I'm
| not sure how all of the various clients are staying compatible.
| None of the message formats seem to versioned in anyway either.
|
| I'm also not sure what are the 'correct' ways to introduce my
| own message types. Should I use a vendor prefix in the type or
| something? Is it bad form to pollute the network with your own
| message types that other clients can't understand?
|
| I've just started diving in to all this so maybe I just haven't
| gotten deep enough yet. I wrote a little toy SSB implementation
| last weekend to learn the protocol. I've really enjoyed playing
| around with it so far but I'm still not sure if it's the right
| fit for the applications I want to build yet.
| jariel wrote:
| Looks very cool - but to the author, you'll have to write some
| introductory 'what is this documentation' because the single
| sentence on the landing page isn't quite enough.
|
| "It's a magofinisticks peer-to-peer storage functionomatic" is
| how it might read to some people. So what it is, what it does,
| the value of it, some general overview of the features and how
| they work would go a long way.
|
| Looks cool though.
| arj wrote:
| Can run directly (https://github.com/arj03/ssb-browser-demo) in
| the browser as well.
| magwa101 wrote:
| Are these new systems created just to get a fun name out there?
| kseistrup wrote:
| Scuttlebot is a pun on scuttlebutt, which is the cask used to
| serve water on a ship, and the protocol was invented by a guy
| who lives on a boat.
|
| Think of it as "water cooler": a place where people can meet
| and smalltalk.
| matlin wrote:
| This is really cool. I dig the API and mechanisms to exchange
| data. I've been working on a DBaaS (like Firebase) that works via
| message passing between users and now I'm thinking Scuttlebot
| could be great tool to integrate with it.
|
| I wish there was more info on how the replication worked...
| seanb wrote:
| SSB Protocol Guide: https://ssbc.github.io/scuttlebutt-
| protocol-guide/
| TimJRobinson wrote:
| Huge fan of Scuttlebutt and think it could (should) be the future
| of the social internet. I recently quit my job to write about and
| work on decentralized tech full time with scuttlebutt being my
| primary focus. I've written about why I think it's so important
| here: https://adecentralizedworld.com/2020/03/what-is-
| scuttlebutt/
| z0mbie42 wrote:
| It was a good read, thank you!
|
| I also really appreciated your other post[0]. This is what
| allowed me to understand that SSB is not really a social
| network but more a p2p protocol to exchange messages where
| applications can be built on.
|
| [0] https://adecentralizedworld.com/2020/03/a-decentralized-
| plat...
| [deleted]
| rapnie wrote:
| What other decentralized tech did you find promising (e.g. Dat,
| AP Fediverse, Solid, IPFS)?
| alephnan wrote:
| > Decentralized social networks have been tried before, the two
| most well known are Diaspora and Mastadon. With these services
| there [is still moderation]... Scuttlebutt is how I believe the
| social web should function in the future.
|
| Even the Hackernews community which lean towards
| decentralization more so than the general public would still
| argue there is value in moderation.
| TimJRobinson wrote:
| For sure, any network without any moderation is going to
| quickly fall into chaos. I'm working on a post on
| decentralized personalized moderation at the moment which
| should be out in the next day or two :)
| moojd wrote:
| I think this right here is a killer app for decentralized
| social networks and is something I've been thinking about a
| lot. I'm looking forward to your post!
| zozbot234 wrote:
| > there is value in moderation
|
| Of course. Everything in moderation, including moderation.
| [deleted]
| moojd wrote:
| It's possible (and important) that moderation can be
| implemented in a decentralized community. Aether[0] has a
| really interesting take on it where it is essentially
| democratic.
|
| [0] https://getaether.net/
| fabianhjr wrote:
| Decentralized doesn't mean unmoderated. Since it is
| decentralized there is no "global feed" and there is a lot of
| great discussion about moderation tools and processes on
| secure-scuttlebutt. Particularly, check the section of
| follow-graph: https://ssbc.github.io/scuttlebutt-protocol-
| guide/#follow-gr...
| Animats wrote:
| Without an identity provider scheme that doesn't allow
| creating unlimited free identities, the spam, scam, and troll
| problem gets out of hand.
|
| China has a strong identity provider scheme. They tie
| everyone's phone to their government issued citizen ID.
| sneak wrote:
| Moderation feeds can be published into decentralized networks
| just like the content messages themselves. Then readers can
| choose their set of preferred moderators (author/item
| killfile-stream publishers) to apply clientside.
| seemslegit wrote:
| Uh, for something touting e2e encryption and security it would be
| better if the site did not serve over plain http by default
| progre wrote:
| https adds nothing to a page when the only trafic is server ->
| client
| seemslegit wrote:
| Of course it does, HTTP is never only server -> client - from
| preventing a passive eavesdropper from seeing what pages are
| being browsed on the server, cookies, UA fingerprinting etc.
| to active content modification in transit.
| progre wrote:
| Pages are still visable in the tls handshake, no coockies
| on this page (that would be client-> server traffic). But
| yeah, good point about the fingerprinting and content
| modification
| seemslegit wrote:
| No, they aren't - just the hostname (domain name).
| cachestash wrote:
| npm install
|
| sigh...
| alephnan wrote:
| Then again, JavaScript can be thought of as the language of the
| distributed web...
| camdenlock wrote:
| Yeah, this is where I lost interest too.
| crispyporkbites wrote:
| I don't understand why npm install is so bad here? It's just
| a package manager. It's not hard to write your implementation
| without npm if you prefer.
| miguelmota wrote:
| Here's a getting started with scuttlebutt tutorial I wrote last
| year for beginners https://miguelmota.com/blog/getting-started-
| with-secure-scut...
| [deleted]
| nanomonkey wrote:
| The protocol guide is quite beautiful:
| https://ssbc.github.io/scuttlebutt-protocol-guide/
___________________________________________________________________
(page generated 2020-04-18 23:00 UTC)