[HN Gopher] Show HN: Open-source obsidian.md sync server
___________________________________________________________________
Show HN: Open-source obsidian.md sync server
https://github.com/acheong08/obsidian-sync Hello HN, I'm a recent
high school graduate and can't afford $8 per month for the official
sync service, so I tried my hand at replicating the server. It's
still missing a few features, such as file recovery and history,
but the basic sync is working. To the creators of Obsidian.md: I'm
probably violating the TOS, and I'm sorry. I'll take down the
repository if asked. It's not ready for production and is highly
inefficient; Not competition, so I hope you'll be lenient.
Author : acheong08
Score : 205 points
Date : 2023-08-24 11:55 UTC (11 hours ago)
| kepano wrote:
| (Obsidian CEO here)
|
| Impressive! It's fun to see the diversity of ways people
| sync/backup their Obsidian files. The nice thing about storing
| all your notes on your device is that it makes it possible to
| move and edit your Markdown files in many different ways. That
| diversity of solutions is what makes the ecosystem of Markdown
| tools resilient over the long term.
|
| There are already a handful of tools that allow you to sync your
| notes for free, including Git, Syncthing, and some other options
| more specialized for Obsidian (see community plugins).
|
| Obsidian is a small company, we're not VC backed (100% user-
| supported), so the Sync pricing helps us stay in business and
| keep the lights on. We also have a 40% educational discount on
| all our services[1] so you could be paying $4.80 instead of $8 :)
|
| Reverse engineering things is a fun technical challenge, and also
| helps us find potential holes in our system. The main problems I
| see with your solution: 1. it could easily break in a future
| update to the app, 2. "Obsidian Sync" is a trademark, so you
| should consider renaming the repo otherwise it can be confused
| for an official tool -- that would be my only request
|
| [1]:
| https://help.obsidian.md/Licenses+and+payment/Education+and+...
| entilzha wrote:
| While you're here, a killer feature for me would be the ability
| to privately host obsidian sites (similar to publish). Even if
| it required subscribing to publish to download a tarball of the
| site (that isn't public), it could still be worth it. My use
| case is sharing obsidian notes with non-users (eg coworkers) in
| a private way.
| kepano wrote:
| There are a few options for this already. A good one just
| came out a few days ago called Quartz:
| https://github.com/jackyzha0/quartz
| entilzha wrote:
| I tried a few a while back. What I really want is as close
| to 1-1 to obsidian UI as possible. I found with some of the
| plugins that it could be hit/miss on working correctly. If
| I were doing only markdown notes, then wouldn't need
| obsidian ;)
| dansalvato wrote:
| I'm a brand new user and thought all this time that Obsidian
| Sync was 100% cloud-only, so I spent a bunch of time trying to
| spin up a custom sync solution. This comment inspired me to
| look more carefully at it, and now I'm a paid user!
|
| I only wish that https://obsidian.md/sync made it more obvious
| that you can sync a local vault to the cloud. Emphasis on "You
| still own your data, no vendor lock-in" is a huge sell for
| users like me who are typically forced to choose between
| seamlessness vs. owning our data. I'm delighted to learn that
| Obsidian offers both.
| vGPU wrote:
| I was previously using a third party sync solution, but guess
| who just got a new subscription?
| intellectronica wrote:
| This attitude is in no small part the reason I'm happy to pay
| those $8 every month - support a great team building and
| continuously improving a package so useful it has become an
| integral part of my life and, as a bonus, get a slick and
| reliable sync service on top.
| rogeliodh wrote:
| Please consider regional prices. US$8 monthly is high for in-
| develop countries
| kepano wrote:
| This is very hard for us to do because we want to remain a
| small team -- it will introduce a lot of complexity for
| payments/localization
| spondyl wrote:
| This sync server project is pretty handy for a headless daemon
| I had been meaning to cobble together in the background.
|
| The sort of thing that could say; pull down markdown files
| inside a CI pipeline from the sync server which is probably
| better handled using Git for sync but it seemed like a fun
| challenge.
|
| I've got most of a working implementation from trawling through
| minified Obsidian code with the debugger, which of course is
| now a little moot given the OP's project basically captures
| everything needed.
|
| Anyway, one core problem I ran into was how to decrypt file
| blobs. I was dreading the amount of time I would have to waste
| trawling through all the bits required and then they just...
| tweeted it out: https://obsidian.md/blog/verify-obsidian-sync-
| encryption/
|
| I'm sure it would be possible to reverse engineer with enough
| time of course but it's nice to see some "unofficially
| official" user script type stuff. It gives the perception (and
| I'm sure it's accurate) that they dogfood and tinker with their
| own stuff as much as anyone else in the community.
|
| EDIT: Ah, I missed that Kepano had already shared that blog
| post below already but it's still a neat writeup
| andymac4182 wrote:
| I have been looking at building a self hosted implementation of
| this service because of compliance reasons.
|
| The $$$ of the service isn't the issue but making sure there is
| a good compliance option and self hosted to ensure that data
| doesn't leave boundaries and is stored securely on company
| servers.
|
| If there was an option to have a paid self hosted option I
| would happily look at paying for that.
| kepano wrote:
| Down the road we might make something like that for the
| commercial use case -- makes sense.
| jpgvm wrote:
| I had never heard of your service until now but your reply has
| probably landed you $8/month.
|
| Bootstrapped startups deserve my money and commendable
| leadership should be rewarded.
| stereoradonc wrote:
| Give him sync for free! Or atleast let me sponsor the kid.
| belthesar wrote:
| A response like this was exactly the reminder I needed to
| upgrade my Catalyst license. My opsec can be a little on the
| paranoid side, so while I believe y'all have the best of
| intentions and would build sync with the expectation that my
| data is safe, I would prefer to not use your sync service. But
| I use Obsidian so regularly that it's definitely worth
| supporting good work done by folks that align with me. Keep up
| the wonderful work, both as a software company and as a beacon
| for how to be a good steward in an increasingly cynical
| industry.
| kepano wrote:
| For reference, below is our blog post on how you can verify
| the Sync encryption. Most people don't have the time or
| skills to set up their own sync service, but if you do
| setting up a DIY solution can be a fun project.
|
| https://obsidian.md/blog/verify-obsidian-sync-encryption/
| dabeeeenster wrote:
| What a wonderful reply. Thank you and kudos!
| ilyin wrote:
| Time and again I keep being impressed with your selfless
| commitment to building and sharing things in the most humane
| and long-term way. A beautiful example to set, thanks.
| LorenDB wrote:
| > Time and again I keep being impressed with your selfless
| commitment to building and sharing things in the most humane
| and long-term way.
|
| And yet they haven't open sourced the app. That would be
| truly humane and long-term. As it is, it's just another
| proprietary app that, if it dies, can't be continued by the
| community.
|
| I don't think monetary issues should be a concern with open
| sourcing the app - companies like Bitwarden, Gitlab, and New
| Vector (the company behind Matrix) are doing just fine with
| their open source products.
| rexpop wrote:
| Under capitalism, it's reasonable to charge for products
| and services.
|
| The jury is still out on "companies like Bitwarden, Gitlab,
| and New Vector". Kudos to them for pioneering novel
| business models.
| slj wrote:
| Interesting. In regards to your first point, do you have
| justification to back your claim? I'd be interested in
| reading more, or perhaps seeing an example?
| brightball wrote:
| If it dies I'll still have all of my content in easily
| portable markdown though.
| JSavageOne wrote:
| What is it that people love about Obsidian? (not trying to throw
| shade, genuine question as I never understood the hype)
|
| Obsidian is not able to open a local folder on my machine stored
| in Windows Subsystem for Linux, which for me makes it a non-
| starter (my personal knowledge base is stored there).
|
| I personally prefer the Notion UI but hate being locked into
| their cloud, so I created an offline-first Notion alternative
| [1]. Offline mode is totally free (no user registration
| required), and aiming to add cloud syncing tomorrow or so. Still
| in active dev so needs a lot of work, but wondering whether
| others might find this useful. Goal is to make it be able to
| seamlessly sync across multiple data sources, and also easily to
| publish files/folders (eg. can currently turn folders into APIs
| when synced to cloud).
|
| [1] https://mindgarden.app
| EspressoGPT wrote:
| > What is it that people love about Obsidian? (not trying to
| throw shade, genuine question as I never understood the hype)
|
| Tribalism.
| noahmasur wrote:
| > What is it that people love about Obsidian?
|
| When Obsidian launched, there was already a lot of hype
| building in the note-taking community around Zettelkasten and
| Roam Research. With Roam, basically any word in your note could
| immediately link to another note, and you could see backlinks
| in your notes prominently.
|
| However, Roam was both expensive and online-only, so when
| Obsidian came out and it was: a) polished, b) provided most of
| the functionality of Roam, and c) was offline and plain-text
| (also unlike Notion), it was an immediate hit.
|
| The other special feature Obsidian offered was easy and
| powerful customization. The major settings are actually core
| plugins that can be enabled or disabled (like daily journal,
| tagging, or slideshows). Plugins and themes are easily
| discovered and installed from inside the app, which has led to
| a vibrant and popular plugin community. All you need is one
| plugin to be indispensable for users to never leave.
| llimllib wrote:
| - it's simple - (it's also customizable so it
| doesn't _have_ to be, but I happen to like pretty much the
| default features)
|
| - it's free if you don't need their sync - I
| just save my vault on an iCloud drive, which is convenient to
| me, and publish to the web with my own tool
|
| - it allows me to keep my files where I want them
|
| - it's available on every platform I use
|
| - the UI is not distracting, and is Good Enough(tm)
| fletchowns wrote:
| For me, it's comments like this from the Obsidian CEO (from
| this very post):
|
| > The nice thing about storing all your notes on your device is
| that it makes it possible to move and edit your Markdown files
| in many different ways. That diversity of solutions is what
| makes the ecosystem of Markdown tools resilient over the long
| term.
|
| I had seen similar comments in the past, and it just made me
| like Obsidian even more.
|
| I've been using Obsidian for a few months now and loving it,
| and I had been meaning to find a way to support them
| financially in their endeavors. They share the same values that
| I do about the longevity of my notes :) I just purchased a
| Catalyst license.
| viraptor wrote:
| It's a bit subtle and I didn't get it for a long time either.
| But the features are great: I want the content available as
| portable simple text, I love the text representation for the
| extras like graphs and math, I love the offline apps. Look
| through the community plugins to see lots of other cool ideas.
| I started using it this year and I've not been that excited
| about basically a text editor in ages.
| MH15 wrote:
| I've been using the main Obsidian git extension,
| https://github.com/denolehov/obsidian-git. Took some work to set
| it up ergnonomically but it works great now. I enabled auto-
| commit and push on save, and auto-pull when you start the editor.
| No merge conflicts yet between two machines.
|
| Should note I use Obsidian for a journal, so it's pretty much
| append-only.
| rodja_ wrote:
| > I'm a recent high school graduate and can't afford $8 per month
| for the official sync service
|
| What happened to the $47000 you got from Apple [1]?
|
| [1]: https://medium.com/bugbountywriteup/how-i-
| earned-47000-usd-a...
| Arch485 wrote:
| That's about 1 year of tuition in NA; OP is probably saving it
| for that
| freedomben wrote:
| Dang, if money isn't plentiful, there are _way_ cheaper
| options in NA than that. A good school can definitely help
| you network with future successful people and to some orgs
| the name on the diploma can help open some doors, but the
| degree is less needed nowadays than it 's ever been.
| vGPU wrote:
| The OP stated at the bottom that he was saving it for tuition.
| aftbit wrote:
| Nice work, keep scratching itches and you'll go far, whether you
| take this as a career or just a hobby.
| dnxx wrote:
| I'm still looking for a way to securely store and sync my
| Obsidian or Logseq notes with iOS. By "securely" I mean data
| encrypted at rest on the device, but also encrypted on an
| external server. It seems impossible.
| fnfontana wrote:
| I'd like to take this opportunity to say that I've been using and
| following Obsidian's development since 2020 and I really
| appreciate the way your company supports the open source
| community!
| kergonath wrote:
| > I'm probably violating the TOS, and I'm sorry.
|
| Reverse engineering is fine. It certainly is from a moral point
| of view, and also from a legal one in a lot of jurisdictions.
|
| TOS depend more on the fact that you'll behave than on whether
| they can actually be enforced (most have unenforceable clauses).
| w10-1 wrote:
| I think we should be careful when advising young or
| inexperienced people hacking for the joy of it, lest they make
| mistakes that are hard to recover. Time is precious -- better
| to choose wisely than learn from experience.
|
| > Reverse engineering is fine...from a legal [perspective] in a
| lot of jurisdictions
|
| Sorry, what jurisdiction permits one to violate a contract
| against reverse-engineering?
|
| This is a pretty big problem that can't be hand-waved away.
| Companies everywhere have mere copyright on some materials that
| they make available if you agree e.g., not to reverse-engineer
| the interface or distribute the materials, essentially hoisting
| copyright to contract. This is how Java has worked for decades.
|
| > Reverse engineering is fine. It certainly is from a moral
| point of view
|
| In this case, the server is how Obsidian funds their work,
| offering free and paid parts. They're clear about it up front;
| they don't entice you and switch terms after you're committed.
| So how is it morally justifiable in this case?
|
| > most have unenforceable clauses
|
| The "enforcement" is not that they win their case, but the
| threat that they can take you to court and force you to pay a
| lot of money to escape the contract, especially in cases where
| you have significant investments or income. Unless you're
| dedicated to being so irrelevantly small that no one cares, why
| waste time building IP you can't benefit from, or telling the
| world that you don't care?
|
| For people starting out, I would recommend picking viable
| projects, i.e., relevant to others, not illegal, not on dying
| technologies. They can exemplify not only your skills but also
| your judgment, and introduce you to the whole lifecycle of OS
| development if they have any legs.
|
| And it may be best to find an unoccupied niche; in this case,
| there are alternatives already (better to invent than re-
| invent). (Unless you aspire to being a giant-killer :)
|
| That's in part why the standard young-entrepreneur pattern is
| to pick new applications of emerging technologies, particularly
| those that irreversibly disrupt old industries or build new
| markets.
|
| In this case there are many note-taking applications with
| syncing server. The unique project today e.g., might be to
| update an existing sync alternative to maintain a semantic
| model and then publish semantic cross-references, or publish a
| results of AI queries over the evolving set of highly-
| trafficked semantic topics.
| dmje wrote:
| FWIW - the sync service provided by Obsidian is _really_ worth
| paying for. Dropbox / Github just aren't as slick and
| integrated. The mobile app doesn't work without sync (not without
| some really nasty workarounds), the option to "view sync version
| history", the fact that it's rock solid... all worth paying for
| IMO.
| ytechie wrote:
| It works great for me just using iCloud Drive.
| xmprt wrote:
| I haven't tried Dropbox or Github or Syncthing like others have
| suggested but having tried Obsidian Sync I'm not as impressed
| as everyone else seems to be. Maybe I need to set my bar lower
| but especially on my phone, background sync doesn't work well
| and it usually takes a few seconds to load and a few more
| seconds to sync the data which feels like ages considering all
| the data required to load it is already on my phone and the
| diff is a few paragraphs at most.
| acheong08 wrote:
| > view sync version history
|
| Actively working on that. It's a feature I'll need since I mess
| up *very often*
| input_sh wrote:
| Yeah I can think of 30 different ways to make the sync work,
| but I can't be bothered to, Sync works really well, and I feel
| like financially supporting the Obsidian team in some way.
|
| $8/month is a bit steep, but I snatched $5/month early bird
| pricing and I consider that to be a lot more acceptable of a
| price.
|
| Edit: worth pointing out there's a 40% discount for non-profits
| and students:
| https://help.obsidian.md/Licenses+and+payment/Education+and+...
| BrandoElFollito wrote:
| > The mobile app doesn't work without sync (not without some
| really nasty workarounds)
|
| You simply synchronize the folder as with the desktop apps.
| Works right of the box without any workarounds.
| threatofrain wrote:
| What's wrong with using Dropbox or Github?
| bitvoid wrote:
| I can't speak for OP's motives, but one reason one might do
| this is because you can't sync via Dropbox or Github on iOS.
| You can only use iCloud or Obsidian Sync.
| rg111 wrote:
| What about MEGA?
|
| Been using this way for years.
| euazOn wrote:
| Please would you care to share your workflow? I, too, would
| love to sync my Obsidian vault via MEGA, but I have only
| recently switched to an iPhone. On Android, it works
| perfectly with the 3rd party Autosync for MEGA app (paid, I
| believe).
| timrichard wrote:
| I use Obsidian on MacOS, iOS and Android, and keep my Vault
| on GitHub. I use Working Copy on iOS and PocketGit on Android
| to pull and commit/push changes.
| msm_ wrote:
| What's wrong with getting an FTP account, mounting it locally
| with curlftpfs, and then using SVN or CVS on the mounted
| filesystem?
| threatofrain wrote:
| Dropbox is a consumer product, while Github is a nerd's
| product. I put them both to indicate the range of options.
|
| Dropbox or iCloud are solutions you could put on a blog for
| folks who have no interest in learning how to use a CLI.
| oktwtf wrote:
| I had to shell out for another app[0] on the iPhone to allow
| for a fairly seamless GitHub workflow. Not sure if it's changed
| in recent times. It works great, but I'd imagine something more
| tightly coupled with obsidian.md would eliminate that need?
|
| 0: https://workingcopy.app/
| cjwoodall wrote:
| I did out together this blog post a while for doing this for
| free using ashell
|
| https://cwoodall.com/posts/2022-01-02-obsidian-ios-sync/
| princevegeta89 wrote:
| They are still 3rd parties - only good if you're using E2E
| encryption. Additionally, Dropbox free version has a 3-device
| limit at any time. On Android you have to use something like
| DropSync to enable background syncing which may not be ideal.
|
| Git(hub) may be an okay choice though. But there would be
| issues like no SSH support for Git on Mobile clients.
| lijok wrote:
| Does anyone know whether Obsidian can do this with some plugin,
| or whether there's another tool that can:
|
| I'm looking for a documentation tool with a decent GUI for
| editing documentation in, which is backed by git for tracking
| changes, without having to have git configured locally (for non-
| engineers).
| jazzyjackson wrote:
| https://www.gitbook.com/
| acheong08 wrote:
| I think there are a few plugins via git etc. Syncthing also
| seems to be recommended but doesn't work on IOS
| acheong08 wrote:
| Oh sorry. I misread. Ignore my other comment. I forgot my
| password and can't edit using the site. Using third party app
| for HN
| AlphaWeaver wrote:
| What I've long been looking for is a sync service that handles
| syncing these directories of markdown, but then provides value
| added HTTP APIs on top of them.
|
| I can sync my markdown files from place to place with ease, but
| what I really want is the ability to POST myserver.com/newpage
| and have it artificially inject a new Markdown file at the path I
| request.
|
| This could be system agnostic, Obsidian, Logseq, and tons of
| other local markdown knowledge bases could benefit from this.
| icedchai wrote:
| So you want WebDAV support?
| aae42 wrote:
| i use https://github.com/vrtmrz/obsidian-livesync and self-host a
| couchdb instance for it...
|
| this looks like it might be a bit cleaner once it's all fleshed
| out...
| codetrotter wrote:
| CouchDB is pretty nice, and gets less attention on HN than it
| deserves I think.
|
| First time I heard about CouchDB was many years ago. I bought
| an O'Reilly book about it, even. At the time MongoDB was
| gaining a lot of traction, and I think because of that many
| people overlooked CouchDB.
|
| I recently remembered CouchDB because I have some projects that
| could potentially benefit from replication between multiple
| machines. And in the case of my projects I think the database
| level would be an appropriate place to do replication. Hence
| CouchDB is an interesting proposition for my projects.
|
| I installed CouchDB on my laptop for the first time in ages,
| and started inserting documents into it with a third party
| library for Rust. So far it is working nicely.
| surge wrote:
| You're prob not violating ToS since you're not doing it for
| commercial use, and you're actually reinventing the wheel as a
| couple of options for that are already available as community
| plugins. I've been using Remote-save for a while with a almost
| free private S3 bucket and encryption to sync between my
| computers and phone for a while.
|
| https://github.com/remotely-save/remotely-save
|
| It supports more than just S3 (as is listed in the README), its
| just what I use.
|
| That said, what you made is pretty cool. I guess you're trying to
| replace the API/backend similar to Vaultwarden does with
| BitWarden paid service.
|
| For me, its not that I can't afford $8 a month, its just that $8
| is to me fairly steep for basic file storage and sync I get for
| free from some services. $12 a year would be a fair price to me.
| I get some of it goes to support Obsidian development, but still
| seems steep, $2-3 a month would be something I'd subscribe for.
| nsteel wrote:
| Remotely save is excellent. I use it with WebDAV to work
| between my computers, phone and tablet.
| echelon wrote:
| If Obsidian stops acheong08, I'm done with Obsidian.
|
| I want a durable way to edit, sync, and preserve my notes, not
| just a pretty interface.
|
| I'm happy to pay Obsidian for sync, but if they PM this to only
| one distribution channel and subsequently enshittify it, I
| don't want it. Stopping acheong08 would be a bad signal
| forewarning what's to come.
| Xeophon wrote:
| CEO Reply: https://news.ycombinator.com/item?id=37253639
| rs_rs_rs_rs_rs wrote:
| Dude, just pay the $8, no reason to cluch your pearls like
| that.
| echelon wrote:
| I don't think my comment was uncalled for. The OP was
| worried they broke the TOS by writing their own service. I
| wanted to reassure them that any move by Obsidian to squash
| them would be met with incredible scrutiny.
|
| I already pay Obsidian, both monthly and with the one-time
| donation. I mentioned it in my comment. My post was upvoted
| +3 until you responded with snark. Now it's at -4.
|
| Not sure how I'm pearl clutching.
| solarkraft wrote:
| Syncthing actually works pretty well when you have a tool that
| automatically resolves sync conflicts:
| https://discuss.logseq.com/t/automatically-merge-syncthing-c...
| freedomben wrote:
| I'm using Syncthing to keep in sync between a desktop and two
| Android phones over tailscale, and it works pretty good, but I
| have to occasionally open the Syncthing app on Android to
| "bump" it and get it to sync. It's in a git repo (which
| syncthing handles great even though there are some older posts
| online telling you not to do that), and on the desktop I can
| review and "sanity check" stuff and then commit it to ensure it
| is safe. Overall a good solution.
| UltimateEdge wrote:
| Why not exclude the .git directory from being synced, if
| you're not going to be committing on your mobile devices?
| This is what I do.
| karpour wrote:
| I'm using syncthing with Obsidian but of course i have regular
| conflicts which i just deal with manually. Thanks for pointing
| me to this!
| runjake wrote:
| FWIW, I get sync conflicts with Obsidian Sync pretty
| regularly. They're always pretty minor and not worth
| complaining about, but they happen nonetheless, and it may
| not be entirely SyncThing's fault.
| packetlost wrote:
| I wish Obsidian had a better outliner mode/plugin so I could
| switch to it from Logseq, but I just love the outliner/block mode
| too much.
| malnourish wrote:
| Yep; I switched from Obsidian to Logseq because Logseq works
| the way I always wanted to take notes. They are both great
| tools, even if Logseq lacks some in the polish department (and
| honestly, it's pretty good).
| packetlost wrote:
| I really wish Logseq had better/more plugins, a less janky
| sync option, and publish. Maybe it'll get there, but I kinda
| doubt it. I've toyed with creating a plugin for neovim that
| parses Logseq formatted markdown and gives me a similar
| experience, but it would be a lot of work.
___________________________________________________________________
(page generated 2023-08-24 23:00 UTC)