[HN Gopher] Architecture.md ___________________________________________________________________ Architecture.md Author : todsacerdoti Score : 425 points Date : 2021-02-06 18:46 UTC (4 hours ago) (HTM) web link (matklad.github.io) (TXT) w3m dump (matklad.github.io) | gorgoiler wrote: | This is predictably unrelated but how can someone go to the | effort of having such beautiful typeface selection and still have | an unhyphenated ragged right? | | Great advice though, otherwise. | matklad wrote: | Yeah, I indeed spend some effort to steal the fonts from | asciidoctor and https://www.teamten.com/lawrence/programming/ | (highly recommend both). | | And yes, I myself am struggling with ragged right. I really | wish to have proper text hyphenation and justification, to have | a book-like feel. Sadly, justification without hyphenation | looks ugly, and `hyphens: auto` doesn't work well (and wasn't | supported in chrome last time I checked). | | I am pretty baffled that such basic (in terms of impact, not in | terms of complexity of implementation) feature isn't widely | available. | | Although I am not a web designer, so I might be missing some | simple way to solve this. | vergessenmir wrote: | The k8s project did this very well in the early days with their | design docs. It made the codebase much more accessible than it | ordinarily should have been. There are few projects that do this | and I wish the JavaScript projects did this a lot more. | | If anyone knows of other open source projects that do this, it | would be particularly useful. | mike31fr wrote: | The author: "Keep it short" | | Also the author: "A good example of ARCHITECTURE document is the | one from rust-analyzer" => redirects to an architecture file that | takes 32 whole smartphone screen scrolls to read | baby wrote: | I have a similar advice, but I will go one step further: add | README.md to other folders as well. It is dope to have a map of | your whole system in an Architecture.md (or a README if it's not | too long), but it's even more dope to be able to click through it | and have submaps of how other components are structured. | | Displaying the folder/file structure and explaining what is what | is a must. An example from Diem[1]: consensus | +-- src | +-- block_storage # In-memory storage | of blocks and related data structures | +-- consensusdb | # Database interaction to persist consensus data for safety and | liveness | +-- liveness # RoundState, | proposer, and other liveness related code | +-- | test_utils # Mock implementations that are used for | testing only +-- consensus-types # Consensus | data types (i.e. quorum certificates) +-- safety-rules | # Safety (voting) rules | | I recently digged into dependabot's code, and I found it | extremely well structured. For example you have an Architecture | section in the first README[2] with a diagram (how awesome is | that!) and with links to the README of all the sub components, | which themselves live in the subfolders[3]. | | What I dread the most is going through a new codebase and not | seeing any documentation in internal packages. Like how the fuck | am I supposed to understand anything in there? By reading all the | code? | | [1]: https://github.com/diem/diem/tree/master/consensus#how-is- | th... | | [2]: https://github.com/dependabot/dependabot-core#architecture | | [3]: https://github.com/dependabot/dependabot- | core/blob/main/comm... | xyzzy_plugh wrote: | > Like how the fuck am I supposed to understand anything in | there? By reading all the code? | | This is one of the superpowers of Go: for most Go projects, | this is exactly what I'd do. Just read the code. It's easy to | follow, it's all formatted the same, very little implicit | behavior, and I don't need an IDE to do it. | | Few languages were designed to be read by others. Thankfully Go | is one of them. | systemvoltage wrote: | I am afraid, this doesn't quite cut the mustard. Code just | can't replace a human readable Architecture diagram + | explanations of whys of choices and hows of the system. | Programming language doesn't replace this no matter how clear | and modular it is. While Go is nice, you're vastly under- | appreciating architectural documentation. | michael_j_ward wrote: | When I first started programming, I thought the description | next to the folder / file name on github was actually | describing the item - as you did above, and not just the | message from the last commit that altered the file. | | Many years later, I still believe that's how it should be. | alixanderwang wrote: | I think diagrams are usually quite helpful in accompanying an | Architecture.md doc, especially if it gets complex. | | I work on a diagram maker that syncs with a Github repo. So | whenever you make changes to this architecture diagram, it'll | push changes to the repo, with screenshots directly in the README | (turning the repo into a diagram presentation). | | This also allows people to colocate the diagrams alongside the | code or docs by including the synced repo as a submodule. | | https://github.com/terrastruct-bot/Demo | Terretta wrote: | An alternative is to use an embedded diagramming syntax like | dot or mermaid such that the diagrams are described as version | controlled text and optionally rendered inline by e.g. VSCode. | Cleaner than a litter of side car images in which it may be | unclear what changed. | | See Markdown Preview Enhanced: | https://marketplace.visualstudio.com/items?itemName=shd101wy... | | Or add Mermaid to the built-in markdown preview: | https://marketplace.visualstudio.com/items?itemName=bierner.... | | You can also edit and enjoy these markdown pages with diagrams | on Mac, iPad, iPhone: https://www.mweb.im/ | | Lastly, if anyone is not sure what to diagram for readers of | architecture.md, consider the C4 model: | | https://c4model.com/ | londons_explore wrote: | I would encourage people to have one ARCHITECTURE file per | directory of source code files. Don't duplicate documentation in | these files - if there is a well documented header file for some | module, just link to that. | | In a big project, when I'm hunting for code that does something | and I have no familiarity of the codebase, I want to be able to | follow a chain of ARCHITECTURE documents from the root of the | project to the implementation of the feature I'm looking for. | | If your documentation is poor enough that I resort to finding a | string used in the feature and grepping the whole source tree for | it, then your codebase will be tricky for someone new to get | started on. | kondu wrote: | That sounds a bit excessive. If your project is going through a | lot of changes and refactoring, the multiple architecture.md | files will be hard to keep up-to-date | andrewflnr wrote: | Disagree. If I have a question about the architecture, I don't | want to go hunting through your directory hierarchy for the | right doc file if I don't have to. Not saying you shouldn't | have some kind of design doc in each directory, but I'd say | it's more important to have a single roadmap than lots of | little interlinked ones. If links were enough, I would just | read the code. | azangru wrote: | > If you maintain an open-source project in the range of 10k-200k | lines of code | | What happens past the 200k mark? | qbasic_forever wrote: | You can take an approach like the Linux kernel where _one_ | person is the sole arbiter and approver of all changes. This | person is tasked with having complete, 100% depth and breadth | knowledge of the entire system. It is their responsibility to | ensure changes adhere to the goals of the system. It is of | course an enormous bottleneck, and a big risk for management | (the old 'what if they get hit by a bus, what do we do then?' | concern). | | Realistically, past the 200k lines of code point you aren't | dealing with a codebase anymore; you're an _organization_. You | need knowledge management--where do architecture decisions | live, how are they approved, how are they taught to new | developers, how are they updated as maintainers come and go, | etc. It takes strong engineering management and leadership to | keep it together. | matklad wrote: | It makes sense to invest into more thorough multi page doc, and | have people/process in place to maintain it. rustc guide would | be a good example: https://rustc-dev-guide.rust-lang.org/about- | this-guide.html | xyst wrote: | Yet another piece of documentation that will be out of date in ~1 | year. I have seen and read too many pieces of internal | documentation that is just outdated because it's no longer | maintained or the project changed too many hands and the internal | architecture deviated from the original. | | I guess it's a nice touch, especially for personal projects that | get abandoned and you need to refresh your state of mind after X | amount of months/years. | bezout wrote: | Absolutely, I was just thinking about adding a similar document | to my new project. Apart from contributors, it also benefits | casual visitors who just want to browse around. | brailsafe wrote: | I this is absolutely crucial for almost any project where you | can't necessarily directly instruct someone on how it works. This | is after having failed to contribute anything to open source | projects that I do know the language of, have read the docs, and | couldn't for the life of me figure out how the bits came | together. | danaliv wrote: | As someone who is in week two of spooling up on a multi-million- | line codebase where most of the original authors have moved on to | other projects, please, I beg you to heed this advice. I spend | the _vast_ majority of my time figuring out where a change needs | to happen. The patches themselves are no more than 10% of the | work. | | (The other significant factor is running tests.) | ChrisMarshallNY wrote: | I like the idea. I may do it for some (not all) of my projects. | exabrial wrote: | So one thing we've done is write all of our applications the | exact same way with well defined terms (on a wiki) and a | commitment to no more than 4 layers. | | * Initiators (things that receive, decode, and validate input) | | * Controllers (Business logic containers. One function refers to | one business action) | | * Services (Used by controllers to effectuate commands. Services | absolutely cannot call other services) | | * Cross Cutting concerns (Common model objects, logging, top | level error handling, etc) | | This allows even a new person to pick up a project and orient | themselves immediately. | kenniskrag wrote: | Why shouldn't a service call other services? One Service may | extend the features e.g. network < encryption < http. | Ma8ee wrote: | I don't think the comment implies that a service can't be | layered. | kenniskrag wrote: | layering is just a stronger agreement compared to | composition imho. | Xunjin wrote: | I read the title at HN and I was like "okay seems interesting" | then I saw "matklad" and went "Holy shit, must be great stuff". | | I know I will sound another Rust evangelist, but people this | person is the main maintainer of RA (Rust-analyzer), a LSP | protocol implementation, anyone who tried RLS (Rust Language | Server) then RA knows how great this tool helps you at learning | and developing stuff with Rust. I use the nightly version (which | updates everyday) and ohh boy... Never had the "opportunity" to | caught a nasty bug or anything. | | Also read his blog, it's a joy :) | | Ty Matklad <3 | ser0 wrote: | Along the lines of an ADR, another useful document to have is | Decisions and Opinions. Often choices are subjective, | highlighting these will let contributors know about your | preferences for the project. Often these relate more to linting | styles, choice of libraries, etc. | matklad wrote: | We have something similar in https://github.com/rust- | analyzer/rust-analyzer/blob/master/d.... | | An interesting systemsy difference is that missing style.md | means more work for maintainers (as they need to do more | cleanup), while missing architecture.md means more work for | contributors. | matheusmoreira wrote: | This is so useful. Even a simple description of the source code | tree helps a lot. Otherwise people will have to find entry point | files and recursively search the entire repository until they | find what they're looking for. | mholt wrote: | Ah, we actually have one of these at Caddy: | https://caddyserver.com/docs/architecture | | (The filename on disk is literally "architecture.md" -- it is a | Markdown file rendered by Caddy's template handler: | https://github.com/caddyserver/website/blob/master/src/docs/...) | | It could use some improvement, but it's been really great for | helping people learn how Caddy 2 works at a high level. Beyond | our docs, I always encourage new contributors to thoroughly | explore the godoc and code: it's very well-commented and | organized, especially once you know how it all comes together. A | single document will never be sufficient. But it can help you map | between concepts and code. | | Code search is also invaluable for this; I recommend Sourcegraph: | https://sourcegraph.com/github.com/caddyserver/caddy | | Edit: One other valuable piece is explaining _why_ the | architecture is the way it is. Our architecture.md doc links to a | video that explains how I arrived at Caddy 2 's architecture (and | why it's not arbitrary): | https://www.youtube.com/watch?v=EhJO8giOqQs | Spearchucker wrote: | Thanks for putting that out there. It's super interesting to | see what we consider architecture to be. Your approach appears | operational, focused on how the completed system functions, | i.e. getting a new team member up to speed on the codebase. | | Typically I try to start with tiers (1, 2, n-tiers...?) that | show how the system might be deployed. I then list list layers | (user/facade/business/data), interfaces between these layers, | and components within each layer. I do that for each tier. | | After that maybe something about quality objectives and how | they might be met, eg: availability (MTTF / (MTTF + MTTR) * | 100), efficiency, flexibility, integrity, interop, usability | and so on. | | This leads to a physical delpoyment model, which shows layers | deployed to tiers. And yes, I'm rather fond of Visio. | | Then a bit about approach (dev/deployment and operational | management) risk and stakeholder management, technical reviews | (change control board maybe?), and project reviews. | | To be fair I come from a predominantly critical systems world | (telecoms OSS and BSS, healthcare, transport and some fintech. | And in that world architecture is very far removed from actual | code until eventually. | kortilla wrote: | How do you distill down critical outcomes of the architecture | for people considering using your project? Based on my | experience so far, engineers will look at a giant document, | see phrases like "stakeholder management" and nope the fuck | out. | | What I want to know is, what are the key performance | considerations, failure modes, recovery procedures, etc. | Spearchucker wrote: | Critical outcomes are defined by quality objectives (I | mentioned some above, others are reliability, robustness | and portability). | | People don't consider using my project. There's a client | with a business problem, there's a vendor who solves | problems for clients. The vendor produces an architecture | document that describes how technology will achieve a | solution +. | | There is no noping the fuck out, as this is a hospital | asking you for a one-off to manage/settle insurance | payments. Or an electoral district asking you to merge | three emergency response systems into one. Or Nokia asking | you to tariff calls going through a switch in real time. | | + This is nuanced. Often a client's procurement department | invites a number of candidate vendors to submit proposals | including a design proposal/architecture and associated | cost estimates. Vendors range from the high end (McKinsey, | Bain & Co, Ernst & Young) to the mid-tier (Wipro, Accenture | and so on) to the niche. | kortilla wrote: | > Or an electoral district asking you to merge three | emergency response systems into one. Or Nokia asking you | to tariff calls going through a switch in real time. | | I'm well aware of what it is. I've been on real time | telecom stuff (your last example) and know for a fact | that engineers nope the fuck out of these huge ass | architecture documents that include stakeholders, change | control, etc. | | Inevitably there is some kind of outage or botched | upgrade with lots of finger pointing and then the vendor | covers their ass by referencing "page 248 under heading | 'assumptions about bisectional bandwidth'" or some | bullshit right before the section on 'renegotiating | requirements during a government declared emergency'. | | There needs to be a better way because I assure you, the | people using your system are (on average) barely going to | skim your document. | | Being a "very serious" industry does not change this. | Look at the disaster that was Healthcare.gov. That had | mountains of documents like you describe they overlooked | the simple requirement of scaling identity lookups. | the_arun wrote: | Off topic - what is the theme used in Caddy doc site? looks | super cool. | mholt wrote: | Matt's Custom Theme? :) I dunno, I haven't named it. Glad you | like it! | closed wrote: | I love architecture docs, but find they're often written using a | funny process: 1. Spend a long time writing the | doc. 2. Wait for a person to chance upon it. 3. Hope | you anticipated their questions. | | It seems like the most important thing a person can do is reverse | this: 1. Say who the doc is for. 2. Find | that person. Ask them to try a lil contribution. 3. | Frantically write / revise the doc. | | IMO it's a lot like creating a presentation. The earlier the | feedback the better! | steveklabnik wrote: | I like this idea a lot, but you will cause a _lot_ of people to | bounce at step 2. Or at least, that has been my experience over | the years. No matter how much you reassure them that it is okay | if stuff is confusing and in fact you 'd like to know about it | so you can fix it, they'll say "great" and then go radio silent | 99% of the time. | closed wrote: | I usually reach for a friend, or someone I've met before, | since using the first version of a doc is asking a lot! (And | they're often part of the target audience). | ralmeida wrote: | Sounds right to me too. One quicker way to improve the 'first | process' is to change only step one - do not spend a long | time, but instead write a few paragraphs with what's most | important and/or top-of-mind. Often, this opens the door to | more contributions and questions. | | Of course, update accordingly whenever you find yourself in a | discussion about something with a contributor (no matter if | the architecture doc is even part of the discussion or not). | dognotdog wrote: | I feel like for any long-running project, that person is at | least ME. If I haven't written down some architectural | information for complex projects, when I revisit a project | after it being dormant for half year, I need to poke around | to figure things out again. | | If I have written down architecture notes in the first place, | they are very helpful at this point; and if I haven't, it's a | good time to start because I'll be acutely aware of the non- | obvious parts as I re-familiarize myself with the code. | j1elo wrote: | A question can be seen as a bug reported against the | documentation... | | Apart from answering them, I end up converting 15% - 20% of | questions into some revised content of my docs. | gjvr wrote: | Yep, feedback is valuable. And the earlier the more valuable | (it's like NPV...). | | I love coding so much, and find it really hard to express ideas | in natural language, so that in the end documentation... | doesn't happen as much as it should. | | What I find that really helps is the following: | | 1. Write down architecture specs (with interface specs etc), | _before_ coding. Not bloated, but really minimalistic. 2. | Review these ideas with peers. 3. Happy coding and refine the | docs. | kstenerud wrote: | I've started doing this in my larger projects e.g. | https://github.com/kstenerud/go-concise-encoding/blob/master... | | An architecture document should be the code equivalent of a | combined street map and tourist guide. Its purpose is to bring | strangers up to a minimum level of familiarity with the code as | quickly as possible. That includes where things are, why it was | architected this way, things to look out for, and a few | interesting points of weirdness perhaps. | ibraheemdev wrote: | The specific rust-analyzer architecture document the author | refers to was discussed here a couple days ago: | https://news.ycombinator.com/item?id=26026309 | _hl_ wrote: | I think it's funny how the blog post is essentially an | architecture documentation of the rust analyzer's architecture | documentation. | wiremine wrote: | > One of the lessons I've learned is that the biggest difference | between an occasional contributor and a core developer lies in | the knowledge about the physical architecture of the project. | Roughly, it takes 2x more time to write a patch if you are | unfamiliar with the project, but it takes 10x more time to figure | out where you should change the code. | | This feels about right to me. Not sure a single doc will help | solve that, but even if it cuts the time from 10x to 7x or 5x, it | feels worth it. | notafraudster wrote: | Even on relatively small projects (1k-10k LOC), I incorporate | this, broadly, into my README file for my repo: what are the | various use cases, which components do what, in one sentence what | are the main functions contained in each of the source files, | sometimes a flow diagram showing which functions call which other | functions. | DyslexicAtheist wrote: | what they suggest is very similar to _Architecture Decision | Records_ (ADR 's). https://adr.github.io/ | | TL;DR: ADR's are a design choice for a lightweight process to | store and manage the history over what architecture decisions | have been made in the past and why. They should be tracked within | git so that the history of decisions and how these evolved is | provided _for free_. Just track all this within an `adr /` | subdirectory at the root of each project. | | "Communicating and documenting architectural decisions" - David | Ayers LeadDevNewYork(2019): | https://www.youtube.com/watch?v=rwfXkSjFhzc | NicoJuicy wrote: | Indeed, I was thinking about ADR's too | orthoxerox wrote: | ADRs are more about the "why" (and are absolutely indispensable | in any long-running implementation project). Architecture.md is | mainly about the "how". | ablekh wrote: | I believe that an optimal architectural document should cover | both "why" and "how". Having said that, back when I was | writing and modifying such documents at a CMM Level 3 | division of a large and well-known tech company (using the | waterfall SDLC process!), relevant information was split | between high-level and low-level design documents. I found it | quite inconvenient and think that having relevant sections | (with cross-referenced info) within the same document makes | so much more sense. | syoc wrote: | I really like this DESIGN.md[1]. Might not be as code specific as | what tfa is talking. I find design decisions really helpful when | evaluating if a tool or solution is useful for my needs. | Especially if it elaborates on priorities made in the project. | | [1] https://github.com/google/stenographer/blob/master/DESIGN.md | politelemon wrote: | I find it useful to include an architecture diagram in the README | for small projects, and the best way is to use the VSCode DrawIO | extension. You can directly edit .drawio.svg files and embed them | into the README. You get live editing and up-to-date images at | the same time! | tommyage wrote: | I recently used plantuml to sketch out a component diagram. | Drawing via code is even more convenient | harikb wrote: | Any word on supporting diagrams inside GitHub flavored markdown? | jhund wrote: | I've started using draw.io [1] for diagrams I want to embed in | markdown files. The cool feature about draw.io is that it can | embed the data structure describing the diagram in a png file. | So you get a PNG file that is both source code for your diagram | (so you can edit it later), as well as the presentation (you | can embed png files and they will render fine in Github hosted | md files). | | Then I use the markdown-images package [2] for Sublime Text, | and I can see those png images in my markdown files in the | editor as well. | | Benefits of this approach: | | * Powerful diagram editor, free to use. | | * Editable diagram and embedable image in one file. | | * Diagrams rendered in text editor. | | [1] https://github.com/jgraph/drawio-desktop | | [2] https://packagecontrol.io/packages/Markdown%20Images | arminiusreturns wrote: | I tend to prefer diagrams as code, so yoi can just embed an | image that be updated, version controlled, etc, instead of | ascii diagrams. Any reason in particular you want ascii instead | of images? | ausjke wrote: | pure text is easy to work with most of the time, in fact | ascii diagram is used heaviyl in RFCs | harikb wrote: | Can you describe what 'diagrams as code' mean? | | I don't mind images, just wondering if Github would make it | easier to generate those. Something like the syntax of web- | sequence/uml but not limited to data-flows and something more | simpler, ideally. | petepete wrote: | There are a few tools for creating diagrams with text, | Asciiflow and Monodraw probably being the most popular. | | * http://asciiflow.com/ | | * https://monodraw.helftone.com/ | | From my experience, you don't want to add anything too | complicated or anything that's volatile to code, but in | some cases a high level overview of how bits of an | application fit together can be handy. These days it might | be a better idea to just embed or reference a PlantUML | diagram instead. | | * https://plantuml.com/ | jtr1 wrote: | Looks like there is an open feature request on GitHub for | supporting Mermaid charts: https://github.community/t/feature- | request-support-mermaid-m... | corysama wrote: | You could publish "html" that is actually http://casual- | effects.com/markdeep/ | monocasa wrote: | I always just use ASCII art in code blocks. | _joel wrote: | Same here, simple works. I use this tool to help | http://asciiflow.com/ | amw-zero wrote: | How do you keep this up to date though? That's the biggest | problem with documentation. Having some kind of append-only | format, like ADR, can help, since the documentation specifically | is tied to a decision at a single point in time. | | Still, those can drift from the actual implementation to the | point where they are both misleading and confusing. Such is the | entropic nature of software. | Uehreka wrote: | That's a fair point, although given how "high level" this is | supposed to be, I'd imagine that needing to update it a lot | might actually be a canary that the project in question is in a | lot of flux. | matklad wrote: | Revise it twice a year. If the document gets stale faster than | that, just delete the stale bits: they are probably too low | level for this kind of documentation. | simonw wrote: | I find filing issues whenever I find out-of-date docs helps. | They tend to get fixed pretty quickly, because they offer a | nice change of pace from working on bugs or features. | amw-zero wrote: | Oh, so the solution is to just add another thing to | "remember" to do. That must be what I've been missing all | these years. | | Yes, that's extra thick sarcasm. | matklad wrote: | I disagree with this characterization. ARCHITECTURE.md is | specifically engineered to be low churn, so in this respect | it is meaningfully different from other "keep docs" advise. | | In practice, I personally didn't find it difficult to | maintain half-decent ARCHITECTURE.md, and I am not at all | good with keeping the docs otherwise. | quantum_state wrote: | Great advice! | [deleted] | rationalfaith wrote: | Yup, I do it way sooner. Then again I'm a visionary :p | laurent123456 wrote: | For my app I've started putting architecture related documents | under /spec [0]. I feel documenting every aspect of the | architecture is hard and time consuming, it also needs to be kept | up to date. However it's worth documenting at least the trickier | parts of the app. | | [0] https://github.com/laurent22/joplin/tree/dev/readme/spec | systemvoltage wrote: | There is an important aspect of writing architectural docs (or | any docs for that matter) that is often overlooked. | | Write simply and clearly. | | Too much verbosity and detail is difficult to follow. That's what | the code is for. It is almost a superpower to be able to write | succinctly and clearly. This isn't some contest to showoff your | deep knowledge of a particular niche. I've seen developers get | 'nerdy' with their docs for a lack of a better term. ___________________________________________________________________ (page generated 2021-02-06 23:00 UTC)