[HN Gopher] Launch HN: Sidekick (YC W23) - AI support bot for de...
___________________________________________________________________
Launch HN: Sidekick (YC W23) - AI support bot for developers
Hi Hacker News! We're Jason and Ayan, the cofounders of Sidekick
(https://www.getsidekick.ai/). We made a GPT-driven bot for
developers that runs in Slack and Discord and answers questions
about your developer docs, while automatically keeping them up to
date with new info. You can join our public Slack where you can ask
Sidekick questions about Airbyte, a popular open source data
connector catalog: https://join.slack.com/t/sidekick-
public/shared_invite/zt-1r... Or if you prefer to not sign up to
our Slack, here's a demo video showing the same thing:
https://youtu.be/fPhP1325RkI We're in the process of making
everything open source (there are some contractual issues we're
working through), but our client side code and basic infra is here:
https://github.com/ai-sidekick/sidekick. Providing technical
support to developers has been expensive for companies because they
need to hire skilled engineers to do it. We've seen community
support channels with a 2000:1 ratio of developers to support
engineers - there's no way every question will get answered. We
built Sidekick to make this much easier. It's particularly helpful
for open-source companies/projects because many OSS communities
have a lot of people asking questions, but hardly anyone helping
troubleshoot. We integrate with Slack and Discord, since that's
where developer support is already happening. On the backend, we
use Weaviate to index the data and OpenAI's text-davinci-3 model to
generate responses. In addition to answering questions, Sidekick
can also update .md files automatically with new information. When
someone reacts to a message in Slack with the emoji, Sidekick will
use Weaviate to find the part in your documentation that's most
related to the message, then use GPT3 to merge the new info into
the documentation. Finally, it will submit a pull request on Github
with the changes. We saw that devrel teams are already making
product announcements and helping users troubleshoot common issues
in the community, so we built this feature to save them even more
time. We use GPT for generating the responses and new
documentation, but are relying less and less on it after learning
that you hit a ceiling on answer quality very quickly by using only
GPT and prompt engineering techniques. Here's some of what we
learned trying to prevent hallucinations in our answers:
https://medium.com/@jfan001/how-we-cut-the-rate-of-gpt-hallu...
What we found makes a much bigger difference is the breadth and
quality of the content you can search through, which is why we now
rely a lot more on cleaning and annotating data, which yields far
better results when combined with prompt chaining. For example,
instead of naively chunking data into 1000 token blocks, we parse
the markdown into semantically meaningful sections (e.g.
paragraphs, lists, code blocks) and tag the content with the header
name and document name so it's more likely to surface for searches
that are match for the section it's from, even if it doesn't
exactly match the content in that chunk. One fun thing we also
learned is that when Sidekick gets added to a #help channel, people
who otherwise wouldn't ask questions start using it. It turns out,
there are a lot of "lurkers" who come to these channels to find
answers, but don't want to bother anyone with their issue. Adding a
tool that they can get answers from instantly brings these people
out into the community, giving founders and community managers an
opportunity to reach out to them. To summarize, Sidekick 1) saves
support engineers time, 2) keeps the docs up to date and 3) helps
engage developers in the community. Long-term we want to provide an
analytics product on top of Sidekick so companies can understand
how their product is being used and where there are opportunities
to add more value to their customers (and charge more money for
it). We'd love to hear from the HN community about this product!
Do you think using a tool to search through and update developer
docs from Slack would save you time?
Author : jasonwcfan
Score : 105 points
Date : 2023-03-16 16:21 UTC (6 hours ago)
| enono wrote:
| This looks really cool. I have a couple of questions
|
| 1. How exactly does this automatically keep docs up to date?
|
| 2. Does sidekick work well across multiple repos? (my org builds
| client facing apps so we have about 30 repos)
|
| 3. Is the doc auto updating expected to be in a particular
| format? My docs are very disorganized throughout all of these
| projects since I inherited a lot of codebases.
|
| Overall looks really interesting. If you can actually deliver on
| the use cases you describe I think this will have value to a lot
| of people
| jasonwcfan wrote:
| Hey thanks for asking!
|
| 1. If someone reacts to a message in Slack with the :memo:
| emoji, Sidekick will take the contents of the message and
| search for a chunk in the docs that closely matches the
| content. Each chunk is tagged with the source url on Github.
| Then it uses Github's Octokit client to pull that file, parses
| out appropriate chunk, and use GPT3 to merge the old content
| with the information from the message. Finally, it'll submit a
| pull request to the repo where the docs are stored with the
| changes.
|
| 2. Yep! It can ingest docs from multiple repos and link out to
| docs across all of them depending on which is most relevant.
| You can also add custom logic e.g. restricting it to search
| through certain repos for specific channels
|
| 3. It works better if there's a consistent structure throughout
| the docs, but it will try its best to merge info into docs that
| aren't structured. The risk here is that if you let Sidekick go
| wild it'll lead to very haphazard looking docs, so some
| curation is still needed.
| louison11 wrote:
| Could this one day be implemented as a chatbot available
| throughout the docs? Example: I could look up Mapbox Docs, then
| simply type in what I'm trying to achieve, and get a code snippet
| with JS Fiddle example generated.
| jasonwcfan wrote:
| Yep! It could be, though we had to be be very focused on the
| channel so we started with Slack and Discord.
|
| Personally I think there's less value in an interface over the
| docs vs a community. Engineers might prefer faster answers from
| docs, but companies are going to prefer having their users in a
| central place where they can talk to them.
| rd wrote:
| This is incredible! Such an intuitive use case and from the demo
| it really looks like it does the job well. Congrats!
| jasonwcfan wrote:
| Thank you! Appreciate the support
| debarshri wrote:
| Would be a winner if there is an API integration to embed it into
| a product or have stripe like embeddable components.
| ayanb9440 wrote:
| An API integration is in our roadmap! Send over an email to
| founders@getsidekick.ai for more details.
| elforce002 wrote:
| Interesting. I'll be blunt here: What prevents Slack, Discord,
| etc... from "cutting out the middleman"?
| jasonwcfan wrote:
| Being a platform, this is just one of a thousand different
| things they could prioritize. Sure they could cut us out and
| release a feature that searches through Github docs, but
| frankly there are much more valuable use cases of generative AI
| for them focus on.
| [deleted]
| emil_srnsn wrote:
| Congrats on the launch! We're building something similar [1] and
| are also observing a lot of the same usage patterns.
|
| What seems to happen when the chatbot says "Hmmm, I don't
| know..." is that the DevRel team uses that to guide them to
| missing documentation sections. That's why we're building a whole
| analytics suite and documentation heatmap around the product to
| give insights about doc usage. Here's a quick demo [2].
|
| [1] https://www.kapa.ai/ [2]
| https://www.loom.com/share/cb2b008141ae4ccda50b526a7597873e
| Hansenq wrote:
| Love that you can open up a PR to immediately update
| documentation directly from Slack! Underrated feature; I hated
| updating documentation so that's an incredible change.
| jasonwcfan wrote:
| Thanks for the support! :)
| ghiculescu wrote:
| I might be a bit confused here. Is this just for open source
| projects (and companies built on top of them)? Or can I use it
| for internal repos?
|
| I'd love the latter use case. A bot that scanned our internal
| engineering chat and answered our own team's questions. I can
| provide many examples.
|
| The difference is we don't have public API docs. Instead we have
| code comments + confluence (reluctantly) + readme files in GitHub
| repos + the code itself. I imagine you could train off that and
| build the knowledge corpus over time?
| jasonwcfan wrote:
| We started with external docs because it was easier to ingest
| them add make it useful immediately, but we're definitely
| thinking about internal use cases too.
|
| In theory this should work just as well on Confluence as on
| Github - we'd love to try it out soon.
| axelfontaine wrote:
| A quick trademark search with the USPTO suggests that this may
| not be the best name to use to start a business, as it is already
| taken in literally every single category that is applicable to
| this.
| jasonwcfan wrote:
| Good point - I suppose we'll cross that bridge when we get
| there.
| isalmon wrote:
| I was about to comment on the trademark issue but then I saw
| this top comment. I think you're making a mistake with your
| approach of 'crossing that bridge'. Trademark violation is no
| joke and it's so much easier to change the name early on
| rather than when you have some brand recognition and
| thousands of users. Learn from other people's mistake, don't
| make your own.
| grittybe wrote:
| Ive seen a few similar projects in the last few weeks.
| https://github.com/arc53/DocsGPT has been here on the frontpage
| just a month ago.
|
| And some other similar ones, but it good to see projects that
| will take a OSS approach, they are a few.
| curl-up wrote:
| Great work! In your medium post, in the Post-processing section
| (step 1) you wrote: "If so, we re-run the evaluation prompt and
| force content_contains_answer to be false." I assume you do this
| forcing by adding 'false' to the prompt (as if GPT completed it),
| just like to seed the JSON structure itself. Could you explain
| what you do with the justification in that case? Use it in the
| same way as if the result was 'false' in the first place?
|
| Do you have some stats on how often this even happens? In my
| experience, I have rarely seen these logprob thresholds actually
| get activated, because for binary token choices the probability
| of one of them is almost always >95%, but maybe this is because
| my prompts were not set correctly.
| [deleted]
| jasonwcfan wrote:
| If we force content_contains_answer to false, we always re-run
| the evaluation step first to get a new justification for why
| the content doesn't have enough information. Then we run that
| justification through another prompt to convert it to a user-
| facing response.
|
| It happens pretty rarely, off the top of my head probably 5% of
| the time. However we do see a pretty wide range in binary token
| probabilities anywhere from low 50% to something that
| asymptotically approaches 100%.
| tln wrote:
| Very interesting features!
|
| I'm not sure why it needs to be focused on developer docs vs all
| docs - any reason you decided on that niche?
|
| I'd suggest creating a "demo" slack AND discord to try out. Is
| the demo slack working as intended? I see "Hey there! Looks like
| I'm not set up yet for this Slack workspace. Please send over a
| link to your documentation to founders@getsidekick.ai and we'll
| get you set up!"
| samstave wrote:
| If they want to make Unicorn status ; make this for legal.
|
| And even better - make it so individuals can use it for legal
| documents.
| jasonwcfan wrote:
| Yeah we're thinking any type of product in a technically
| complex field could benefit. Legal, compliance, software,
| etc. For these types of products companies need highly
| trained employees with specific expertise to provide support
| and to create content, which makes it extremely expensive to
| scale.
| samstave wrote:
| Good... afternoon Mr.... Anderson.... I am here with my ...
| ... AIssistants... We are here to help you, when you have
| no mouth to speak.
|
| https://media.makeameme.org/created/SAM-SHUT-UR.jpg
| jojosbuddy wrote:
| Makes a lot of sense. But beware you're up against attorneys:
| no one wants putting their sensitive info in a common report
| for example. Paper files in a vault is "tangible".
|
| I tried to do this decades ago in hollywood (top media
| company) using DAM/AssetMgt tech and ran up against the
| entire attorney corp being protective of their deal
| notes/documents both for confidentiality with the talent and
| to monopolize their connection to that talent (only they can
| negotiate) aka gatekeepers. They purposely wanted "islands"
| of data. Likely similar for any legal outfit in a global
| 2000.
| samstave wrote:
| >> _" But beware you're up against attorneys"_
|
| THIS IS THE FUCKING REASON
| samtp wrote:
| Then you can end up like Donotpay and be in trouble for
| giving legal advice by a chatbot that is not a lawyer and has
| no obligation or directive to be correct.
| astn-austin wrote:
| What if the AI can pass the bar, such as GPT-4. When can we
| allow AI's to be legal experts?
| barking_biscuit wrote:
| When they get off the drugs and stop hallucinating.
| jasonwcfan wrote:
| We started out with all docs actually, and decided to narrow to
| dev docs because 1. Markdown files are structured and easier to
| parse 2. Engineering teams tend to have more influence in their
| orgs. We tried talking to a lot of traditional customer support
| teams and while they were interested in the product the sales
| cycle was long. Whereas with many teams working on developer
| facing products the conversation was just "hey that looks cool,
| let's try it!"
|
| And yep we're running into some issues the new Slack we set up
| for public testing. Sorry for the inconvenience, we should have
| tested it more thoroughly. It should be up in the next hour or
| so.
| gukov wrote:
| Interesting idea, however the sign up process is confusing:
|
| - "Add to Slack" opens a Stripe page which says $49 is due today
| while the landing page claims there's a 30 day trial
|
| - "Get Sidekick now" opens a contact us form
| jasonwcfan wrote:
| Thanks for the feedback! It was just something we added for our
| Product Hunt launch since that community loves promo codes. You
| can use the code SIDEKICKPH2023 for the 30 day free trial :).
| The code works until the end of the week!
| FailMore wrote:
| If that's the price point per month I don't know if it will
| sustain. I feel like there will be a lot of companies like
| this? I imagine there will be some downward pricing pressure
| xbar wrote:
| Sad this is not a new Danger Sidekick.
| throwaway110535 wrote:
| Great demo! Meta question: what'd you use to record the demo?
| Seems like a great tool.
| napworth wrote:
| Wow dude, read the room[0]
|
| [0]: https://news.ycombinator.com/item?id=35181139
| harold_b wrote:
| [dead]
| echelon wrote:
| AI is going to eat the world. It's software + functional
| metahuristics, which enables it to automate an order of
| magnitude more than we ever have in the past. Any problem we
| can describe seems tractable in time.
| lancesells wrote:
| We're running out of fresh, drinkable water. Fetch me the AI!
| rasfuranku wrote:
| This is amazing!
|
| Have you thougth in integrating it to update docs in Confluence?
| jasonwcfan wrote:
| We have! The reason we didn't start there is because Confluence
| is usually used for internal docs, which means a longer sales
| and adoption cycle. The nice thing about developer docs is that
| they're public so we can ingest them and start adding value
| ASAP.
| outcoldman wrote:
| FYI. I would suggest changing the name. First, I thought from the
| title that was something related to Sidekick browser. Also, if
| you search crunchbase for the organizations, there are at least
| two Sidekick AI already.
| swyx wrote:
| congrats on launch! i'm exploring vector database alternatives -
| any painpoints/wishlist items you care to mention with Weaviate?
| jasonwcfan wrote:
| We actually migrated from Pinecone to Weaviate and it's been
| great far and I definitely recommend it. Top 2 features are: 1)
| the way they handle class properties is super intuitive and
| allows you to configure which properties to index, and 2) the
| cross-referencing feature which gives you relational DB like
| features in a vector store. Major props to Bob and the team,
| they've also been super helpful answering our questions.
|
| The main painpoint is probably around the UI for their cloud
| version. The web interface is not great, but if you do most of
| your testing with the client libraries it works very well.
|
| As for wishlist - maybe I'm biased but ironically the docs
| could be better haha it's a bit difficult to find what you're
| looking for sometimes.
| bobvanluijt wrote:
| Thanks, Jason! Very soon, the public beta of the cloud
| version will be live with a brand new UI, and 100% self-
| service hosted Weaviates
| gk1 wrote:
| You probably already know about Pinecone. If you have questions
| or want help testing anything let me know here or
| greg@pinecone.io
| woollyhat wrote:
| How accurate is it in its responses?
|
| Have you had any instances of it writing a load of bullshit to a
| user, and if so how did you address this issue from a technical
| perspective?
| jasonwcfan wrote:
| Never bullshit, but like any application built with GPT it does
| occasionally hallucinate. We published info on how we solved
| that problem here: https://medium.com/p/f3bfcc10e4ec
|
| We've also made some improvements in how we chunk and parse the
| content to make sure the information it finds is useful, since
| we've noticed hallucinations tend to happen when the context
| you give it is irrelevant.
| MatthewB wrote:
| You should put a video on the homepage, front and center, of the
| product in action.
| jasonwcfan wrote:
| Thanks for the suggestion! Just updated:
| https://www.getsidekick.ai/
| herpderperator wrote:
| Appreciate how quickly this was done. When I visited the
| homepage there was already a video, and it made it really
| easy to understand the product.
| jasonwcfan wrote:
| It was great feedback. We've been deep in the weeds
| building so it's hard to pull ourselves up out of the
| weeds, but for something that reads from and updates
| documentation it's probably easier to see it in action than
| than to read about it.
| xur17 wrote:
| Strongly agreed. I was about to post asking if there was a
| video, but luckily saw the link on the OP.
| DeathArrow wrote:
| Do you plan to support Teams also?
| jasonwcfan wrote:
| We might if we hear a lot of people asking about it! Would you
| use it if it supports Teams?
| gregschlom wrote:
| OT, but the Youtube's auto-generated captions on that video
| (https://youtu.be/fPhP1325RkI) often mis-transcribe "sidekick" as
| "psychic" and it's amusing that both can work equally well as a
| name for this product.
| 6510 wrote:
| That should be the line -1 support with maxed out hallucinated
| responses.
| jasonwcfan wrote:
| LOL I agree that's a pretty good name!
| roncesvalles wrote:
| The point about chatbots lessening the shyness barrier in a
| public forum is very interesting.
|
| I think just like (imo) much of the value of the data science
| craze of the last 10 years came not from the final analysis of
| the data but from inducing businesses to start measuring and
| storing operational metrics from every corner of the company in a
| structured way in the first place, much of the business value of
| ChatGPT will come from things like forcing businesses to start
| writing and maintaining good quality internal documentation.
|
| The journey of getting your business to a point where AI can be
| useful for you is itself just as valuable as what AI will do for
| you.
| wwilim wrote:
| It's potentially even better than the big data craze - actually
| using the operational data was often stuck in a vicious cycle
| of (nobody has yet successfully used the data -> data is poor
| quality because nobody motivates making it better), and I bet
| most companies have never actually completed a single feedback
| cycle of their data-leveraging initiatives. Hooking a chatbot
| up to your documentation will be comparatively easier and offer
| near-instant feedback
| jasonwcfan wrote:
| Yeah - we think similarly to how Gong and other sales
| enablement products helped companies quantitatively measure
| and improve sales rep performance, there will be a lot of
| value in products that help measure and improve customer
| service performance. With software products getting more
| complex (salesforce has like 100 products), companies can
| increasingly get just as much value from upselling their
| existing customers as on finding new ones.
| jasonwcfan wrote:
| Exactly. I didn't mention it in the post but the master plan is
| to give companies analytics over their customer conversations
| and documentation. This was technically possibly years ago but
| no one did it because customer support was seen as a cost
| center and the goal was to minimize spend. Companies rarely
| invested in CS instrumentation, or even outsourced the function
| entirely, which I find super odd because no one talks to
| customers more than the CS team.
|
| With language models being where they are today, companies can
| get some really valuable insights out of these conversations,
| but like you said they do need to measure and store the right
| things.
| mgkimsal wrote:
| You're telling me that for all these years, my calls being
| recorded for quality assurance was all in vain? ;)
| Vasyl_R wrote:
| Congrats on this launch!
|
| We at IngestAI[1] are in similar space albeit with different
| focus - we answer from all internal documentation in
| Slack/Discord/Whatsapp. We did a show hn just a few weeks ago[2]
| & has been wild since then - we have close to 10k+ users and
| counting.
|
| If you would like build slack/discord/whatsapp/telegram bots
| answer your customer questions from internal docs - please try us
|
| [1] : https://ingestai.io/
|
| [2] : https://news.ycombinator.com/item?id=34909921
| itake wrote:
| Interesting you're using discord instead of Glitter. I'd think an
| open protocol would help your seo footprint.
|
| The only devs I know on discord are crypto and AI generators.
| 0x457 wrote:
| A lot of rust communities use Discord. Nix community uses
| discord. Espressif, JetBrains, Blink (ipad terminal app), large
| FreeBSD community, Planetary, QMK (keyboard firmware)...
|
| I rarely see alive Glitter community, but I see plenty of
| active Discord communities, so active that I have to leave some
| of them. Every time I see "ask us on glitter", I'm already made
| peace with never getting an answer.
| jasonwcfan wrote:
| We're also considering adding the bot to Gitter and Discourse
| but frankly just haven't had the time to build out those
| integrations yet! It just happened that some of the first
| companies we reached out to used Discord. One of them is a
| crypto company so you're right on the mark there.
| echelon wrote:
| Discord is huge. My experience is that it's way more important
| than Glitter or Slack for open source development. Even if the
| platform itself is closed and inaccessible.
|
| I see it used in Rust, Gamedev, AI/ML (as you say), DevOps,
| JavaScript, ...
| CreepGin wrote:
| Most of Unity Asset Store publishers/devs have Discord
| communities. I'm the author of OneJS [1] and 90% of my support
| takes place on Discord. =)
|
| [1] https://onejs.com
___________________________________________________________________
(page generated 2023-03-16 23:00 UTC)