[HN Gopher] Show HN: Web search using a ChatGPT-like model that ...
___________________________________________________________________
Show HN: Web search using a ChatGPT-like model that can cite its
sources
We've trained a generative AI model to browse the web and answer
questions/retrieve code snippets directly. Unlike ChatGPT, it has
access to primary sources and is able to cite them when you hover
over an answer (click on the text to go to the source being cited).
We also show regular Bing results side-by-side with our AI answer.
The model is an 11-billion parameter T5-derivative that has been
fine-tuned on feedback given on hundreds of thousands of searches
done (anonymously) on our platform. Giving the model web access
lessens its burden to need to store a snapshot of human knowledge
within its parameters. Rather, it knows how to piece together
primary sources in a natural and informative way. Using our own
model is also an order of magnitude cheaper than relying on GPT. A
drawback to aligning models to web results is that they are less
inclined to generate complete solutions/answers to questions where
good primary sources don't exist. Answers generated without
underlying citable sources can be more creative but are prone to
errors. In the future, we will show both types of answers.
Examples: https://beta.sayhello.so/search?q=set+cookie+in+fastapi
https://beta.sayhello.so/search?q=What+did+Paul+Graham+learn...
https://beta.sayhello.so/search?q=How+to+get+command+line+pa...
https://beta.sayhello.so/search?q=why+did+Elon+Musk+buy+twit...
Would love to hear your thoughts.
Author : rushingcreek
Score : 195 points
Date : 2022-12-08 17:53 UTC (5 hours ago)
(HTM) web link (beta.sayhello.so)
(TXT) w3m dump (beta.sayhello.so)
| isthispermanent wrote:
| There's a reason why you can't do this commercially and why
| Google isn't doing it already... Pulling the meat of the content
| from a site like StackOverflow ends up as a copyright/anti-trust
| violation.
|
| I'm fairly certain that this was the reason why Google had to
| tamp down it's rich results that were made mostly from Wikipedia
| entries.
|
| More recently it was shopping...
|
| "Google argues that 'rich results' in Search provide more direct
| experience in antitrust suit response":
| https://9to5google.com/2020/12/17/google-search-antitrust-re...
|
| "Google loses appeal, faces EUR2.4 billion shopping antitrust
| fine": https://arstechnica.com/gadgets/2021/11/google-loses-
| appeal-...
| worldsavior wrote:
| But it's siting the sources, how is it a copyright violation?
| fabianhjr wrote:
| Citing does not confer a license
| moralestapia wrote:
| You don't need a license to cite others.
| shawnz wrote:
| But what about when you're also reproducing the content
| on your own page like what's being done here?
| moralestapia wrote:
| It's tricky but you don't need a license for that either.
|
| With tricky I mean that only under very specific
| circumstances you would be infringing copyright laws,
| like maybe if the content was private in the first place;
| but then in that case you wouldn't be infringing
| copyright either, you would just be breaking privacy
| laws/terms.
|
| I honestly can't think of an example where you would get
| in trouble by citing a piece of content that belongs to
| someone else, but I'm not closed to the possibility that
| it could happen.
| thewataccount wrote:
| > only under very specific circumstances you would be
| infringing copyright laws
|
| It's the reverse of this. Any work public or not by
| default is all rights reserved to the owner. The fair use
| doctrine provides an exception from this if you meet
| specific criteria.
|
| An extreme example, you cannot just upload a complete
| movie and just add "credit to disney".
|
| > you wouldn't be infringing copyright either, you would
| just be breaking privacy laws/terms.
|
| Maybe we're from different countries, but with US law it
| would be either under theft, the computer fraud/abuse act
| and/or copyright violations, there are no privacy laws
| applicable here unless we're talking about PII.
|
| Extracting very specific examples from an article or blog
| is almost certainly going to fall under fair use. However
| I've seen several cases where it essentially just returns
| an entire article as the answer which would certainly be
| legally ambiguous.
| crystaln wrote:
| Look up the fair use doctrine. You can reproduce parts of
| content.
| moralestapia wrote:
| It's not, I disagree with GP's argument. Safe harbors in
| copyright law exist to allow this.
| gnopgnip wrote:
| Safe harbor covers hosts of user uploaded content. The
| copyright owner can pursue the infringer, they are not
| protected by safe harbor.
| isthispermanent wrote:
| It's quite likely a fair use violation...
|
| 1. the purpose and character of the use, including whether
| such use is of a commercial nature or is for nonprofit
| educational purposes; 2. the nature of the copyrighted
| work; 3. the amount and substantiality of the portion used
| in relation to the copyrighted work as a whole; and 3. the
| effect of the use upon the potential market for or value of
| the copyrighted work.
|
| Particularly 3 & 4.
|
| https://en.wikipedia.org/wiki/Fair_use#U.S._fair_use_factor
| s
| worldsavior wrote:
| Tho now I think about it more, it might be damaging the
| site money of income (ads, etc). But it's still not a
| copyright violation.
| isthispermanent wrote:
| _" it might be damaging the site money of income"_
|
| Which is one of the key factors of determining Fair Use
| and Fair Use falls under copyright.
| moralestapia wrote:
| It could also be _bringing_ new money to those sites by
| referring users to them.
|
| So, -\\_(tsu)_/-.
|
| The issue with Google had more to do with antitrust
| behavior than with copyright infringement.
| shawnz wrote:
| I might be bringing new money to movie publishers by
| pirating their movies and sending clips to my friends,
| but that's not a valid basis for calling it fair use
| lossolo wrote:
| What's funny is that most of this ground breaking LLMs you see
| now are based on Google published research about transformers,
| and they have better performing models in house than anything
| publicly available on the market.
| moralestapia wrote:
| You're confused, copyright != antitrust violations.
|
| Both sources you provide have zero mentions of the word
| 'copyright' in them.
|
| Those lawsuits have to do with Google's dominating the search
| market and using that to their advantage in ways that are
| allegedly unfair.
|
| Copyright law actually _allows_ a service like Google to exist
| in the first place.
| drusepth wrote:
| According to OP's model, for anyone wondering:
|
| >Copyright infringement and antitrust violations are two
| distinct types of improper use. Plagiarism is an ethical
| violation that occurs when someone attempts to pass off
| someone else's work or ideas as their own, without properly
| giving credit to the original source. It is not against the
| law, but can have serious consequences such as failing
| grades, termination, and difficulty finding new employment.
| On the other hand, copyright infringement occurs when a party
| takes an action that implicates one or more of the rights
| listed above without authorization from the copyright owner
| or an applicable exception or limitation in the copyright
| law, such as fair use. The most common antitrust violations
| fall into two categories: agreements to restrain competition
| and efforts to acquire a monopoly.
| freediver wrote:
| The reason is likely simpler:
|
| - It is expensive (~0.5c per generated answer)
|
| - It is (currently) slow (2-3 seconds to result)
|
| - It is hard to place ads inside direct answers (probably the
| most important)
| iso1631 wrote:
| If it's a good result I'm sure there are many people that
| would pay 1c per search. I've made 16 searches today, far
| less for stuff I didn't find with ddg. If I was after
| something specific I could charge my account with $5 and
| search away.
| freediver wrote:
| I agree, but that is not how Google search (currently)
| operates.
| isthispermanent wrote:
| Thinking more on this... I don't think any of these sites will
| live if they get big enough. And if enough of them pop up it'll
| draw tons of attention from content sites.
|
| If you want to show that data you'll end up having to work out
| a license from StackOverflow. Possible, but far more difficult
| than the current ease of plug-and-play GPT drop-in.
|
| Do we really think Google hasn't thought of this exact thing
| already?
| jchw wrote:
| Stack Overflow user content is licensed under the Creative
| Commons license, so it's possible you actually could satisfy
| the license terms. That said, IANAL, and I have no idea if
| it's possible to fulfill the SA clause without distributing
| the model, or something like that.
| jtxt wrote:
| Very interesting, likely useful for many types of questions.
| Here's a test of say hello:
|
| >why is the earth flat Answer The Earth is flat because it is a
| level surface without raised areas or indentations. This is why
| people can see buildings, hills, trees, mountains and other
| things that are not actually there...
|
| I've been working on this prompt to work with ChatGPT:
|
| >Please answer the below question, and for each researched
| statement of fact, especially numbers, add a bracketed number
| like this: [1] If you need to use math, write each calculation as
| it can be written into a calculator in curly braces like this:
| {13 / 3 = 4.3333333333 } Then after your answer, for each of
| those facts, write search terms that could be used to verify each
| fact from academic sources.
|
| Question: why is the earth flat
|
| >The Earth is not flat, it is a sphere. This can be observed
| through various methods, such as observing the curvature of the
| horizon, the way shadows change throughout the day, and the way
| objects appear to change size as they move closer or farther
| away. [1] Additionally, the Earth's circumference has been
| measured to be 24,901 miles {24,901 / 3.14 = 7899.857143 }. [2]
|
| Search terms to verify statements:
|
| - curvature of the horizon, observation of shadows, changing
| sizes of objects
|
| - measurement of Earth's circumference, circumference calculation
| using pi
|
| Then here's a WIP follow up question:
|
| >Below, I listed a result from each search term with URL and
| excerpts from the page. Please use the following to revise your
| answer, replacing the numbers with source, add actual quotes when
| useful, and cite sources, then instead of search terms, write
| your sources.
|
| It's not perfect, but pretty neat. I imagine that some statements
| of facts and calculations could be corrected by having it talk to
| google/a calculator through a defined protocol, then use that
| information to revise its answer.
| rushingcreek wrote:
| Thanks for the comparison. At the moment Hello is highly
| dependent on the underlying results returned by Bing, which
| makes it non-deterministic.
|
| I just tried your question and got an answer that seemed more
| plausible.
|
| > why is the earth flat
|
| > The Earth is not flat. It is a spherical surface with a
| horizon that is closer when viewed from a lower altitude, and
| it casts a round shadow no matter which direction it is
| pointed. This shape causes a number of phenomena that a flat
| Earth would not, such as ships disappearing over the horizon
| progressively, the coastline or mountain appearing normally,
| upside down, and broken into pieces from across Lake Michigan,
| and stars being visible from different locations on Earth at
| the same time. Additionally, surveys from all over Earth show
| that its shape is everywhere locally convex, confirming that it
| is very close to spherical.
| jtxt wrote:
| yeah, google > bing on this question.
|
| I tested again and got some better ones for yours. Very neat!
|
| Does it base the answer on the search summary?
| wayy wrote:
| The answer is based on the contents of the websites
| returned by the search engine + other some sources.
| revskill wrote:
| It's nice that it allows unsupported countries from openapi.
| aaron695 wrote:
| tonto wrote:
| really like the ability to cite sources, great effort
| lucb1e wrote:
| Out of curiosity, what kind of costs are we incurring per query?
|
| I don't know that much about these AI systems aside from that
| it's apparently too big to run on my own PC. Having a powerful
| but regular desktop system temporarily dedicated to answering my
| queries is quite a lot of compute power already and apparently
| that's not enough for something like Dall-E or chatGPT.
|
| Perhaps to avoid being pinned down on this later ("why is this so
| expensive if it only costs X"), it's probably good to multiply
| any answer by five or ten, or factor in some of the development
| time/costs.
| vineyardmike wrote:
| I can't find the citation, but someone related to OpenAI said
| that ChatGpt cost a few cents per query.
|
| Anecdotally, I looked at running an open-source GPT competitor
| from Meta (OPT), and the requirements for running that are
| vaguely on the scope of "8 GPUs with at least 400gb of GPU
| memory" which would be VERY expensive, even in the cloud.
|
| An 8gpu AWS EC2 instance costs $3.40/hr (p2.8xlarge), and if
| the model runs for 20s per request (roughly how long chatGPT
| takes me to get a response), that gives you $0.02 a request,
| not including things like storage or idle capacity. So a
| request could likely be $0.05-0.1, not including overhead like
| dev costs, training, etc.
| chrisbaker98 wrote:
| > I can't find the citation, but someone related to OpenAI
| said that ChatGpt cost a few cents per query.
|
| Here's the citation, from Sam Altman of OpenAI:
| https://twitter.com/sama/status/1599671496636780546
|
| Also, this person who doesn't appear to actually be
| affiliated with OpenAI estimates the running costs at $100k
| per day: https://twitter.com/tomgoldsteincs/status/1600196995
| 38936627...
| [deleted]
| linuxdude314 wrote:
| Looks interesting but I keep encountering:
|
| The inference service could be unavailable - we have alerts for
| this and will be fixing it soon.
|
| Just an FYI if you aren't getting those alerts.
| rushingcreek wrote:
| Thanks, we're getting 10+ requests per second at the moment.
| Frantically scaling to keep up. Didn't expect this post to blow
| up so fast!
| sillysaurusx wrote:
| 11B is a chonky model! Curious to hear what kind of hardware you
| used to train it. GPT-J (a 6B model) was trained on a v3-128, I
| think, so this is about 2x that.
| rushingcreek wrote:
| This latest model was trained on a distributed multi-node
| cluster of 8 servers each with 8 NVIDIA A100 GPUs. Need a lot
| of GPU memory when training with long sequence length.
| Jayakumark wrote:
| Was it trained on Common Crawl Dataset or custom scraping ?
| alchemist1e9 wrote:
| For how long do you know?
| EastSmith wrote:
| Looks like a single NVIDIA A100 costs $23,667.29 [1]? So 8
| servers X 8 cards X $23,667.29 means the whole setup costs
| $1,514,706.56?
|
| https://www.dell.com/en-us/shop/nvidia-
| ampere-a100-pcie-300w...
| rushingcreek wrote:
| At list prices, perhaps. Importantly, we don't own the
| hardware. H100s are coming soon and they are much faster.
| alchemist1e9 wrote:
| As a follow up to that question, which I'm also very curious
| about.
|
| Do we have rough estimates of the training GPU compute needed
| to train GPT-3 and has OpenAI said anything about compute
| resources used to train ChatGPT?
| s1k3s wrote:
| Other people calling copyright issues but this is far from GPT's
| biggest problem..
|
| > how to build a successful startup
|
| > Building a successful startup requires hard work, dedication,
| and the right resources. It is important to have a good business
| plan that includes both short-term and long-term goals, as well
| as how you plan to grow your business. Additionally, it is
| important to be original and not copy anyone else's ideas.
| Networking is also essential for building a successful startup,
| as word of mouth is stronger than ever. Professional networking
| can help you find the right people and talent, while also helping
| you build up an environment in which everyone participates.
| Finally, it is important to keep up with the latest trends and
| maintain a balance between life and work.
|
| It's literally spelling out generic stuff that you can find in
| any reddit post. I understand this is what's been trained on, but
| relying on this as a knowledge base is very far fetched
| wayy wrote:
| What is the correct/expected answer to that query?
| rushingcreek wrote:
| The more specific you are, the more detailed the answer will
| be. If you simply add "step by step":
|
| > how to build a successful startup step by step
|
| > Building a successful startup requires several steps. The
| first step is to identify a problem and solution that can be
| filled by your business idea. This will help you create a
| website that meets the needs of your target audience. You
| should also write a company plan, which is necessary for
| obtaining startup funding. Once you have identified a problem
| and solution, you can build an MVP . This should focus on the
| most crucial must-have features needed by your target market.
| It is important to involve potential customers in the building
| process and ask for their feedback. After building the MVP, it
| is time to launch it. If you have a large following or media
| attention, organizing a media launch with a presentable version
| of the product may be a good idea. However, most startups don't
| need to make a big deal out of their launch.
| AndrewKemendo wrote:
| Then it sounds like you want a "clarification" feature, as your
| query isn't in any kind of discernible language that doesn't
| require extreme context assumptions
|
| For example, if I was a librarian and someone walked up and
| said literally: "how to build a startup" I would ask follow up
| questions to help you ask a better question, perhaps something
| like: "What would be the steps I would have to take in order to
| build a profitable company"
|
| So is that what you are asking for it to do?
| fleddr wrote:
| That's actually a quality answer given the garbage input.
| titaniumtown wrote:
| no source code, sad
| ddtaylor wrote:
| Almost every other search engine is closed source.
| titaniumtown wrote:
| and?
| MarcelOlsz wrote:
| I asked it what the fingerings for a C trumpet scale are and it
| instantly mentioned a fourth valve. Still a ways to go.
| rushingcreek wrote:
| Just worked for me:
| https://beta.sayhello.so/search?q=+fingerings+for+a+c+trumpe...
|
| Unfortunately searches can be non-deterministic, as the links
| Bing returns to us can change from query to query even if
| search wording stays constant.
| jdpigeon wrote:
| This is amazing!
| codewithcheese wrote:
| Interested to see a GPT version of this and compare.
|
| How about a browser extension that scrapss Google results and
| asks GPT to summarise?
|
| There are many prompts beyond summarisation that would be
| interesting, for example ask GPT to present conflicting point of
| views in the results.
| wayy wrote:
| We're always actively exploring different prompting - a general
| question answering prompt most definitely won't work for all
| searches. Finding what people agree on and disagree about in
| e.g a reddit thread could be very useful.
| theGnuMe wrote:
| What does aligning models to web results mean?
| wayy wrote:
| The answers are explicitly based on information found on the
| web instead of some knowledge found within the model
| parameters.
| odysseus wrote:
| > how does a tuple differ from a list and a dictionary in python
|
| > A tuple is an immutable data structure in Python that stores
| key-value pairs. It is similar to a list, but it does not allow
| for any changes after it has been created. Tuples are represented
| by and lists by [].
|
| This is not the whole answer given (it talks about dictionaries
| as well), but it's missing a few things that could be useful:
|
| * It doesn't actually say what tuples are represented by (the
| characters seem to be stripped out)
|
| * It doesn't show examples
| rushingcreek wrote:
| Thanks for the example. We'll take a look.
| aw1621107 wrote:
| Isn't that answer technically incorrect as well? A more precise
| answer would be that tuples _can_ store key-value pairs, but
| even then I think that part of the answer is unnecessarily
| narrow.
| whatshisface wrote:
| Tuples are integer-indexed and contiguous in Python, you
| can't index them with strings, and you can't have sparse
| arrays.
| nemacol wrote:
| I am amused that, once upon a time, it was a not uncommon joke
| that an inept user might sit at their computer and type into a
| search bar "Please take me to my email" or ask their search
| engine "I would like a recipe for cookies".
|
| Turns out those folks might have been ahead of their time. :D
| adamsmith143 wrote:
| >I would like a recipe for cookies"
|
| This has been doable on Google for years. Maybe not the same
| kind of intent understanding but enough keywords are there that
| you would end up getting lots of results for cookie recipes.
| wayy wrote:
| yep, it's super interesting to watch how cheap and plentiful
| compute starts to change the way we live
| lucb1e wrote:
| > "Please take me to my email"
|
| Results in
|
| "To take you to your email, you will need to sign in to your
| account. Once you are signed in, open your inbox to check your
| mail. You can also use the Help tab to contact customer support
| or see training videos. Additionally, you can right-click an
| email in the Message List to show more message options and
| right-click the Reading Pane for additional email commands such
| as translation. Finally, you can use Search to find other
| commands or search through your email, contacts, and calendar."
|
| Citing resources from google and microsoft support.
| gavinray wrote:
| It would be great to be able to leave feedback on individual
| search results
|
| For example, the explanation text of _" What is RCU (Read-Copy-
| Update) and how is it different than a Mutex?"_ was (mostly)
| great:
|
| https://beta.sayhello.so/search?q=what+is+rcu+%28ready-copy-...
|
| The code examples were not the best. I looked for a way to relay
| this information on the search results but couldn't find it (only
| "GOOD/BAD" vote)
|
| ---
|
| Can you just put a <form> tag or an <input> at the bottom and
| hook it up to some DB table?
|
| EDIT: It seems to re-generate the text each time?? So, if someone
| takes my word for it and then gets an awful description, that's
| why.
| wayy wrote:
| You can leave feedback on individual search results via the
| check and X buttons below the answer. Leaving feedback e.g
| votes on each code snippet is on the roadmap.
| gavinray wrote:
| Right but that's just a blanket "good" or "bad"
|
| I hit the green check, but I wanted to leave a detailed
| comment
| wayy wrote:
| Ah we only put the option for detailed comment on negative
| feedback (try clicking the x and a form will pop up). Will
| also give that option for positive feedback in the future.
| gavinray wrote:
| Oh, got it, well at least there's a way to leave it!
| lcnPylGDnU4H9OF wrote:
| > what is the difference between an objects class and its
| singleton class in ruby
|
| > The main difference between an object's class and its singleton
| class in Ruby is that the singleton class is a metaclass of the
| object, which defines the methods that can be called on the
| object. This allows for more specific methods to be defined for a
| particular instance of the object, rather than having to define
| them in the class itself. Additionally, the singleton class is
| hidden from the inheritance chain, so it does not show up when
| calling ancestors, but it is the first ancestor when it comes to
| dispatching methods.
|
| I don't see anything wrong with this explanation.
|
| > how do i modify a ruby object's singleton class
|
| > The singleton class in Ruby is a special type of object that
| contains all the methods defined for an object. It is used to
| define unique behavior or methods which the other objects of the
| class do not own. To modify a Ruby object's singleton class, you
| can open the singleton class with the syntax class << obj and add
| code to it or override existing methods. You can also use the
| extend method to include a module in the singleton class. This
| will make the module methods available on that object.
|
| Given the phrasing of the question, the first two sentences here
| seem a bit out of place. It seems like it should just start with
| the third sentence. The explanation otherwise is correct about
| both suggestions.
|
| > what does object#method in ruby do (I wrote it as
| Object#method)
|
| > The object#method in Ruby is a method that can be used to
| access the methods of an object. It returns a Method object which
| represents the given method, and it can be invoked indirectly
| through Method objects.. For example, Enumerable#sort uses #< =>
| to compare two objects and return one of the following values:
| -1, 0, 1 or nil. If self is smaller than other, then #< => should
| return -1, if self is equal to other, then #< => should return 0,
| and if self is bigger than other, then #< => should return 1.
|
| The first sentence is correct but the second kinda reads like
| nonsense. I'm not sure exactly what that sentence is intended to
| mean. The rest of it appears to be mostly unrelated (except that
| it's Ruby-specific).
|
| > what is the officially recognized way to remove a peanut butter
| and jelly sandwich from a vcr by the united states department of
| sandwich-in-vcr removal
|
| > The officially recognized way to remove a peanut butter and
| jelly sandwich from a VCR by the United States Department of
| Sandwich-in-VCR Removal is to use a butter knife or spoon to
| spread the peanut butter on both slices of bread. This will help
| to prevent the moisture from the preserves from entering the
| slices of bread. Once the peanut butter has been applied, the two
| slices of bread should be pressed together quickly to avoid any
| leakage. To ensure that the preserves are thoroughly contained,
| the open sides of the sandwich can be sealed with a plastic wrap
| or wax paper. Additionally, other ingredients such as fresh
| fruit, savory ingredients, and texture can be added to the
| sandwich for an upgraded flavor.
|
| :)
|
| (Actually, upon reading, I notice that the explanation of how to
| make a PB&J doesn't include spreading the J.)
| breck wrote:
| Interesting!
|
| It's pretty slow for me. But will give it a more extensive test
| later when it's faster.
|
| Small nit: someone on here recently suggested it might be better
| to build products "made with rage", rather than "made with <3".
| not sure, perhaps the truth is somewhere in the middle.
|
| Edit: this is really cool! If you are raising money please send
| me wire instructions or a NEAR coin address.
| rushingcreek wrote:
| We've scaled up now, so it should be a lot faster :)
|
| Not raising right now, but feel free to reach out to us at
| founders@sayhello.so.
| metacritic12 wrote:
| The current market leader in this seems to be
| https://metaphor.systems/
|
| Could you talk about how it's similar or different?
| rushingcreek wrote:
| Metaphor is cool, but it's a different product. They use
| language models to generate links, while we use them to
| generate answers.
| Y_Y wrote:
| > The inference service could be unavailable - we have alerts for
| this and will be fixing it soon.
|
| > In the meantime, try searching this on Google or DuckDuckGo.
| You can also prefix your question with !g or !ddg shortcuts. You
| may have to enable popups for this to work.
| rushingcreek wrote:
| We were temporarily overloaded. Fixed now.
| qclibre22 wrote:
| Still getting "inference service could be unavailable".
| rushingcreek wrote:
| Try again now, we've increased capacity. Increasing more
| now.
| qclibre22 wrote:
| Worked!
| JadoJodo wrote:
| One of the things that worries me about ChatGPT-driven products
| is the issue of confidence. We've seen in other posts where it
| can generate convincing-sounding instructions from thin air. How
| can I trust that following the instructions for "How to set
| cookie in FastAPI?" is going to be legit?
|
| We obviously also have this issue with the open web, but real
| authors are almost universally doing it for others' benefit (as
| opposed to their detriment/trolling). ChatGPT generating
| instructions for processes isn't "real" and so it has no notion
| of them being "correct" or "incorrect"; they just "look" the same
| as the training data.
| wayy wrote:
| We think the solution is simply having good sources and answer
| transparency. If you mouse over part of the answer, we try to
| show you the source of that sentence. Obviously this system is
| early and will improve over time, but if can easily check if an
| answer is from say, the FastAPI official documentation, then
| the false-confidence effect of these models become less of an
| issue.
| motoxpro wrote:
| Been using this for a little bit. Fascinated to see if this gets
| more traction now that it's being pitched in this way and with
| all of the buzz of "ChatGPT is going to eat searches' lunch"
| rushingcreek wrote:
| So happy to hear that you've been using Hello already. We've
| been picking up the pace of improving our models.
| aix1 wrote:
| I'm getting "The inference service could be unavailable".
| rushingcreek wrote:
| Should be fixed.
| return_to_monke wrote:
| sadly you guys still got the Hug of Death :( :)
| rushingcreek wrote:
| Yep gives us 5 minutes, we're frantically scaling the infra
| :))
| trynewideas wrote:
| Asked the same history question[1], "what was the french-speaking
| population of france during the french revolution", multiple
| times and got different answers each time. Each answer was wrong
| about at least one significant detail, and in one answer it
| decided to just not try to answer the question I asked.
|
| Attempt 1: The population was about 25 million (in line with
| estimates) and only 3 million spoke specifically Parisian French
| (a potentially accurate statement, but not what I asked). The
| sources it cited had contradicting information -- 23 million
| instead of 25 million total population -- or additional context
| that would've answered the question accurately, such as about
| half of the population speaking some form or quality of French.
|
| Attempt 2: "The French-speaking population of France during the
| French Revolution is estimated to have been around 29 million
| people" -- completely wrong across the board. One cited source is
| a Statista user-created graph that gives 24.8M for revolutionary
| France and 29M for France in 1800; it also has no details about
| language use. "This figure is based on demographic analysis
| conducted by Gudin and confirmed by a crosscheck with the graph
| showing population changes in France between 400 B.C. and 1975
| contained in Colin McEvedy and Richard Jones, Atlas of World
| Population History." This is lifted almost verbatim from another
| cited source, but the source's next paragraph acknowledges
| discrepancies important to the context, including potential
| errors by Gudin or his translator. The source also has no details
| about language use.
|
| Attempt 3: "The French-speaking population of France during the
| French Revolution was estimated to be around 55.81 million
| people, according to a 2007 Adult Education survey by the Insee.
| This survey found that French was the mother tongue of 87.2% of
| the total population, followed by Arabic , Portuguese , Spanish
| and Italian . People who spoke other languages natively made up
| the remaining 5.2% of the population." Completely wrong. The only
| cited source is the "Languages of France" Wikipedia article.
|
| 1: https://beta.sayhello.so/search?q=what+was+the+french-
| speaki...
| wayy wrote:
| Thanks for trying it out - we're still quite early so the model
| isn't going to be perfect + we're focused on programming
| related queries at the moment. What is your use case? Are most
| of your searches related to history?
| trynewideas wrote:
| Most of my searches are related to researching and fact-
| checking potentially spurious statements, like the ones that
| Hello apparently produces.
|
| This particular query about the French revolution seems to
| give GPT fits across its iterations, and I suspect it's
| because:
|
| - the most correct answer is "I don't know"; when answering
| questions, GPT isn't really trying to answer a question by
| reasoning through it, it's trying to mimic what usually
| happens after someone asks a question, and especially online
| that usually isn't someone saying "I don't know"
|
| - the most authoritative sources on subjects like this
| include books, and GPT doesn't seem to read a lot of those,
| or if it has then it doesn't seem to be able to cite them
| without inventing false details about the sources themselves
| (titles that don't exist, attributing works to the wrong
| authors)
|
| Seeing "model that cites sources" made me excited that there
| was a solution to the second point. But specific to Hello:
|
| - the most authoritative web sources are historians who cite
| sources, which aren't often written for SEO, on popular
| platforms, or not paywalled
|
| - a search result being in a first page of Bing/Google
| doesn't make it authoritative
|
| - this doesn't seem to stop GPT from coming up with false
| inventions/hallucinations that aren't in or relevant to the
| cited source at all
| wayy wrote:
| You're right - there are so many good sources not often
| surfaced by a cursory web search like books and experts.
| Adding better sources is something we're improving on. A
| simple next step we're looking into is to expand sources to
| research papers e.g explicitly pulling from arxiv on some
| queries.
| trynewideas wrote:
| There's zero attempt to cache answers or cited sources, I
| suppose, because I keep getting different answers even from the
| same cited source just by clicking the search icon on the bar
| with the same query:
|
| "The French-speaking population of France during the French
| Revolution was estimated to be around 25 million people"
| (wrong, citing that same Statista graph that doesn't mention
| language use)
|
| "It is difficult to accurately estimate the French-speaking
| population of France during the French Revolution, as there was
| no official census taken at the time." (honestly the most
| accurate answer yet, but citing that same Statista graph)
|
| "It is impossible to accurately estimate the French-speaking
| population of France during the French Revolution, as there is
| no reliable data available. However, it is estimated that
| around 80% of the population spoke French in 1800." (first
| sentence is correct, second is wrong, citing that same Statista
| graph)
|
| "At the time of the French Revolution, less than half of
| France's population could speak French. This was due to a
| combination of factors, including the fact that French was an
| extremely popular language with the elite and higher classes,
| as it was adopted by nearly all European courts and it even
| reached the other side of the Atlantic." (cites a babbel.com
| article; I think the first sentence is the first time the
| answer approaches both accuracy and consistency with both the
| question and source, largely by plagiarizing part of the
| source. It still omits useful context about the lack of
| literacy/fluency that's in the _same sentence_ lifted from the
| source. Then it goes back off the deep end with the second
| sentence, which the source uses to emphasize how broadly French
| was spoken, and not as a reason why so few people in
| revolutionary France spoke French.)
| wayy wrote:
| Interesting, we are trying to wrangle with the nondeterminism
| but sometimes it can't be helped as Bing itself can produce
| different results. Always actively working on the model
| though.
|
| The citing sources feature can definitely be improved - right
| now it works at a sentence granularity and insists on finding
| the best source when sometimes not appropriate. Thanks for
| pointing all of this out
| notafraudster wrote:
| I asked three clearly stated, graduate level questions in my
| field and all three answers were useless.
|
| > what is the relationship between commodity prices and civil
| war? answer in a way that cites published scholarship.
|
| > The relationship between commodity prices and civil war is
| complex. Research has found that a negative, one-standard-
| deviation commodity price shock in a single year can increase the
| risk of civil war outbreak in Sub-Saharan Africa by between 40%
| and 70% of the baseline risk of civil war outbreak. This holds
| true for all large countries in Africa, the Middle East, Latin
| America, and Asia. Additionally, there is evidence to suggest
| that a rise in international commodity prices may also lead to an
| increase in civil wars. For example, research published in the
| May 2010 issue of the Economic Journal found that a 10% increase
| in the price of international commodities was associated with a
| 22% increase in the probability of civil war outbreak in African
| countries that are big exporters. Furthermore, research from
| Oeindrila Dube and Juan F. Vargas of Universidad del Rosario
| concluded that "price shocks generate contradictory pressures on
| conflict.
|
| This definitely synthesized information, but it's nonsense. The
| beginning of the thesis is that a negative price shock increases
| civil war risk. Then it suggests a positive price shock increases
| civil war risk. Then it concludes that price shocks generate
| contradictory pressures on conflict. Dube and Vargas is a famous
| paper, so it was correct to cite it.
|
| If the answer is that research is mixed, specify that. If the
| answer is that the research is unambiguous but it depends on the
| location or commodity type, the answer should disambiguate.
|
| > are ideologically extreme candidates punished in u.s. house of
| representatives elections? what about primary elections?
|
| > No, ideologically extreme candidates are not punished in U.S.
| House of Representatives elections. In fact, there is evidence to
| suggest that extremism may actually be rewarded in some cases.
| For example, the landslide defeats of Barry Goldwater and George
| McGovern in presidential politics have been seen as evidence that
| American electorates punish extremism. However, this view is
| thinly supported by systematic evidence. Additionally, there will
| be at least one special election to the United States House of
| Representatives in 2023 during the 118th United States Congress.
| This suggests that ideologically extreme candidates may actually
| benefit from these elections.
|
| Another nonsense answer. The first sentence appears to gesture to
| the finding in the Zaller et. al research linked. Then it
| plagiarizes another sentence from the abstract of that paper.
| Then it introduces a completely nonsense non sequitur and uses it
| in support of a point that is not in evidence. It never answers
| the second question at all. The papers linked are all relevant,
| but not sorted by any useful metric and there are major papers
| not listed.
|
| > what are the disadvantages of using two-way fixed effects to
| measure treatment effects in a natural experiment
|
| > The two-way fixed effects regression is a popular method for
| measuring treatment effects in natural experiments. However, it
| has some drawbacks that should be considered before using it.
| First, the model does not account for heterogeneous effects
| between groups or over time, which can lead to misleading
| estimates. Additionally, the LSDV estimator is inconsistent for a
| dynamic panel data model with individual effects, whether the
| effects are fixed or random. This means that the bias of the LSDV
| estimator in a dynamic model is generally higher than other
| estimators. Finally, the 2FE model may produce biased estimates
| due to the fact that it only adjusts for unobserved unit-specific
| and time-specific confounders at the same time. Alternative
| estimators such as the instrumental variable approach or the
| difference-in-differences approach may provide more accurate
| estimates of treatment effects.
|
| First three sentences are fine. Fourth sentence introduces an
| estimator that is not previously defined, probably by
| plagiarizing one of the source papers. The point starting with
| "finally" is a restatement of the limitation under "first". The
| instrumental variable approach is another econometric technique
| but it is not a substitute for 2WFE. D-i-d is most commonly
| implemented as 2WFE as an estimator, and so it is also not a
| separate technique.
| ethanbond wrote:
| I think you're holding a (suitably) high bar and I'm still
| wildly impressed it got as close to it as it did. To me these
| seem extremely "synthesis-heavy" as opposed to merely fact-
| finding, which is what I would have seen as the more
| obvious/achievable task?
|
| I do wonder if some of the first question is just that the
| answer an expert _should_ give you is similarly equivocal
| (granted being more explicit about the equivocality) and we 've
| just gotten too enamored with the idea of clear answers even
| when they don't actually exist.
| 6gvONxR4sf7o wrote:
| I plugged that first question into another of these kinds of
| products (called Elicit), and I think it did a much better job:
|
| https://elicit.org/search?q=what+is+the+relationship+between...
|
| It found relevant sources and had decent summaries of each.
|
| (edit: I just noticed it also has a summary of the whole
| question on the left too. Maybe someone who knows the econ side
| of it can attest to whether it's correct:
|
| > SUMMARY OF TOP 4 PAPERS
|
| > These papers suggest that lower commodity prices are
| associated with increased civil conflict. Demuynck 2008 found
| that a drop in prices of tropical agricultural commodities
| increases the attractiveness of other 'economic' activities
| such as rebellion/warfare and can, therefore, trigger civil
| conflict. Morgan 2014 found that commodity price volatility
| causes a commitment problem that should increase the
| probability of armed conflict. Ciccone 2018 found that
| international commodity price downturns sparked civil wars in
| Sub-Saharan Africa. Dube 2013 found that a sharp fall in coffee
| prices during the 1990s lowered wages and increased violence
| differentially in municipalities cultivating more coffee.
|
| )
| trompetenaccoun wrote:
| Elicit uses only scientific papers as sources iirc. It's an
| amazing tool and somehow not as well known as it should be.
| rushingcreek wrote:
| Thanks for bringing this up. We'll take a look at better
| sourcing for questions related to scientific research.
| whatshisface wrote:
| That sounds like a glowing review to me, you have to be an
| expert in the field and ask graduate-level questions to be able
| to find fault with the fourth sentence after three good ones.
| :)
| Nowado wrote:
| That's worse than 500 error.
|
| Not only is the provided answer is wrong, but the confidence
| remains high.
| [deleted]
| kridsdale1 wrote:
| The Internet is overflowing with high confidence low
| accuracy information produced by biological humans.
|
| What could set this apart from them would be an indication
| to the user of its confidence value, which Watson did for
| his debut on Jeopardy.
| palata wrote:
| Except that I don't think it can have any notion of
| confidence. It would need to understand the world, for
| that.
|
| This tool just makes it orders of magnitude easier to
| produce noise. You think the web is overflowing with low
| accuracy information? This may send us in a whole new
| level!
|
| I actually wonder if this has the potential to break
| search engines as we know them, because they can't judge
| the quality of the content.
| dr_dshiv wrote:
| Seriously. Like, I know we have high expectations for AGI but
| it was just born so...
| Imnimo wrote:
| So far not very impressed.
|
| https://beta.sayhello.so/search?q=how+can+i+use+a+list+of+st...
|
| None of these results are really even on topic, let alone
| helpful.
| kristopolous wrote:
| This is just a branding and promotion exercise for them.
| They're riding coattails.
|
| Might work
| nonameiguess wrote:
| I like the idea. I think separating out creative writing from
| fact retrieval makes a lot of sense and the attempt at a single
| general AI that can do both is clearly not there yet.
|
| It's slow as hell, though. Maybe a more limited invite-only beta
| would have been a good idea if your infra isn't there yet,
| compared to an announcement on Hacker News. It spun for 5 minutes
| doing nothing, then eventually gave me an inference engine down
| message. Retried and it spun for another 5 minutes doing nothing.
| Tried a third time, and at some point while I was back here
| reading the comments again, it loaded an answer finally. Tried a
| fourth time and got the inference engine down again.
| rushingcreek wrote:
| It's much faster now :)
| dzink wrote:
| If you guys want to launch a consumer business, you need a domain
| that can be said across the dinner and remembered and typed
| correctly by the person on the other side. Any other domain means
| you depend on your future users finding you through Google, your
| main competitor. Find a better domain if you want to grow.
| rushingcreek wrote:
| You're completely right. We have a much better one that we'll
| be switching to soon.
| spullara wrote:
| Sounds very similar to https://openai.com/blog/webgpt/
| mooneater wrote:
| I interviewed an author of WebGPT here, OpenAI co-founder John
| Schulman if you're interested, and he talks about it
| https://www.talkrl.com/episodes/john-schulman
| williamstein wrote:
| Some minor feedback: It's a little confusing that right below the
| search it asks "Was this answer better than Google?" when the
| answers you get to compare with on the right are from Bing. That
| question could be changed to "Was this answer better than the
| Bing results to the right?" Also, shouldn't the Bing results to
| the right be labeled as Bing results? I only know they are from
| Bing because it says "We also show regular Bing results side-by-
| side with our AI answer." in the post.
| rushingcreek wrote:
| Interesting. We'll take a look at this. Thanks!
| aarkay wrote:
| The inference seems to be taking very long and it gets stuck at
| times forever (> a few minutes). My guess is that this is
| happening due to the load? How fast is this when it's not under
| load? Is it as fast as ChatGPT?
| aarkay wrote:
| Great! All the best scaling :) I'll give it a shot again in a
| bit.
| rushingcreek wrote:
| Thanks! It's much faster now :)
| rushingcreek wrote:
| It is faster than ChatGPT normally. We're frantically scaling
| our infra right now.
| wayy wrote:
| The p50 time is around 2s end to end not under load. Bear with
| us as we scale our systems!
| tobr wrote:
| This might help: https://beta.sayhello.so/search?q=the+p50+ti
| me+is+around+2s+...
|
| It has the code you need: INPUT X IF X
| > 10 THEN PRINT "X is greater than 10." ELSE
| IF X < 10 THEN PRINT "X is less than 10."
| ELSE PRINT "X is equal to 10." END IF
| END IF
| vorpalhex wrote:
| how many ducks can fit into a cargo van
|
| > Answer
|
| > It is impossible to answer this question as there is no way to
| determine how many ducks can fit into a cargo van. The dimensions
| of the back doors on U-Haul cargo vans are 5'1-1/2'' x 4'1-1/2''
| , and you have 9'6'' x 5'7'' x 4'7'' in the cargo area totally
| 245 cubic feet of space for your household goods. This means that
| it would be difficult to fit more than 6,000 ducks at a time,
| while large farms usually have around 50,000 and 100,000 ducks
| reared simultaneously. The middle ground is 10,000 to 50,000 per
| batch.
| dinkleberg wrote:
| It looks like you've stumbled upon the next big "impossible
| question" that is going to require future technology to help us
| solve.
|
| It is interesting to see how it attempted to "solve it" though
| (I put that in quotes since it isn't actually doing the math)
| wayy wrote:
| If you can't find the information yourself via a web search,
| it's going to be difficult for the model to find the answer
| too... for now ;)
| rahimnathwani wrote:
| This looks similar to https://www.perplexity.ai/
|
| I tried Perplexity this morning with the following prompt:
|
| "What is the evidence for and against systematic phonics
| instruction?"
|
| It not only gave me the stuff 'everyone knows' but also pointed
| to a 2020 paper that goes against the conventional wisdom. (I had
| read that paper a few days ago, and was curious whether the AI
| would mention it.).
| mike_hearn wrote:
| A few thoughts as per request:
|
| 1. It's really cool that you guys are making your own models. It
| was looking like OpenAI might end up with a Google-style
| dominance of these things but apparently it's still possible to
| train one and get good results without them!
|
| 2. It's a bit unclear how programming specific it is. The sample
| queries are very programming dominated. Every results page seems
| to include a code snippets section even for non-programming
| queries like your Elon Musk query which presents a couple of
| "code snippets" that's just meaningless garbage. Presumably
| because it's not a coding question.
|
| 3. I tried a real question I had earlier today whilst
| programming, and which I'd also tried on ChatGPT. The first
| paragraph is the correct answer!
|
| 4. It seems to have trouble disambiguating different languages.
| My question was how to do something with the AWS S3 Java API, but
| the cited links were of a JavaScript repository showing how to
| solve the task. It also tried to generate a code snippet which
| was just totally garbled, it was obviously meant to be JavaScript
| but it wasn't even wrong, just non-syntactical nonsense.
|
| 5. In the same vein it showed me a code snippet that claimed to
| be Java but which was clearly C# (or meant to be C#).
|
| It did manage to cite the AWS SDK docs but only right at the
| bottom. That was the most useful results. For comparison when I
| asked ChatGPT a very similar question earlier, it hallucinated a
| completely convincing answer which relied on an SDK method that
| didn't actually exist. Sadly not so useful. I think I marginally
| preferred the Hello results.
|
| 6. I gave it a really hard test by asking who I am. It's tough
| because there are several guys with my name who have a web
| presence. It proceeded to mash together details of all our lives
| into a single biography. The citation feature was useful here
| because it made it clear that several different people were being
| conflated, whereas the answer alone might have sounded
| convincing. On the other hand, just doing a regular web search
| and picking a page about the right person would have been faster
| and clearer, without risk of being misled.
|
| The big questions in my mind are:
|
| 1. Business model? As someone with a developer tool to sell I've
| been kind of frustrated at how useless AdWords is for
| communicating the existence of tools to developers. They could
| surely use some competition there.
|
| 2. Truthfulness still seems like a fundamental challenge. The
| citation feature makes it really much more obvious but is that
| really a win? I finished by asking it a straightforward question
| about the population of my local city that Google answers
| instantly in the auto-complete box, I don't even have to press
| enter. Hello gave an answer that's roughly right but the snippet
| of the citation made it clear that the actual answer was a
| different number. The number the AI generated doesn't seem to
| appear anywhere in the cited page.
|
| Overall, really exciting to see a LLM that can both cite its
| sources and is exposed as a product! But right now I'm not sure
| it's faster to use this than regular web search :(
| wayy wrote:
| Thanks for trying it out!
|
| We can definitely be clearer on our focus for programming
| related queries. We usually don't display code snippets for
| non-programming questions but definitely still tuning a couple
| things there.
|
| We're not focused on simple factoid answers like population of
| cities because that's not where people get the most value.
|
| AWS API is a bit tricky because it is a rather broad technology
| with SDKs in different languages and thus the search results
| for a question will return a mishmash of solutions which we
| then try to make sense of. If you share some sample queries you
| tried related to this, I'd be happy to look into them and
| improve our answers there.
|
| Business model: We're currently just focused on building
| something developers want. Agree that ads and dev tools aren't
| the most synergetic.
| lossolo wrote:
| 11B parameters, so I guess it's FLAN-T5[1]? combined with
| retrieval transformer (something like RETRO[2])?
|
| 1. https://arxiv.org/abs/2210.11416
|
| 2. https://arxiv.org/abs/2112.04426
| dvh wrote:
| It gives answer that vaguely look like answer should look like
| but all the important details are wrong.
|
| > A solar eclipse is a partial eclipse of the Sun caused by a
| solar coronal mass ejection . A solar coronal mass ejection
| occurs when Earth passes directly in front of the sun.
|
| > You will also need to calculate number of seconds it takes for
| earth to rotate around the sun which is calculated as Math.PI * 2
| / 60
| acqq wrote:
| Yes, the answer I got was nonsensical too:
|
| > what is the diameter of the sun in sun radii
|
| > ANSWER
|
| > The diameter of the sun in solar radii is 1,392,000 km. This
| is calculated by taking the mean radius of the sun, which is
| 432,450 miles , and multiplying it by the circumference of the
| sun, which is 2,715,396 miles .
|
| The correct answer is, of course: 2.
|
| BTW: Google is also confused, but not in the same way,
| returning:
|
| 2.0018 R(sun)
| rushingcreek wrote:
| Doing unit conversions or general math is a known weak spot.
| I believe that 1,392,000 km is the correct diameter of the
| sun -- it just doesn't know how to convert those units.
| acqq wrote:
| No, it's exactly
|
| 2 * 695700 = 1391400
|
| based on the most recent measurement from 2015:
|
| https://en.wikipedia.org/wiki/Solar_radius
|
| Any other value found on the web is based on some older
| estimate or some arbitrary approximation.
| shhsshs wrote:
| The radius of the sun is not _exactly_ 695,700 km. The
| radius of the sun is _approximately_ 695,700 km.
|
| The _nominal solar radius_ , which is and is intended to
| be an approximation, is defined to be _exactly_ 695,700
| km.
| dannywarner wrote:
| They did a Launch HN recently: Launch HN: Hello (YC S22) - A
| search engine for developers'[1]
|
| Nothing much has changed since. So they appear to be trying to
| cash in on the interest in ChatGPT.
|
| Interesting they didn't include that they are backed by Y
| Combinator in the recent S23 cohort. Is being backed by YC a
| negative for startups here now?
|
| [1] https://news.ycombinator.com/item?id=32003215
| rushingcreek wrote:
| The models have gotten significantly better since our Launch
| HN. It might as well be a different product now from a UX
| point of view.
| trompetenaccoun wrote:
| Well, that's what these language models do in a nutshell. If
| you think social media and blogs are bad in terms of
| misinformation, this is the same thing rehashed by a machine
| that can not actually think or judge for itself (which to be
| fair, is arguably almost on a level already with how some
| humans interact with information they consume online).
|
| I asked some controversial questions and the answers are
| _interesting_ to say the least. The source is often reddit,
| random sites that are probably the top search engine result or
| some yellow press outlet.
| [deleted]
| tobr wrote:
| > _how do i mix two colors (hexadecimal rgb string) in js_
|
| > _ANSWER_
|
| > _To mix two colors in JavaScript, you can use the color-mix
| property. This property takes three parameters: the colorspace,
| the color to be mixed, and the percentage of that color to mix.
| For example, to mix the colors #FFFFFF and #8FBC8, you would
| use the following code:_
|
| > _let newColor = document.createElement;_
|
| > _/ / Outputs: #8FBC8_
|
| This is all just wrong from beginning to end. It seems to be
| confusing JS and CSS, too.
| low_tech_punk wrote:
| Similar idea: https://ought.org/elicit
| kirillzubovsky wrote:
| That's a pretty neat use of AI for a subset of use cases. I like
| it. I just asked it "Why does JavaScript keep crashing", with no
| particular details, and it was smart enough to show me how
| console.log() works in order to try and debug this unknown error.
| It feels good, like asking a friend for help.
|
| I am curious, how do you decide on what kind of content goes in
| the links bar? It doesn't seem be just news, or just links. I am
| having hard time figuring out when to resort to the links, vs
| just using the AI response.
| rushingcreek wrote:
| The links are just regular Bing results.
| trynewideas wrote:
| So what sources does it cite?
| wayy wrote:
| If you mouseover the answer paragraph, you'll see the
| relevant url / search result. You can click to go to that
| source.
| trynewideas wrote:
| Those sources appear to come directly from the Bing
| search results. So it's just pulling information from
| whatever Bing frontpages for the query?
| wayy wrote:
| Exactly, we use some information from traditional search
| engines as input to our model.
___________________________________________________________________
(page generated 2022-12-08 23:00 UTC)