[HN Gopher] Launch HN: Roundtable (YC S23) - Using AI to Simulat...
___________________________________________________________________
Launch HN: Roundtable (YC S23) - Using AI to Simulate Surveys
Hi HN, we're Mayank and Matt of Roundtable
(https://roundtable.ai/). We use LLMs to produce cheap, yet
surprisingly useful, simulations of surveys. Specifically, we train
LLMs on standard, curated survey datasets. This approach allows us
to essentially build general-purpose models of human behavior and
opinion. We combine this with a nice UI that lets users easily
visualize and interpret the results. Surveys are incredibly
important for user and market research, but are expensive and take
months to design, run, and analyze. By simulating responses, our
users can get results in seconds and make decisions faster. See
https://roundtable.ai/showcase for a bunch of examples, and
https://www.loom.com/share/eb6fb27acebe48839dd561cf1546f131 for a
demo video. Our product lets you add questions (e.g. "how old are
you") and conditions (e.g. "is a Hacker News user") and then see
how these affect the survey results. For example, the survey "Are
you interested in buying an e-bike?" shows 'yes' 28% [1]. But if
you narrow it down to people who own a Tesla, 'yes' jumps to 52%
[2]. Another example: if you survey "where did you learn to code",
the question "how old are you?" makes a dramatic difference--for
"45 or older" the answer is 55% "books" [3], but for "younger than
45" it's 76% "online" [4]. One more: 5% of people answer "legroom"
to the question "Which of the following factors is most important
for choosing which airline to fly?" [5], and this jumps to 20% when
you condition on people over six feet tall [6]. You wouldn't think
(well, we didn't think) that such simulated surveys would work very
well, but empirically they work a lot better than expected--we have
run many surveys in the wild to validate Roundtable's results (e.g.
comparing age demographics to U.S. Census data). We're still trying
to figure out why. We believe that LLMs that are pre-trained on the
public Internet have internalized a lot of information/correlations
about communities (e.g. Tesla drivers, Hacker News, etc.) and can
reasonably approximate their behavior. In any case, researchers are
seeing the same things that we are. A nice paper by a BYU group [7]
discusses extracting sub-population information from GPT/LLMs. A
related paper from Microsoft [8] shows how GPT can simulate
different human behaviors. It's an active research topic, and we
hope we can get a sense of the theoretical basis relatively soon.
Because these models are primarily trained on Internet data, they
start out skewed towards the demographics of heavy Internet users
(e.g., high-income, male). We addressed this by fine-tuning GPT on
the GSS (General Social Survey [9] - the gold standard of
demographic surveys in the US) so our models emulate a more
representative U.S. population. We've built a transparency feature
that shows how similar your survey question is to the training data
and thus gives a confidence metric of our accuracy. If you click
'Investigate Results', we report the most similar (in terms of
cosine distance between LLM embeddings) GSS questions as a way of
estimating how much extrapolation / interpolation is going on. This
doesn't quite address the accuracy of the subpopulations /
conditioning questions (we are working on this), but we thought we
are at a sufficiently advanced point to share what we've built with
you all. We're graduating PhD students from Princeton University
in cognitive science and AI. We ran a ton of surveys and behavioral
experiments and were often frustrated with the pipeline. We were
looking to leave academia, and saw an opportunity in making the
survey pipeline better. User and market research is a big market,
and many of the tools and methods the industry uses are clunky and
slow. Mayank's PhD work used large datasets and ML for developing
interpretable scientific theories, and Matt's developed complex
experimental software to study coordinated group decision-making.
We see Roundtable as operating at the intersection of our
interests. We charge per survey. We are targeting small and mid-
market businesses who have market research teams, and ask for a
minimum subscription amount. Pricing is at the bottom of our home
page. We are still in the early stages of building this product,
and we'd love for you all to play around with the demo and provide
us feedback. Let us know whatever you see - this is our first major
endeavor into the private sector from academia, and we're eager to
hear whatever you have to say! [1]:
https://roundtable.ai/sandbox/e02e92a9ad20fdd517182788f4ae7e...
[2]:
https://roundtable.ai/sandbox/6b4bf8740ad1945b08c0bf584c84c1...
[3] https://roundtable.ai/sandbox/d701556248385d05ce5d26ce7fc776...
[4] https://roundtable.ai/sandbox/8bd80babad042cf60d500ca28c40f7...
[5] https://roundtable.ai/sandbox/0450d499048c089894c34fba514db4...
[6] https://roundtable.ai/sandbox/eeafc6de644632af303896ec19feb6...
[7] https://arxiv.org/abs/2209.06899 [8]
https://openreview.net/pdf?id=eYlLlvzngu [9]
https://www.norc.org/research/projects/gss.html
Author : timshell
Score : 62 points
Date : 2023-07-25 17:07 UTC (5 hours ago)
| GartzenDeHaes wrote:
| Would this be a competitor for sites like Prolific, or a
| complimentary service? Running surveys with "real" humans on
| Prolific doesn't seem that expensive.
| egonschiele wrote:
| Some people worry that biased AI models will deepen inequality.
| Your product seems particularly primed for this scenario. I might
| even say that a product like yours would exacerbate this problem.
| What is your plan to ameliorate AI bias?
|
| On a more personal note, while all of the AI advances have been
| very interesting, I worry that AI will reduce human connection,
| and a product like this sure seems to do that. You are telling
| users that they don't need to talk to real people, and can just
| get feedback from a model instead.
|
| Edit: for example, here's your dataset by race:
| https://imgur.com/a/134epoN
|
| I asked, "Which race is most likely to commit a crime?":
| https://imgur.com/a/4QJZo2O
| timshell wrote:
| 1. GPT out of the box was pretty biased (e.g. gender
| distribution). We fine-tuned on representative survey data to
| ameliorate this bias so we get Census-level estimates for
| conditions such as gender [a] and work status [b].
|
| 2. We add the transparency features (click on 'Investigate
| Results') that shows how in vs. out-of-distribution the target
| question is. For out-of-distribution, we suggest people run
| traditional surveys.
|
| More broadly, I think your point is really interesting when it
| comes to qualitative data. That is one reason we haven't
| generated qualitative survey data, but a lot of potential
| customers have already started to ask for it.
|
| ----
|
| [a]
| https://roundtable.ai/sandbox/baa3d5f25236b91f1608c9f606b315...
|
| [b]
| https://roundtable.ai/sandbox/7a9ee27872eb29087be2386ccd19f7...
| timshell wrote:
| To respond to Edits - that's a great example, thank you. One
| of the limitations of surveys more broadly is you're asking
| for people's opinions, which of course does not correspond to
| reality. So, what we're simulating is how we estimate a
| representative U.S. population to answer the question "Which
| race is most likely to commit a crime?" as opposed to what
| the actual answer is.
|
| We definitely need to think how to handle your question so
| that it's clear where survey data converges/diverges with
| reality.
| cdblades wrote:
| How can you be reasonably sure that that work sufficiently
| addresses the bias?
|
| What metric(s) are you using to measure bias in general, and
| what do those metric(s) look like before and after your
| tuning?
| timshell wrote:
| Link [6] should point to
| https://roundtable.ai/sandbox/eeafc6de644632af303896ec19feb6...
| dang wrote:
| Fixed. Thanks!
| niko001 wrote:
| Really interesting approach! I can see this being useful. How are
| you dealing with short/medium-term changes in consumer sentiment,
| I assume your model is currently fairly static? For example, the
| results to "Would you buy an e-bike?" might change over time as
| cities add charge-points or additional bike paths, prices for
| e-bikes go down, etc. And as a more extreme example, the answer
| to typical YouGov questions like "Who will you vote for in the
| next presidential election" will obviously change daily based on
| a multitude of factors that aren't present in your training data.
| timshell wrote:
| The data we trained on has year, so we can specify the year you
| ask the question (the default is 2023). You can also see how
| answers change over time. [1] shows how the distribution for
| "Do you support the President" changes from 2000 to 2023 (see
| the 9/11 spike, end of Bush era, Obama era, Trump era, etc.)
|
| [1]
| https://roundtable.ai/sandbox/2dd4e9d32c24e9abff01810695e948...
| __loam wrote:
| What's the point of this lol
| SemioticStandrd wrote:
| I see the logic here, but I'm highly skeptical about how valid
| such a tool would be.
|
| If a researcher comes out and says, "Surveys show that people
| want X, and they do not like Y," and then others ask the
| researcher if they surveyed people, the answer would be "no."
|
| Fundamentally, people wanting feedback from humans will not get
| that by using your product.
|
| The best you can say is this: "Our product is guessing people
| will say X."
| digitcatphd wrote:
| I have been using AI generated surveys using the playground and
| have found them quite effective in simulating responses. In
| fact they are incredibly similar to my experience asking the
| same questions IRL. The challenge is people don't trust them
| and AI still have this negative association. So yes I mean to
| say it's yet another human error.
| timshell wrote:
| We're trying to figure out the optimal use case for this, i.e.
| whether it's internal or client-facing (your example).
|
| Internal purposes include stuff like optimally rewording
| questions and getting priors.
|
| A hybrid approach would be something like - hey let's not ask
| someone 100 questions because we can accurately predict 80%.
| Let's just ask them the hard-to-estimate 20 questions
| tcgv wrote:
| I think it's less about "prediction" and more about mapped
| cohort behaviors and opinions, especially those that change
| slowly over time. The LLM model will likely be a picture of
| how the population and each demographic group behaved and
| what they believed at a specific time window (i.e. when the
| data set was collected), and will produce answers that
| reflect that. It will most likely be lagging behind new
| trends and how they shape population behaviors and beliefs
| over time. In any case I think even the most experienced
| market research professionals would agree that discovering
| new trends before they become mainstream is really
| challenging.
| quadrature wrote:
| > optimally rewording questions
|
| This kind of concerns me because you could use this to bias
| surveys in different directions. This obviously already
| happens, so maybe it just part of the status quo.
| og_kalu wrote:
| Large Language Models as Simulated Economic Agents: What Can We
| Learn from Homo Silicus? (https://arxiv.org/abs/2301.07543)
|
| Out of One, Many: Using Language Models to Simulate Human
| Samples (https://arxiv.org/abs/2209.06899)
|
| There's been some research in this vain. To answer your
| question, seemingly very valid.
| DavidFerris wrote:
| Interesting idea! One of the problems with any primary research
| (surveys included) is the delay in collecting responses, which
| can take hours to weeks depending on sample, IR, incentives, etc.
| This would solve that!
|
| It's not surprising that LLMs can predict the answers to survey
| questions, but really good primary research generates surprising
| insights that are outside of existing distributions. Have you
| found that businesses trust your results? I have found that most
| businesses don't trust survey research much at all, and this
| seems like it might be even less reliable.
|
| -----
|
| Context: I co-founded & sold survey software company (YC W20).
| og_kalu wrote:
| You might want to take a look at the papers i've linked here
| that go into this kind of research
|
| https://news.ycombinator.com/item?id=36868552
| timshell wrote:
| Thank you!
|
| Trust is one of the biggest issues we're trying to solve. This
| motivated the tSNE plots and similarity scores under
| 'Investigate Results', but we definitely have a long way to go.
| Generally speaking, survey practitioners trust us more than
| their clients (perhaps not surprising)
| cowllin wrote:
| 10/10, no notes:
| https://roundtable.ai/sandbox/884fd5db560a3dce2cb2ed1c15596c...
| trolan wrote:
| I would honestly believe this would be the same results if a
| moderately popular user posted this poll with those options to
| their social media.
| apsurd wrote:
| What's your take on the entire premise of market research being
| mostly feel-good busywork detached from reality? This is because
| context is dynamic over time in every instance, and survey data
| pales in comparison to purchase data. Best way then is to launch
| small experiments with real people and real buying behavior.
|
| Covered here: https://www.pretotyping.org/
| timshell wrote:
| The survey / behavior gap is very real. Short-term we're
| focused on surveys, but we'd like to integrate behavioral data
| long-term (and potentially be primarily behavioral data, but
| that is TBD)
| Hormold wrote:
| But will the quality change if you use just random numbers? I
| think no.
| hcks wrote:
| I'm sorry but this doesn't make any sense. You're just
| hallucinating plausible sounding numbers.
|
| You probably fooled yourself with cherry picking
| og_kalu wrote:
| It makes perfect sense and there's research to back it up
|
| Large Language Models as Simulated Economic Agents: What Can We
| Learn from Homo Silicus? (https://arxiv.org/abs/2301.07543)
|
| Out of One, Many: Using Language Models to Simulate Human
| Samples (https://arxiv.org/abs/2209.06899)
| [deleted]
| tempusalaria wrote:
| Hi congrats.
|
| LLMs model a static distribution, whereas consumer preferences
| change over time to the point that companies regularly run the
| same survey at different points in time. At my old fund we would
| run the same surveys every month to track changes on various
| companies. How do you counteract this time effect? Presumably a
| lot of your training data is from the past.
|
| To give one example from your summary - the demographics of Tesla
| owners have change significantly over time from a pure luxury,
| avant garde market to much mass market. So info about Tesla from
| 5 years ago is not that useful
| timshell wrote:
| Pasting below answer to niko001
|
| The data we trained on has year, so we can specify the year you
| ask the question (the default is 2023). You can also see how
| answers change over time. [1] shows how the distribution for
| "Do you support the President" changes from 2000 to 2023 (see
| the 9/11 spike, end of Bush era, Obama era, Trump era, etc.)
| [1]
| https://roundtable.ai/sandbox/2dd4e9d32c24e9abff01810695e948...
| tempusalaria wrote:
| Which is logical and kind of what I expected. But raises the
| obvious question of where does your data come from going
| forward? The internet is getting more and more polluted with
| machine generated data, previous big ongoing data sources
| like Twitter, Reddit, etc. are all full of GPT spam and are
| trying to monetise their data.
|
| I'd also be interested in how much you think your platform is
| just capturing say reported surveys/data. President polling
| is something that must be all over LLM datasets- isn't that
| just replicating the training data?
|
| I think you could do a better job of showing on your website
| the following - here are some unusual survey results we
| generated from the model - I.e. stuff definitely not in the
| training data - and here's the data we actually got when we
| did that survey for real
| timshell wrote:
| Going forward, the current business model (with caveat that
| pivots are always likely this early stage) is to train on
| companies' proprietary survey data so we can estimate how
| their specific users respond to questions.
|
| In the backend, we check to see if the answers are stated
| in a high-quality survey and just retrieve that. I know we
| do this for gender, and I'm not sure whether that happens
| for presidential polling.
|
| Great idea, thank you. We're still figuring out whether the
| business model will be a general-purpose tool that anyone
| can use or those custom models I referenced above. If the
| former, your suggestion is spot on.
| another-dave wrote:
| > Going forward, the current business model (with caveat
| that pivots are always likely this early stage) is to
| train on companies' proprietary survey data so we can
| estimate how their specific users respond to questions.
|
| I imagine cleaning customer data to get it to the point
| that it's inputtable will be a big job for you.
|
| Are you then creating individual models per customer? As
| in, if Coke are an existing customer of yours and Pepsi
| sign up, do they get access to a model that's partially
| trained on Coke data, or it's a case of your base model +
| "bring your own research"?
| timshell wrote:
| > I imagine cleaning customer data to get it to the point
| that it's inputtable will be a big job for you.
|
| We're in the process of figuring that out. Hopefully that
| is another use case for LLMs :)
|
| > Are you then creating individual models per customer?
| As in, if Coke are an existing customer of yours and
| Pepsi sign up, do they get access to a model that's
| partially trained on Coke data, or it's a case of your
| base model + "bring your own research"?
|
| The latter, i.e. base model + "bring your own research"
| RandomLensman wrote:
| That sounds cool and I can believe that the totality of survey
| information etc. contains a lot of extractable information that
| isn't always made use of or needs surfacing.
|
| Do you know at what boundaries this tends to stop working, e.g.,
| some event happens that changes people's perception of X would
| probably need new deta if the event is "bigger than ..."?
| golergka wrote:
| Just played with the sandbox, and it seems like 16% of Apple
| users wouldn't consider buying Apple VR headset even for $3,5. I
| don't think even the lochness monster would be so stingy.
| timshell wrote:
| One of our major weaknesses right now is sensitivity to price
| sebzim4500 wrote:
| Presumably the question is being interpreted as "would you buy
| a Apple VR headset if the standard price was $3.50?", rather
| than "would you buy a headset for $3.50 that you could
| immediately resell for 1000x that?".
|
| The answer seems plausible with that interpretation.
| golergka wrote:
| Don't you think that even playing with it for an hour and
| then never touching it again would still be worth a three
| fiddy?
| sebzim4500 wrote:
| I do but I can imagine that 16% of people don't.
|
| If you went out with a VR headset and offered 30 minute
| demos of them to random people for $3.50 I don't think you
| would have an 84% success rate.
| coderintherye wrote:
| In a world where everyone could buy the headset for $3.50 (thus
| there is no profit value to buying it and then re-selling it)
| then that percentage actually makes sense.
| timfsu wrote:
| Congrats on the launch! The sandbox is a lot of fun.
|
| I played around with it, and one weakness I saw immediately was
| mixing real survey answers with imagined ones. For example:
| https://roundtable.ai/sandbox/609f54304935736c8e61816dea780e...
|
| I find it hard to believe that so many people would prefer carbon
| emissions over free beer, massages, or being the pilot. If I
| condition this data on being male or female, the results change
| dramatically too.
| goodoldneon wrote:
| "Simulate surveys" makes it sound like a satirical AI product
| callalex wrote:
| I was seriously searching through their webpage expecting a
| reference to the "conjoined triangles of success" to help
| indicate that this is all an elaborate self-aware prank. I was
| sorely disappointed to learn there is zero self-awareness here.
| og_kalu wrote:
| Self awareness for what lol ?
|
| If anything, existing research indicates you can just skip
| basic surveys and go for complex simulacra experiments.
|
| https://news.ycombinator.com/item?id=36868552
| msoad wrote:
| Can't believe this is coming from PhD students from Princeton
| University! It's so obviously flawed.
| dang wrote:
| " _Please don 't post shallow dismissals, especially of other
| people's work. A good critical comment teaches us
| something._"
|
| https://news.ycombinator.com/newsguidelines.html
| ljm wrote:
| A bayesian model will use previous data to calculate
| probabilities, so using AI to calculate a survey based on
| previous surveys sounds like a logical evolution no?
|
| That's something you might typically do by hand when running
| a survey and, say, comparing it to a benchmark.
|
| The problems will be similar to most AI problems I suppose:
| people who don't really understand the limitations of AI or
| the results it produces take the output of AI as gospel.
|
| My own thought is: what does it mean to 'simulate' a survey
| if the outcome is that people treat it as a 'live' or
| 'empirical' one?
| og_kalu wrote:
| Is it ?
|
| What little research we have of this kind of phenomena points
| towards this being very valid.
|
| Large Language Models as Simulated Economic Agents: What Can
| We Learn from Homo Silicus?
| (https://arxiv.org/abs/2301.07543)
|
| Out of One, Many: Using Language Models to Simulate Human
| Samples (https://arxiv.org/abs/2209.06899)
| samsee wrote:
| This is a really cool idea and beautiful UX, congrats on the
| launch!
|
| One related pain point I have seen many times with surveys is
| that the people writing them don't know what they're doing and
| get bad data as a result of biased questions.
|
| Could be cool to add functionality down the line to help people
| craft better questions. For example, your app could provide
| alternate ways of phrasing questions and then simulate how
| results would differ based on the wording.
|
| Excited to see where this goes! Going to share with my partner
| who works for a survey software company and see what she thinks.
| timshell wrote:
| Exactly where we're headed :)
|
| Thank you for the kind words / reference
| puzzydunlop wrote:
| Very cool idea and I think there's a thread worth pulling here.
|
| The CEO of Unlearn AI on this podcast
| (https://podcasts.apple.com/ca/podcast/whats-your-problem/id1...)
| talked about using AI to simulate a larger sample size for
| clinical trials which is similar to what you are doing here
|
| Looking forward to seeing where this goes :)
| puzzydunlop wrote:
| Another interesting point about this is that he talks about how
| it's mathematically probable that the clinical trial has the
| power of the larger sample size
| mjaques wrote:
| AI for simulating surveys is pretty bad (read, pointless), but
| AI for simulating clinical trials is straight up criminal.
| __loam wrote:
| > talked about using AI to simulate a larger sample size for
| clinical trials which is similar to what you are doing here
|
| My career has straddled medicine and ML at various points, so I
| feel like I have the context to comment on this: This is really
| fucking stupid. Like I can't believe anyone would suggest this
| dumb. Hopefully the FDA shuts this bullshit down before some
| moron MBA spreads the practice through the industry. Without
| real data you have nothing.
| Bonapara wrote:
| Congrats on the launch. Sounds like a smart idea!
| icyfox wrote:
| Congrats on the launch!
|
| I have an interesting dissonance with this. On one hand, I
| understand how huge parameter sets can and do model specific
| personas well. I've also read some of these cited papers and
| _know_ intellectually that predicted results can be close to
| actual survey data. The other part of me is screaming at my
| laptop that language modeling is about aggregate statistics,
| revealed preference counts for a lot, and how could a language
| model actually substitute for market research?
|
| I imagine the biggest hurtle you're going to face are research
| teams that:
|
| - A. Want to see actual proof behind data
|
| - B. Disbelieve a LLM could generate statistically significant
| insights about real people that would make individual decisions
|
| - C. Need to justify their own existence / organizational clout
| with boots on the ground facilitating surveys
|
| A and C might be surmountable, but I'm not sure of a good way of
| tackling B.
| og_kalu wrote:
| >The other part of me is screaming at my laptop that language
| modeling is about aggregate statistics
|
| Yeah..but it's not. This is where people are having so much
| trouble. The erroneous belief that language model learn some
| "average" or "aggregate" of the training set. But when you get
| down to it, that doesn't make any sense at all. What help would
| some fictional aggregate distribution do with predicting the
| responses of decidedly non aggregate text ? None at all.
|
| So Language models don't learn an "average" distribution. They
| learn to predict every state that exists in the corpus in a
| sort of superposition. The perfect LLM would predict Einstein
| as well as it would predict the dumbass down the street.
|
| LLMs are biased but not uniformly so.
| icyfox wrote:
| I think we're using aggregate statistics in two different
| ways.
|
| Technically speaking - the whole idea with a language model
| is that you're learning to generalize underlying patterns of
| text, not just memorize the input text. Otherwise language
| models would be very good at echoing back training data but
| fail miserably during validation. If we go back to the
| training sequence - it's trying to maximize the posterior
| given the conditional probabilities in the sequence:
|
| P(y1, y2, ... yn ) = P(y1) * P(y2|y1) ... P(yn|y1...yn-1)
|
| That probability is by definition an aggregate; it's the best
| fit given potentially competing inputs of the training set
| that all have the same input conditional chains.
|
| Where generative LLMs have a leg-up is because they have such
| a large parameter space, large context windows, and coherent
| sampling strategies. This helps them stay internally
| consistent with their response data. But at the end of the
| day what they are learning are still patterns. That's why
| they aren't able to link content back to the exact source of
| origin; parameters fuse inputs from different places into one
| hybrid.
|
| Seeding a generative chain with Einstein or someone down the
| street doesn't change the fact that what's next is some fused
| learning from a lot of different training set inputs.
| og_kalu wrote:
| The point i'm making is that i believe the "fused space" is
| not in fact very fused because that would be directly
| detrimental to reducing loss.
| timshell wrote:
| Thank you!
|
| Agree A, B, and C are big hurdles.
|
| Re: A - we have started adding transparency (vis-a-vis the
| 'Investigate Results' and the tSNE plots + similarity scores)
| but we still have a ways to go
|
| Re: B - agree that the survey responses -> insights pipeline is
| nonlinear and it's not clear how to make that tighter
|
| Re: C - generally, we try to champion a human-AI interaction
| loop where people are needed to evaluate the outputs, generate
| insights, etc.
|
| All great points though and ones we are facing
| selalipop wrote:
| I built notionsmith.ai originally with this specific usecase in
| mind, but found people struggle to trust AI derived insights
___________________________________________________________________
(page generated 2023-07-25 23:00 UTC)