[HN Gopher] Bloom's 2 Sigma Problem
___________________________________________________________________
Bloom's 2 Sigma Problem
Author : oidar
Score : 85 points
Date : 2023-02-25 19:05 UTC (3 hours ago)
(HTM) web link (en.wikipedia.org)
(TXT) w3m dump (en.wikipedia.org)
| ultra_nick wrote:
| Our startup started working on solving this problem with
| extremely customized continuous AI Tutoring last year. In his
| research paper, Bloom says that private tutoring is the most
| effective learning method, but similar results can be obtained
| through a combination of scientific learning methods. However, in
| his time, these methods were too computationally expensive for
| teachers to implement.
|
| Now, we have the technology (LLMs) for everyone to have an
| accelerated contextual learning experience. We started by
| tracking mastery learning with graphs, but the effort required
| was too much for users. Now, we're augmenting the previous system
| with ML to automatically track and explain new concepts. In the
| far future, I'm hoping we'll be able to accelerate learning rates
| by 150%, so that motivated highschool students can graduate with
| a bachelor's degree.
|
| Our first AI Tutor should be out by April.
|
| Follow us on LinkedIn to get an alert when we launch!
| https://www.linkedin.com/company/conceptionary/
|
| To learn more, checkout our website:
|
| https://conceptionary.app/
| a-dub wrote:
| i wonder how much all the redundant and slightly different
| resources on the internet today close that gap... this paper is
| from 1984.
|
| also would be curious how good today's llms are at getting
| someone to grok a topic they're having trouble grasping... (this
| can be actually a hard problem as it can require understanding
| the misunderstanding in order to guide away from it effectively.)
| ipnon wrote:
| If I ask ChatGPT how transformers work the results are
| sufficient but brief. It's difficult to probe ChatGPT, and it
| doesn't have an intuition for your weak points like a tutor
| does. If I ask ChatGPT about who influenced John Maynard Keynes
| however it will hallucinate a whole corpus of economists and
| papers that never existed but whose existence is proposed so
| confidently it is incredulous.
|
| Mainly we should appreciate the dearth of capabilities that
| arise merely as emergent properties of ever larger language
| models. Some kind of architecture will be needed for each use
| case, which means we once again find ourselves facing difficult
| engineering problems.
| [deleted]
| bowsamic wrote:
| Do we need to fix this? I am not sure that everyone needs a
| "mastery" of education
| medler wrote:
| Education is important in the modern world, especially now that
| the Industrial Revolution has led to an unprecedented growth in
| knowledge work.
| HDMI_Cable wrote:
| I wonder how aging populations and large-scale language models
| are going to change the difficulty of this problem. It seems to
| me that we're just about to unlock 1-to-1 learning, without
| actually needing 1 teacher per child.
|
| It isn't hard to imagine a language model (or something similar,
| I'm not in AI myself so I might not have the best idea of the
| specifics) giving personalized instruction to a student based on
| a knowledge bank and generative prompts + reinforcement from what
| the child understands.
|
| Add to that aging populations leading to less children and higher
| worker-child ratios, + automation in other industries, and its
| plausible to imagine that there could be a large movement towards
| training new teachers, thereby reducing class sizes. This could
| increase feedback between the two, and probably reduce
| behavioural problems as teachers could address them more easily.
|
| I guess one problem of this approach is that this 'atomizes' the
| student, where they only engage with their device with the
| language model, removing some of the collaborative aspects of
| modern instructional methods. One easy fix to this is to increase
| the amount of recess or structured play, thereby increasing
| socialization. This seems to work in Finland [1]. If advances in
| AI can make learning more efficient, we can increase the amount
| of time for unstructured play.
|
| ---
|
| [1]: https://www.theatlantic.com/education/archive/2014/06/how-
| fi...
| civilized wrote:
| It's a nice dream. I would just caution that we will unlock
| people _claiming_ that this works and foisting a half-baked
| version of it on large populations to save money, look modern,
| and make profits for tech companies... long before we unlock
| the actually good version.
| [deleted]
| swyx wrote:
| theres a lot of hope in the educational AI space that we can do
| personalized teaching and essentially raise everybody to the 90+
| percentile implied by the 2 sigma problem (see
| https://lspace.swyx.io/p/chatgpt-gpt4-hype-and-building-llm 20
| min mark - disclaimer its my podcast with openai's new
| representative)
|
| i think this is a "last mile" type situation where we've seen
| great progress with ChatGPT on the first 90%, but the last mile
| will take the remaining 90% to do. not impossible, just won't
| come anywhere as soon or as easily as people want. still worth
| pursuing ofc, but not as a quick hustle but more if you are just
| intensely insightful and passionate about education and figure
| out how to apply AI to personalize it properly
| bee_rider wrote:
| Since ChatGPT often writes untrue statements, it seems like
| quite a poor teacher. I guess it could be used as an assist for
| an instructor (send me a question, chat GPT will answer it,
| I'll check the output) but that seems not so revolutionary.
| ohhhhhh wrote:
| "too costly for most societies to bear on a large scale" this is
| changing as we speak with language models.
| c0brac0bra wrote:
| Makes me think of the book in Diamond Age.
| hgsgm wrote:
| What's that book called?
| pantalaimon wrote:
| _A Young Lady 's Illustrated Primer_
| nullc wrote:
| "A Young Lady's Illustrated Primer"
| l_theanine wrote:
| Education is going to be radically changed when AI can manage to
| helpfully teach students with a Microsoft Tay situation
| happening.
|
| Very quickly after that, I think a sort of global shift will
| begin, as schools all around the world begin to see the power of
| legitimate curriculums, instead of lobotomized "think how I
| think" bs. I'm thinking of Florida right now, but I know the same
| shit goes on in India, China, countries in Africa, and probably
| other places too.
|
| On a longer scale, as models become more and more advanced, maybe
| there will be teachers that are constantly absorbing new
| information and are able to keep students up-to-date with best
| practices, new research, etc. That would not bode well for the
| leagues of McKinsey-ites and that whole ilk.
|
| I am particularly hopeful that AI will be able to radically
| advance education.
|
| That is, if this hypothetical technology is democratized and kept
| open, rather than horded and sequestered to the ruling capital
| class until there exists a financial, educational, biological and
| altogether insurmountable moat between the ultra-wealthy and the
| rest of us.
| hinkley wrote:
| We need to study what amounts of 1:1 tutoring have positive
| outcomes. If society can't afford 1:1 tutoring for everyone, all
| the time, can we afford 20% of the time?
| hgsgm wrote:
| That's why schools should not waste time in lecture. Students
| should read/listen/watch and at home and bring questions to
| school
| 082349872349872 wrote:
| My understanding is that the knee of the curve is ~10-12
| students. Much cheaper than 1:1 or 1:2, but still more
| expensive than the 30+ classrooms of my youth.
| zozbot234 wrote:
| You can teach large classes efficiently, but it takes a very
| direct, rote, scripted approach that's exactly what most
| teachers _don 't_ want to use - because fancy education
| schools have told them that it "demeans" the profession, and
| they've drunk the Kool-Aid. A key component is to come up
| with memorable text that the students must be able to repeat
| _word for word_ when prompted - both as a group and
| individually. (Problem solving similarly starts with trivial
| cases that the students must answer on the spot; difficulty
| is very gradually increased, and there 's a lot of retracing
| and review of earlier steps.)
|
| This is as close as you can get to "mastery learning" in a
| large classroom; keep all students on lockstep and highly
| engaged, trying to ensure that no one falls through the
| cracks.
| trane_project wrote:
| I am already working on fixing this: https://github.com/trane-
| project/trane/
|
| No AI needed. Just an old fashioned depth first search through a
| graph of skills and dependencies.
|
| I made it to help me practice music, but I have been branching
| out and using it to study math for a few weeks. I find myself
| saying "just one question more" and then spending another half
| hour in it.
|
| Still needs more material to be useful to other people but it's a
| solid experience. I learned and memorized how to play most of the
| notes in the saxophone with good intonation in about a week, as a
| complete beginner.
| cjblack wrote:
| I really like simple, reliable, tools like this - just watched
| the demo video. I hadn't heard of mastery learning before, but
| now I'm thinking about trying to map some Excel lessons towards
| this and see if I can implement it at work.
|
| I've found that our youngest employees are coming out of school
| with little-to-no knowledge of how to work spreadsheet software
| - even those who are otherwise pretty technically proficient.
|
| While programmers tend to eschew tools like Excel, in
| consulting (or at least the consulting we do) it's critical to
| being able to understand how a process works before designing
| an automated solution. Excited to potentially have a way to
| share that understanding.
| trane_project wrote:
| Thanks for the feedback. An excel course should be doable.
| Currently the most general way to write material is to use
| this: https://trane-
| project.github.io/generated_courses/knowledge_... Basically a
| folder of lessons where the flashcards are pairs of markdown
| files.
|
| There's also another tool to write simple flashcards and
| lessons in a json file, run a command, and build all those
| directories and markdown files for you, but I have not
| written the documentation yet. Here's an example for a course
| based on a reworking of Euclid's Elements:
| https://github.com/trane-project/trane-
| math/blob/master/cour...
|
| But plain markdown files for now should get you going.
| wizzwizz4 wrote:
| Trane looks good. I've been trying to design a similar system,
| but for mathematics: the difference there is that there can be
| multiple routes towards a particular understanding, with
| different routes better for different people.
|
| Prerequisites for calculus: an understanding of the standard
| reals OR an understanding of the hyperreals. Yes, real and
| hyperreal analysis are radically different in many ways, but
| they're provably equivalent, and they get you to the same
| answers. Some people find one more intuitive than the other,
| and some problems are easier in one than the other.
|
| Imo, the biggest issue with mathematics education is that it
| tries to push you down one, standardised way of thinking about
| things: if you don't _get_ it, that cripples your ability to
| think and reason for yourself. The main thing I do in one-to-
| one mathematics tutoring is finding how the student thinks, and
| nurturing _that_ (instead of stamping it out and replacing it
| with a "more formal" approach).
|
| I'd like to see something branchy, but I haven't found a good
| solution. Do you have any ideas for tackling this?
| trane_project wrote:
| I started working on this a few weeks ago:
| https://github.com/trane-project/trane-math It currently
| needs a README, but you could take a look at the courses on
| how I am building the flashcards. It's easy to reference
| external resources, so that's what I have been doing, rather
| than trying to create exercises of my own.
|
| I am starting with a very basic olympiad-style book and a
| book based on Euclid's Elements, because I don't have the
| understanding required to clearly work out the dependencies
| of more advanced stuff. And I also would like to start at the
| beginning to make sure I don't miss anything.
|
| The ideal end state is to have courses that cover all the
| undergrad and grad math curriculum. I am also curious on
| whether this could be used by researchers to keep up to date
| with the latest research on their fields. But all of that is
| a long way out.
|
| As for your question, there are a couple of ways that Trane
| could handle multiple paths through similar material.
|
| 1. Just have separate curriculums. You could copy the
| courses, but the second copy has different dependencies,
| courses/lessons IDs. For example, one could have a series of
| courses teaching the undergrad MIT math curriculum and
| another the Harvard curriculum. They might share a lot of the
| material, but the order will be different.
|
| 2. Trane does not lock you into a specific order. There are
| filters that let you specify which parts of the graph you
| want to study. You are free to get questions from specific
| courses and lessons. You can also use the metadata in the
| courses to say things like "give me questions from all
| lessons teaching linear algebra" or "give me questions from
| all courses on real analysis but not from the lessons on set
| theory". The dependencies between the lessons that match that
| metadata are still respected. There are a few more options,
| but you get my point. The dependencies are not set in stone,
| and there's freedom to jump around and study specific topics.
|
| I actually use option 2 most days. If I want to practice
| guitar, I just set a filter to give me exercises from the
| guitar. Similar thing when I want to practice the saxophone.
| wizzwizz4 wrote:
| Mathematics doesn't really _have_ dependencies. Everything
| can be thought of in terms of everything else: you need an
| in, which is usually arithmetic or geometry (but, for
| physicists, is often physics - and for programmers, is
| often concepts related to their favourite programming
| languages), and then everything _can_ be defined in terms
| of something else.
|
| However, many people find it easier if you have concrete
| anchors, rather than making everything an abstract ivory
| tower. A certain kind of musician might get Fourier series
| quite quickly by analogy to the behaviour of sound, and
| then pick up Taylor series by analogy to that; somebody
| like me might find it easier to translate the most general
| Fourier series to a Maclaurin series, then compare
| coefficients. All these approaches would give you an
| understanding of Fourier, Maclaurin and Taylor series',
| which then lets you learn things that build upon them.
|
| The "separate courses" approach is doable (and the best
| I've come up with so far), but it doesn't scale: courses
| that cover the same things without providing further
| insight don't get merged together. It can't handle the
| combinations that, intuitively, I feel like there should be
| a way to handle. Human teachers can cope with it just fine!
|
| One nitpick on trane-math: running `cargo test` will wipe
| out the .trane directory, if it exists. That seems not very
| good; perhaps there should be a way to point trane at a
| temporary directory, so you can test it better?
| airstrike wrote:
| This looks great, thanks for sharing. I do think it would be
| more ergonomic to have this served via a website given that it
| feels CRUD-like with some browsing of exercises, relationships
| between them, etc.
|
| The CLI approach assumes people are comfortable using the
| terminal, which unfortunately limits the number of potential
| users
| trane_project wrote:
| Definitely. I have not worked in frontend/ux stuff for over
| 10 years, and I am working on this by myself, so I have been
| focusing on the CLI to quickly iterate and prove that it
| works reasonably well. Part of the reason of trying it out on
| math was to a find a more technical group of users.
|
| The ideal state is a graphic interface where the exercises
| are presented on the screen to reduce the friction. Gotta
| start somewhere.
| jackthetab wrote:
| I'm more than happy to do a CLI app, but I think this needs a
| way to run it on a phone (noticed I don't necessarily mean
| "app" ;-). I can see me doing this on my phone while waiting
| for the barista to make my coffee.
|
| I'm going to do something similar with Anki. I'll try to
| compare the two...
| trane_project wrote:
| Main difference with anki is that anki explicitly
| discourages trying to make dependencies between flashcards
| or decks. I tried to work out something like that into
| Anki, but as I saw it, dependencies were core to what I had
| in mind. There's some similar functionality in supermemo,
| but it's based on a tree, not a graph, so it was inherently
| limited.
|
| No experience in mobile at all so that'll have to wait, but
| a GUI/website/app that works on most platforms is something
| I would like to work out eventually.
| admsmz wrote:
| This sounds very similar to the system behind
| http://mathacademy.com . You follow a graph of skills and
| dependencies and because you always have the necessary
| prerequisites it is always doable.
|
| Their team of mathematicians have created a curriculum from
| k-12 to a bachelors in mathematics.
| trane_project wrote:
| Yeah, this is cool and similar to the end state I have in
| mind. The main difference is that I am not trying to create
| the curriculum/exercises. I am just creating flaschards that
| say "Solve exercise x.y.z in that textbook". Plus the whole
| free-software angle and being able to share the material as
| text files.
|
| The math thing is just a side project at the moment to try to
| see how it works for other fields. I am primarily using this
| for music. I was hoping something like this existed already,
| but all the solutions are either very specialized like this
| one or do not support dependencies as a core feature, like
| Anki. Had Anki supported something like this, I wouldn't have
| needed to make my own thing.
| zozbot234 wrote:
| It would be nice to simply include support for dependency
| tracking in existing flashcard software. Unfortunately, the
| most commonly used flashcard software, viz. Anki, has not
| been adding such features. I can only assume that the
| project is very lightly maintained and/or working on paying
| down technical debt - new innovative features don't seem to
| be a priority.
| trane_project wrote:
| I think ultimately it's a very different approach to
| their current one, so it might be too difficult to do in
| the existing codebase. Maybe not, I have not looked at
| the code. But the public docs and learning philosophy
| around anki all seemed to discourage it, so I decided to
| make my own thing.
|
| And yeah, I don't think Anki gets a lot of support. I am
| an open-source maintainer myself, so I know that story.
| soferio wrote:
| This looks great. My son uses "beast academy" from Art of
| Problem Solving (https://beastacademy.com/), which is
| fantastic. But mathacademy looks like it might be a good
| competitor at the post year 6 level.
| precompute wrote:
| Similar to the "2 Sigma humor gap".
|
| Also related: sharply increasing odds of social maladjustment /
| mistreatment after a certain IQ score.
| hgsgm wrote:
| It's completely different. The only similarity is that a
| standard deviation is involved.
| lofatdairy wrote:
| Also discussed in the aristocratic tutoring article posted here:
|
| https://erikhoel.substack.com/p/why-we-stopped-making-einste...
|
| and discussed here:
|
| https://news.ycombinator.com/item?id=30698624
| levzettelin wrote:
| What's problematic about this? Why is this named a "problem"?
| swyx wrote:
| it is a core driver of compounding inequity, assuming the rich
| can afford personal tuition and the poor cannot, which then
| gives rich kids access to better opportunities to continue
| being rich. it gives the lie to "equality of opportunity not
| equality of outcome" because any game of consequence involves
| outcome being exclusive opportunity.
| choxi wrote:
| I believe it's that Bloom's study implies that 1-on-1 pacing
| and teaching produces the best learning results, but that Bloom
| thought it was infeasible to scale to everyone: it's "too
| costly for most societies to bear on a large scale"
| trane_project wrote:
| That personal tutoring leads to better outcomes is not the
| problem. That will obviously be the case.
|
| The problem is how to bridge the gap in a way more people
| realize similar benefits.
| elcomet wrote:
| The problem is that we are not giving the best education we can
| to everyone, as it is too costly to have one-to-one tutoring
| for all. So is there a method that works as well as one-to-one
| tutoring but affordable by our society?
| Taikonerd wrote:
| Has anyone used any of the "intelligent tutoring systems"
| mentioned in this Wikipedia article?[0]
|
| Lots of comments here are saying "we'll have an intelligent
| tutoring system one day, based on LLMs" -- but there are many
| such systems that already claim to be getting good results
| now.[1] I'd be interested to hear from anyone who's used one.
|
| [0] https://en.wikipedia.org/wiki/Intelligent_tutoring_system
|
| [1] https://www.lesswrong.com/posts/vbWBJGWyWyKyoxLBe/darpa-
| digi...
| admsmz wrote:
| I am a customer of mathacademy.com, an intelligent tutoring
| system with a focus on math. Right now they are putting the
| finishing touches on their courses like abstract algebra and
| discrete mathematics. They already have linear algebra and
| calculus.
|
| It absolutely works as advertised and I highly recommend it.
| SunghoYahng wrote:
| I took a look at the site you mentioned, but I don't see how
| it's useful for learning. It uses AI to determine what the
| learner's current level of knowledge is, but the way the
| learner learns seems completely typical. Then I may as well
| read classic textbooks. It doesn't seem to me to be that
| meaningful to determine the learner's current level of
| knowledge.
| admsmz wrote:
| Can you explain your objections? I don't really understand
| this comment.
| trane_project wrote:
| Going through the list, it seems like all of them only work on
| very specialized subjects and/or are not published. I have read
| about the DARPA tutor, but I don't think it's been made
| publically available. So I doubt anyone here has used it,
| unless they were in that specific branch of the military at the
| time it was used.
|
| I don't see how LLMs deliver on this. I think they could speed
| up how the curriculum is generated, and I have played around
| with ChatGPT to do just that, but the questions only need to be
| generated once. It's the guiding of students through the
| exercises that needs to be personalized, not the exercises
| themselves.
|
| As for their effectiveness, I am the author of Trane (see my
| other comment). It's worked pretty well for me, but I am
| splitting my time between my job, working on the software,
| creating materials, and actual practice, so I can't really say
| what the upper limits are until I can focus on just the last
| part.
___________________________________________________________________
(page generated 2023-02-25 23:00 UTC)