[HN Gopher] Tinygrad
___________________________________________________________________
Tinygrad
Author : tosh
Score : 203 points
Date : 2022-04-04 17:48 UTC (5 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| tartakovsky wrote:
| For a novice in this space, can anyone provide a link to an
| article providing a comparison of: Pytorch, Pytorch lightning,
| tinygrad, micrograd.....? I would like to point my clients to a
| reference here. It's no longer enough to say "PyTorch" I guess.
| modeless wrote:
| I find this related page more interesting: A Breakdown of AI Chip
| Companies
| https://geohot.github.io/blog/jekyll/update/2021/06/13/a-bre...
|
| I especially like that he outlines an actual plan for an AI chip
| startup that he thinks will work, and has an update explaining
| why he was subsequently convinced that it wouldn't work.
| whimsicalism wrote:
| Reminds me of https://github.com/minitorch/minitorch
| mdp2021 wrote:
| Minitorch is intended to be both an engine and high quality
| didactic material, from Cornell University, for the course
| 'Machine Learning Engineering'.
|
| > _the full student code for minitorch. It is designed as a
| single repo that can be completed part by part following the
| guide book_
|
| > _Basic Neural Networks and Modules ; Autodifferentiation for
| Scalars ; Tensors, Views, and Strides ; Parallel Tensor
| Operations ; GPU / CUDA Programming in NUMBA ; Convolutions and
| Pooling ; Advanced NN Functions_
|
| See: http://minitorch.github.io/
| Ninjinka wrote:
| Upvoted for geohot
| hwers wrote:
| I like this guy mostly for preserving a hacker-ish culture in the
| atmosphere and for just being a real human being in public. Too
| often do I feel like I'm just interacting with interfaces to
| corporations or people who self censor themselves so much they
| barely feel human. I'm not sure this library matters much to me
| but I maintain positive perspective on this guy for these
| reasons.
| travisporter wrote:
| Its funny you say this because the repo mentions Andrej
| Karpathy's repo. I absolutely love his work especially his ML
| tutorials, but there was one tesla automation day where he
| shuts down this guy asking a question with robotic efficiency.
| I wish I could find it.
| thspimpolds wrote:
| Having gone to college and knowing this guy personally from
| then, you would have a much different opinion... Maybe he has
| grown up, but back then... yeeesh
| throwawaylien wrote:
| Jealousy tends to make you react that way
| hwers wrote:
| I have gone to college with similar overachievers/braggers,
| and now that you mention it if they were in the spotlight as
| much as him I'd roll my eyes pretty hard. Still though,
| having not gone to college with this particular one somehow I
| can stand it.
| sydthrowaway wrote:
| I'm guessing only child syndrome + rich as hell parents.
| CyberRabbi wrote:
| I cannot take this criticism seriously unless you are more
| specific than "yeesh." If you aren't willing to be specific
| then better to not say anything at all.
| [deleted]
| bko wrote:
| I mean, he's obviously not a normal guy. I don't know what
| you expect. From a recent profile:
|
| > Apart from a criminal streak, Hotz shares with Raskolnikov,
| Dostoyevsky's antihero, a predilection for instrumental
| reason and an urge to test his own mettle, to know himself by
| knowing his limits. As a young adult Hotz allowed himself to
| become addicted to prescription opiates almost as an
| experience in self-mastery. "I did it, I was addicted, and I
| quit," he told me. "I think I had to have that experience. I
| don't think I ever could have been the type who never tried
| it. Because in some ways I feel that if I'm not strong enough
| to defeat that and overcome it..." He paused for several
| beats before assuring me he'd never want anyone to follow his
| example. "In order to quit," he continued, "it required me to
| rethink what I wanted out of life. After that, one of the
| biggest things that changed is I stopped caring about money."
|
| https://return.life/2022/03/07/george-hotz-comma-ride-or-
| die...
| mdp2021 wrote:
| > _not a normal guy_
|
| I assure you some people try themselves - and I do not see
| what is not "normal" about it. To experience, voluntary,
| then grow, is the norm.
| bko wrote:
| I don't know. I ran the idea of purposely getting
| addicted to opioids to see if i can quit by my wife and
| she assured me i was crazy.
| mdp2021 wrote:
| This direction goes off-topic, but: look, if you married
| her,
|
| -- either you go along well, hence that she agrees with
| your opinion is a weak test;
|
| -- or you married her to test yourself, which would prove
| my point.
|
| TND; QED. /J
| Zacharias030 wrote:
| What does TND mean?
| canjobear wrote:
| Who cares if he's a nice guy?
| mhh__ wrote:
| Talented but bad people breed a new generation of
| unnecessary untalented bad people.
| bri3d wrote:
| If he's being held up as some shining beacon of the
| "hacker-ish culture" as the grandparent poster is doing, it
| would be nice if he weren't also a jerk.
| bitexploder wrote:
| Everyone should kill their heroes. They almost never live
| up to the pedestals people place them on. They are all
| just people and have flaws. They don't have to be some
| perfect person to do cool things we can respect.
| bri3d wrote:
| I respect geohotz' achievements (generally). I would
| never hold him up as an example of hacker culture, or his
| persona as one to emulate. There is a difference between
| these things.
| recuter wrote:
| Is hacker culture not filled with "weird jerks"? Linus
| doesn't qualify? Who does?
| aantix wrote:
| Why the need to conflate personality traits with skill?
|
| If I have a rare disease, I don't care if the doctor is
| nice. I want them to get the diagnosis correct.
|
| Van Gogh painted brilliantly. And he cut off his ear.
|
| Eminem is a great rapper. His themes can be violent.
| bri3d wrote:
| The post I replied to was one about personality traits,
| not skill.
| mdp2021 wrote:
| The poster probably meant that "<<it would be nice>> -
| yet largely irrelevant". You can abstract from those
| "personality traits" as long as """he is one who
| delivers""" - respectfully saying, as he is not obliged:
| he provided us with another tool, for our use, for free:
| he is a benefactor.
| hwers wrote:
| I don't think of him as shining. I like that he's a real
| person, flaws and all. Not someone to idolize. Many great
| hackers that I know and in general are total jerks (I
| suspect the trait helps wrangling a machine somehow but
| idk). I wouldn't find it accurate to my experiences if we
| had some polished perfect optics guy as representative
| (not that that's what he is)
| bko wrote:
| It's crazy that his company (comma ai) and actions stick out so
| much. They're in the auto-driving space and they have a real
| product that you can buy today and use it for a better lane-
| assist and hook it up to most newish cars.
|
| It's open source so you could run it on your own hardware for
| free.
|
| You have a complaint or question? Just check out their discord
| where you can talk to engineers working there instead of some
| outsourced chat bot.
|
| They just figured out the minimum you would have to do to
| control a car and are making incremental improvements with the
| insane business strategy of charging more than it takes for
| them to build it.
|
| Oh and if you're a business executive and want to partner, you
| can schedule a 30 minute phone call with comma's VP of Business
| Development for $1,000!
| leeoniya wrote:
| there is also https://github.com/photopea/UNN.js
| Buttons840 wrote:
| I'd like to ship a video game that does machine learning and
| trains on its experiences with the player. Yes, I know there are
| many potential problems with this.
|
| What is the best way to ship training code in a game? Do I embed
| Python and PyTorch or something? Do I code my own NN training
| algorithm? Do I use a library such as Tinygrad?
| d915 wrote:
| How about we show simple admiration and appreciation? We could
| learn way more if we try to understand the actions rather than
| criticize. He's a nerd. If we are here we are nerd too. Keep up
| the nerd sht geohot!
| yt-sdb wrote:
| Also: https://github.com/karpathy/micrograd
| londons_explore wrote:
| > tinygrad will always be below 1000 lines. If it isn't, we will
| revert commits until tinygrad becomes smaller.
|
| I applaud this. Committing to keeping a project small and simple.
|
| So many projects start small and simple, and before long they've
| been extended in many different directions and now have thousands
| of options and things to understand before you can get started.
| throwawaybner3 wrote:
| I like his honest streams. Also this geohot can code for 12 hours
| straight. That's just amazing. If you take in to account his
| speed of thought, its more like 36 hours.
| jonahx wrote:
| >tinygrad will always be below 1000 lines. If it isn't, we will
| revert commits until tinygrad becomes smaller.
|
| Love this ethos so much. Wish more projects would follow suit.
| babelfish wrote:
| Completely disagree - limiting lines of code as an ethos makes
| as much sense as using lines of code written as a business
| metric.
|
| What if they want to add a new feature that takes another ~1000
| LoC? Can they just write it as a separate library and include
| it as a dependency? IMO trying to minimize LoC creates a
| perverse incentive to split up your package when it might not
| need it (in the same way maximizing LoC creates a perverse
| incentive to write the most verbose code possible, not
| necessarily the most readable/maintainable)
| dleslie wrote:
| > What if they want to add a new feature that takes another
| ~1000 LoC?
|
| They don't.
|
| The LoC limit exists as much to avoid scope creep and
| maintain focus as it is about anything else.
| mhh__ wrote:
| This one arguably does have a bit of scope creep it's just
| in a separate directory.
| blueblob wrote:
| It trades off utility for managing scope creep. Keeping
| this code base tiny necessarily means that anything that
| uses it would need to write more code. If the point is a
| demonstration instead of a usable library, then that is
| good because it is much easier to follow without excess
| complexity.
| dleslie wrote:
| Necessarily is a strong word; there are assuredly many
| users of this tool who find it sufficient in its scope as
| it is, based on the project activity. If a user needs
| significant feature additions they can write them as a
| separate tool that interacts with this one.
| a-dub wrote:
| simplicity is a virtue. the project goal apparently isn't to
| have a lot of features, it's to do the core thing in the
| simplest way possible.
| lazide wrote:
| And if someone does some code golf to stay under the 1000
| line limit rather than go to 1050 lines that's an
| improvement?
| recursive wrote:
| Then the human responsible for approving the PR will
| decide whether it's worth it. Personally, I love it.
| a-dub wrote:
| no that's another bloated ml library with code
| compression.
|
| i suspect a part of the motivation for this library is
| learning and teaching.
|
| and maybe a little flexing on overcomplicated autograd
| libraries.
| alwaysjoe wrote:
| Maybe don't add that feature then? Not every library is meant
| to solve all problems
| dymk wrote:
| No, but every library tries to solve at least _one_
| problem, and maybe it needs to go over 1000 lines to do
| that.
| dleslie wrote:
| This one doesn't, apparently.
| canjobear wrote:
| It serves an important gatekeeping function by discouraging
| low-effort contributions.
| robbedpeter wrote:
| Not all software is intended to grow indefinitely. Keeping
| things small and grokkable is very valuable for learning.
| recuter wrote:
| You say 1000 lines is an arbitrary number. Do you actually
| have a feature in mind that takes it above this artificial
| limit? Did you even read the code?
|
| They like this round number. Fork. Why argue.
| striking wrote:
| It's how you get commits like these, though https://github.
| com/geohot/tinygrad/commit/050636bcb1068beff6...
| Subject: here's two extra lines of precious code (#307)
| diff --git a/tinygrad/__init__.py b/tinygrad/__init__.py
| index 0deab3e9..31ac75d5 100644 ---
| a/tinygrad/__init__.py +++ b/tinygrad/__init__.py
| @@ -1,3 +1 @@ -import tinygrad.optim -import
| tinygrad.tensor -import tinygrad.nn +from
| tinygrad import optim, tensor, nn
|
| There's a difference between "I will refuse features like
| X/Y/Z" and "I want the length of the code file to be N
| lines at most". The former tells you both explicitly and
| implicitly which features not to bother contributing. The
| latter is just nonsense.
| recuter wrote:
| This is why the good lord gave us:
| https://github.com/psf/black
|
| Import line is actually improved with this commit. LGTM,
| ship it.
|
| We now return to your regularly scheduled bikeshedding.
| striking wrote:
| You're saving yourself from one kind of golfing, but not
| all of them.
|
| Additionally, I feel like installing an additional tool
| is kind of against the spirit of 1kLOC simplicity.
| recuter wrote:
| I'm getting distinct uses Eclipse to write vogon poetry
| and Jira Master vibes from you.
|
| Any notes on the _actual code_ in the OP beyond the
| import lines formatting?
| striking wrote:
| I'm sorry to have offended you, but I don't feel personal
| insults over the README of an open source project are
| particularly reasonable.
|
| This affects the actual code, too! See, for example, http
| s://github.com/geohot/tinygrad/commit/cfb7a4c41a2b6bcc09.
| .., which includes this gem: diff --git
| a/tinygrad/ops/ops_cpu.py b/tinygrad/ops/ops_cpu.py
| index a454f56f..0686f810 100644 ---
| a/tinygrad/ops/ops_cpu.py +++
| b/tinygrad/ops/ops_cpu.py @@ -2,22 +2,15 @@
| from ..tensor import Function class
| CPUBuffer(np.ndarray): - def log(x): -
| return np.log(x) - def exp(x): - return
| np.exp(x) - def relu(x): - return
| np.maximum(x, 0) - def expand(x, shp): -
| return np.broadcast_to(x, shp) + log = lambda x:
| np.log(x) + exp = lambda x: np.exp(x) +
| relu = lambda x: np.maximum(x, 0) + expand =
| lambda x,shp: np.broadcast_to(x, shp) + permute =
| lambda x,order: x.transpose(order) + type = lambda
| x,tt: x.astype(tt) + custompad = lambda x,padding:
| np.pad(x, padding) def amax(x, *args, **kwargs):
| return np.amax(x, *args, **kwargs) - def
| permute(x, order): - return x.transpose(order)
| - def type(x, tt): - return x.astype(tt)
| - def custompad(x, padding): - return np.pad(x,
| padding) def toCPU(x): return x
| @staticmethod
|
| The " _actual code_ " is being golfed for no other reason
| than to keep the line count down, for the sake of a
| promise that never needed to be made. This commit
| recovered a total of seven lines wherein new code can be
| added, in exchange for readability. It is not just a
| quirk, not just some sort of interesting aside like one
| could argue for suckless' `dwm`: this promise actively
| makes the code worse over time, for no reason.
| recuter wrote:
| No offense taken and no offense intended at all friend.
|
| In my opinion we are down the rabbit hole of arguing
| style over substance. I just think you are grasping at
| straws. Tools like black (if there was a just god it
| would just be part of the language, like gofmt) can
| easily settle stylistic choices.
|
| Really not worth endlessly debating, what next, tabs vs
| spaces?
|
| This commit seems fine to me as well. Consider the
| golfing serves the purpose of "the lols" if you must.
| Heck this particular commit looks slightly better with
| the change!
|
| _Excessive_ golfing to the detriment of code quality
| over time is a well known and wholly irrelevant point to
| make with something like Tinygrad.
|
| If you are so convinced of what you are saying, fork and
| rewrite it as verbosely as you prefer and demonstrate
| some significant improvement to the project quality.
| Would be insightful.
| dundarious wrote:
| Dev dependencies and release dependencies are distinct.
| There would be no sense in complaining about hypocrisy if
| a zero-dependency C project uses clang-format or
| valgrind, and similarly for a python project using black
| or mypy.
| [deleted]
| elcomet wrote:
| This is a semi-educational project, so it makes sense to keep
| the number of lines small.
| [deleted]
| hoosieree wrote:
| Bonus challenge: total number of bytes monotonically decreases
| over time.
| eternityforest wrote:
| I think it's a bit unnecessary. A small set of operations
| should be enough to make it easy to port to new accelerators,
| and there's lots of nice to haves in many programs that don't
| affect compatibility.
|
| If you don't plan on ever doing anything but the core it does
| seem pretty reasonable though.
| [deleted]
| erwincoumans wrote:
| Agreed. It still leaves the loop hole of writing infinitely
| long lines of code (or is there a line length limit?)
|
| I also enjoyed watching the life streaming videos
| https://www.youtube.com/watch?v=Xtws3-Pk69o and the videos
| videos documenting the neural network ANE M1 chip reverse
| engineering efforts for this project. See video links here:
| https://news.ycombinator.com/item?id=30852818
| hmate9 wrote:
| This is the guy that did some iOS jailbreaks, reverse engineered
| the PS3 and now runs a self driving car startup.
| rickdg wrote:
| Not sure if it's a startup when it's profitable and not looking
| for investors.
| Teknoman117 wrote:
| Long enough ago that it was iPhone OS jailbreaks.
| pixelpoet wrote:
| And the shady cheapETH business last year:
| https://oldreddit.com/r/cheapETH/comments/lkzkso/george_hotz...
| jrib wrote:
| fixed link:
|
| https://old.reddit.com/r/cheapETH/comments/lkzkso/george_hot.
| ..
| fortydegrees wrote:
| Adding a comment as I don't think this is a fair
| representation of George. He livestreamed the creation of
| cheapETH as a technical demonstration of web3 development[0]
| and continually talks about it being worthless while
| developing it. He's also on record as being a serial 'no-
| coiner'.
|
| [0] https://www.youtube.com/watch?v=9LaIezgiUmw
| [deleted]
| jallen_dot_dev wrote:
| > continually talks about it being worthless
|
| Kinda weird to try to sneak millions of a worthless coin
| into your wallet.
| throwawaybner3 wrote:
| tromp wrote:
| I had to replace oldreddit by reddit to make that link
| work...
| ZephyrBlu wrote:
| Yeah should be old.reddit.com, not oldreddit.com.
| hbn wrote:
| I'm thinking he meant to link to old.reddit which forces
| the old (read: non-broken) reddit UI
| sydthrowaway wrote:
| So he's a two-bit scammer too.
| throwawaybner3 wrote:
| Who is goku?
___________________________________________________________________
(page generated 2022-04-04 23:00 UTC)