It was a great video. You're doing interesting work.
I've also seen implementations of realistic neurons, spiking models, etc. In software implementations, what combo of libraries and hardware would equal your 200,000 biological neurons in performance (esp training)? How many GPU's are we talking about?
(Note: If you haven't already, it might be helpful to publish a stack like that so people can experiment with encodings or reinforcement methods at no cost to you.)
We focus on using real neurons, I'm not aware of a software based equivalent. But users can `pip install cl-sdk` to get started with our API. The SDK is still early but supports playing back a recording of real data so applications can be built with a realistic spike frequency. (We'll be releasing a set of recordings for this)
For academic papers, the phrases they use include "biologically plausible," "Hebbian learning," and "backpropagation free." Searching for those with "neural networks" or "neurons" will turn up cutting-edge techniques.
We'll likely leave the answering of your equivalence question to our users. While we do plenty of internal research, the primary goal of our cloud platform is to dramatically expand access to this field. There are no doubt many low-hanging fruits to be discovered.
I remember trying out Macromedia Flash 6.0. My GUI apps were ugly at the time. Learning to build something like I saw in the movies could take years. Then, Flash let me throw together beautiful, animated interfaces like it was nothing. One could do quite a bit after one tutorial.
(Note: Quick shoutout to Dreamweaver 6.0 which was a power, WYSIWYG editor. Today, things like Pinegrow might fill the niche.)
It's death as a hugely-popular tool was largely due to Apple and Adobe. SaaS model isn't helping it far as wide adoption goes. It also got popular through piracy which hints the replacement should be profitable and widely deployed like open source.
I think this might be a good opportunity for a license like PolyForm Non-Commercial. Free users either can't commercialize their content or, like CompCert Compiler, must make the outputs GPL'd (or AGPL'd). The Flash replacement would have a fair, one-time price for unrestricted use with source or you share like they shared with you. What do you all think?
Of the two, I think Adobe is most responsible for the decline of Flash. Even if smartphones had never entered the picture, laptops (where efficiency is important) were quickly becoming the most common form of PC, which would've eventually made Flash as it existed under Adobe untenable as well. The timeline was just accelerated by smartphones.
Honestly I can't understand the mental calculus that went on in the heads of Adobe execs at the time. Yes, cleaning up the ball of mud that the Flash codebase had become and making it not so battery hungry wouldn't have been an easy task, but it would've futureproofed it significantly. Instead they decided to keep tacking on new features which ended up being entirely the wrong decision.
EDIT: The constant stream of zero-days certainly didn't help things either. A rewrite would've been worthwhile if only to get a handle on that.
Flash was not particularly battery hungry (My go to example when HTML 5 demos started coming out was rebuilding a HTML 5 demo that was using 100% of 1 core into a flash app that used 5%).
The reason it burned CPU cycles is that non-coders could make programs with it and they would produce the world's worst code doing so that "worked". The runtime itself was fine (efficiency wise, not all the other things).
I think Apple is more responsible. One of Flash's chief benefits to the customers who paid the big bucks was that it 'just worked' everywhere. Once Apple stopped supporting Flash on the iPhone, that story was a lot less attractive.
The bugs were definitely Adobe's fault: as with most tech companies, they were far more interested in expanding the feature set than they were on fixing the bugs and stabilizing the platform.
The first iPhone had 128Mb of RAM and a 400Mhz processor and could barely run Safari. If you scrolled to fast you would get a checkerbox while Safari was trying to catch up.
If it couldn’t run Safari with decent performance, how was it going to run Safari + Flash?
In fact when Flash finally did come to Android in 2010, it required a 1Ghz processor and 1GB of RAM and it barely ran then and killed the battery. An iPhone with those specs didn’t come out until 2011.
Another anecdote is that the Motorola Xoom was supposed to to be the “iPad killer” because it could show you the “real web” with Flash.
But it came out without Flash because Adobe was late. You couldn’t even see the Xoom home page on the Xoom when it was first introduced because it required Flash.
It was probably hard to imagine the rise of laptops, internet, and Flash.
Flash itself was acquired via Macromedia as well.
Adobe's business is keeping technologies billing, and while Flash had it's flaws, the world was not ready for it to depart as soon as it did, because there was not a capable replacement available.
The "pay to sell your work" model is basically what Autodesk does to provide a version of Fusion that's free/accessible to the hobby 3d printing market while still protecting their b2b revenue.
I haven't looked in a while, but I believe there's music and audio production tools with similar approaches.
More impressively, Da Vinci Resolve is actually free with no restrictions. It is a high end video editor that film makers and professional film studios use (together with hardware and some paid features) from black magic design. Incredibly impressive. Affinity Photo and PhotoPea are also now free without restrictions.
There was an open, real-time strategy game created for this purpose long ago. I think it was intended for designs like the Starcraft AI's of the time. Anyone remember or use it?
I feel like Chris way, way understates the prior art on C language. It's not just two compilers and some textbooks. That would actually be impressive.
If Internet trained, the training data probably has so much stuff about C compilation in it. Books, step by step articles with code, debugging, StackOverflow answers, mailing lists, blog posts, compiler results... endless. It's one of the most specified and pretrained things in existence for a multi-TB, Internet-trained LLM.
A real test for Claude Code Compiler would be something that had hardly any search results. Then, the few results it had were the language description, some examples, and maybe the existing compiler. Could it output a compiler for that? Can it do ZL (C++ in Scheme), Pony, Mercury, or Cray's Chapel?
Even easier, Lattner should try to have it write a LLVM-compatible compiler for Mojo. Then, lots of multithreaded, SIMD, and GPU implementations of business and ML algorithms in Mojo. That might not only be a good demonstrator but help the business, too.
"Is Google hurting itself in its confusion? Google is the largest scraper in the world. Google's entire business began with a web crawler that visited every publicly accessible page on the internet, copied the content, indexed it, and served it back to users. It did this without distinguishing between copyrighted and non-copyrighted material, and it did this without asking permission. Now Google is in federal court claiming that our scraping is illegal."
The above was my take on the situation, too. If SerpStack is illegal, then Google is illegal.
If SerpStack has to cease to operate or pay damages, nearly everyone with a web server will be collecting a check from Google as they're being liquidated.
1. Generic, mask layers and board to handle what's common across models. Especially memory and interface.
2. Specific layers for the model implementation.
Masks are the most expensive part of ASIC design. So, keeping the custom part small with the rest pre-proven in silicon, even shared across companies, would drop the costs significantly. This is already done in hardware industry in many ways but not model acceleration.
Then, do 8B, 30-40B, 70B, and 405B models in hardware. Make sure they're RLHF-tuned well since changes will be impossible or limited. Prompts will drive most useful functionality. Keep cranking out chips. There's maybe a chance to keep the weights changeable on-chip but it should still be useful if only inputs can change.
The other concept is to use analog, neural networks with the analog layers on older, cheaper nodes. We only have to customize that per model. The rest is pre-built digital with standard interfaces on a modern node. Given the chips would be distributed, one might get away with 28nm for the shared part and develop it eith shuttle runs.
Bruce Schneier had the Friday Squid Blogging posts that he allowed random stuff on. Most of the noise was contained in the Friday threads.
Lobsters had a weekly thread by caius called, "What are you doing this week?" People would post personal projects or experiences in it. On top of interesting tech, that let us pray more specifically for and encourage some in need.
You could charge people to fine-tune or customize models. Maybe synthetic data. Maybe rent it on vast.ai but careful about your bandwidth and energy. I'm not sure of the security implications of vast.ai, either.
I've also seen implementations of realistic neurons, spiking models, etc. In software implementations, what combo of libraries and hardware would equal your 200,000 biological neurons in performance (esp training)? How many GPU's are we talking about?
(Note: If you haven't already, it might be helpful to publish a stack like that so people can experiment with encodings or reinforcement methods at no cost to you.)
reply