Yeah i remember that Ed article ! I don’t think the technical aspects are relevant to the newer generation of models, but yeah of course any attempt to compress inference costs can have side effects : either response quality will degrade for using dumber models, or you’ll have re-inference costs when the dumb model shits its pants. In fact the re-inference can become super costly as dumber models tend to get lost in reasoning loops more easily.
Yeah that’s also something that you have to train for, i’m not super aware of the technicals but model routing is definitely important to the AI companies. I suspect that’s part of why they can pretend that “inference is profitable” as they are already trying to squeeze it down as much as possible.
I mean what’s the point of morging if you’re gonna do it intermittoucly?
To clarify : model collapse is a hypothetical phenomenon that has only been observed in toy models under extreme circumstances. This is not related in any way to what is happening at OpenAI.
OpenAI made a bunch of choices in their product design which basically boil down to “what if we used a cheaper, dumber model to reply to you once in a while”.
I don’t know about other bands but the bit about iron maiden is really stretched.
I guess if you consider the first lineup to be the one for their first concert in a bar’s basement, alright. But if you take the first album, Dave Murray was already in the band and still is.
Companies that have had their whole data on Google/MS servers for 20 years certainly don’t care for privacy the way you and I do. But they are certainly realizing that US providers are not the way to go. Baby steps I guess.
Oh yes. Because Merz builds data centers, and Der Leyen is known for making IT decisions at EU companies. They also happen to be the queens of private investors.
testing is doubting
I’m sorry but no, models are definitely not collapsing. They still have a million issues and are subject to a variety of local optima, but they are not collapsing in any way. It is not known whether this can even happen in large models, and if it can it would require months of active effort to generate the toxic data and fine-tune models on that data. Nobody is gonna spend that kind of money to shoot themselves in the foot.