the biggest reason for fully open models is science, and the downstream effects
1. rebuild it, but with your own domain-specific mid-training data
2. try methods out on several snapshots
3. attribute answers to specific documents in the training dataset
4. …
the biggest reason for fully open models is science, and the downstream effects
View original thread
on #3, this paper uses a method where they can directly attribute specific documents from the pretraining dataset
they used it to show that LLMs do in fact learn procedures, not just autocomplete. But you could take this so much further with Olmo3
arxiv.org/abs/2411.12580
they used it to show that LLMs do in fact learn procedures, not just autocomplete. But you could take this so much further with Olmo3
arxiv.org/abs/2411.12580
15
2