Bluesky Thread

the wild part — this probably doesn’t have anything to do with AI or LLM arch...

View original thread
the wild part — this probably doesn’t have anything to do with AI or LLM architecture

e.g. the gospels of Matthew, Mark, Luke & John are all rephrasings of the same stories

Rabbinic teaching involved repetitive rephrasing

oral traditions in general are less about precise recitation
Tim Kellogg @timkellogg.me
Physics of Language Models: Part 3.1

If you show a fact to an LLM in pre-training once, it’ll memorize the form but not the fact itself.

but if you (synthetically) rephrase the text several times, it’ll memorize the fact

arxiv.org/abs/2309.14316
36 3
take it even further — precise recitation religious methods often result in people being able to recite without actually understanding

so this is probably a physics thing

something about information, learning, and some core law about cognition
9
36 likes 3 reposts

More like this

×