"LLMs predict the next token"
this is insane, right?
predict? as if there's a ground truth? like there's some predetermined state of the universe that it's predicting? but also somehow not a part of?
"LLMs predict the next token"
View original thread
27
2
i understand how we got here, we stumbled into it
sure, during pre-training it makes sense. There actually is a ground truth it's predicting
but in inference it's just selecting the next sensible thing to say. that's a wildly different thing
sure, during pre-training it makes sense. There actually is a ground truth it's predicting
but in inference it's just selecting the next sensible thing to say. that's a wildly different thing
11