Applied AI·How models work
this is the phase that gave the model its knowledge, and the same phase that gave it a cutoff date.
Pretraining
Draft summary, pending review
The first and most expensive training phase: the model learns next-token prediction over trillions of tokens of web text, code and books. This is where its knowledge and language ability come from, and it is why the knowledge has a cutoff date.