Duration :
Andre Carpathy, an AI researcher and founding member of OpenAI, delivered a talk on the state of GBT and the evolving ecosystem of large language models. The talk consisted of two parts, covering GBT training and how to use these models effectively for applications. The GBT training includes four major stages, namely free training, supervised fine-tuning, reward modeling and reinforcement learning, following a pre-processing step of tokenization. Pre-training involves gathering internet-scale data sets for a base model that undergoes pre-processing and a maximum of 1.4 trillion token sequences in a month of training. The resulting model can have billions of parameters and requires considerable computation resources.