GPT-1 was trained with BookCorpus, a large-scale dataset of more than 11,000 unpublished books designed for training and evaluating machine learning models whose primary focus was language understanding.