irodkin/gpt2-wiki2

metrics on 1024 context:

• —valid_perplexity = 14.79
• —validcrossentropy = 2.69
• —train_perplexity = 13.77
• —traincrossentropy = 2.62

metrics on 252 context:

• —valid_perplexity = 17.35

metrics on 378 context:

• —valid_perplexity = 16.4

metrics on 504 context:

• —valid_perplexity = 15.86

Dependence of the cross entropy loss on the length of the context for prediction

• —x-axis*128 = context length
• —y-axis = cross entropy