Chuan Li, PhD reviews GPT-3, the new NLP model from OpenAI. This paper empirically shows that language model performance scales as a power-law with model size, datataset size, and the amount of computation.
Update June 5th 2020: OpenAI has announced a successor to GPT-2 in a newly published paper. Checkout our GPT-3 model overview [https://lambdalabs.com/blog/demystifying-gpt-3/]. OpenAI recently published a blog post on their GPT-2 language model [https://blog.openai.com/better-language-models/]. This tutorial shows you how to run the
BERT is Google's SOTA pre-training language representations. This blog is about running BERT with multiple GPUs. Specifically, we will use the Horovod framework to parrallelize the tasks. We ill list all the changes to the original BERT implementation and highlight a few places that will make or break the performance.
...