I got a question. How can researchers who are not working at the largest tech companies even "compete" with models like GPT-3? From what I understand, these currently best performing models need an amount of training data and specialized hardware that is not obtainable for 99% of ML researchers. If you think you came up with a better architecture for a language model, one that could in theory beat GPT-3 on benchmarks, wouldn't you face the problem that you cannot actually prove that it would perform better?
This is a very reasonable question! And one that researchers think about regularly.
The reality is that you are right. There are simply questions that are inaccessible with the resources available at universities. We think about the questions that we can ask that don't compete or we find ways to compete, like having collaborations with corporations that have deep resources. Never mind access to engineers, which we almost totally lack in academia.
But at the end of the day. Yeah, if a random researcher got everything right about GPT, they couldn't have published it first, because they couldn't even have tested out a proof of concept. This is in part why people move to industry.
