I think the main disconnect between large language models and what people expect from "AI" is the fact that LMs learn meaningful representations whereas most people associate intelligence with meaningful behaviors.
The term meaningful is a bit tricky to define. But when GPT-3 says one thing in one paragraph but the exact opposite in the next paragraph, my impression is it hasn't learned "meaningful representations" but just "plausible associations". A lot of language is just loose associations and so GPT-3 can do that as well as many people there and it can pull some logic and seem to reason (but can't do that reliably). IE, I think even just informal language has state that human track and so GPT-3 falls at this part of informal language (while can seem to otherwise do well).
What you observe there is context limitation. GPT-3 as is, has a fixed length context window which limits its capacity to remember and be coherent across long horizons. A hierarchical version that contained summaries e.g. for previous sentences, paragraphs, and so on, could in theory, behave much better.
This is the more "sophisticated" approach. The brute force approach says to just throw more compute, larger contexts and see what happens.
The term meaningful is a bit tricky to define. But when GPT-3 says one thing in one paragraph but the exact opposite in the next paragraph, my impression is it hasn't learned "meaningful representations" but just "plausible associations". A lot of language is just loose associations and so GPT-3 can do that as well as many people there and it can pull some logic and seem to reason (but can't do that reliably). IE, I think even just informal language has state that human track and so GPT-3 falls at this part of informal language (while can seem to otherwise do well).