I think you and I are pretty aligned in our thoughts on ChatGPT. I think its revolutionary, and I use it every day. But I don't consider it generally intelligent, I do think that novel thoughts of demonstrable value are the real test for that, and a useful invention would convince me beyond all doubt.
In my opinion, the mental model you should use when evaluating LLMs-vs-Human capability: When asked a question, the LLM has to answer without being able to iterate, back-track, or even use a scratch pad such as pen&paper or a text editor. Basically it's the same as an "oral exam", where you have to stand in the middle of a room and get grilled by a professor to determine your knowledge on a subject.
Don't compare "human with tools and unlimited time" to "LLM with no tools and seconds of time". Compare "human being interrogated in an empty room" and then it is much more clear where an LLM rates.
GPT 4 is definitely super-human in some areas, such as general knowledge and translation between languages.
No human knows as much, or can speak as many languages.
Ask yourself this: can any human, when asked to "invent something", just do it, then and there?
Give it a billion subjective hours. Give it databases and network access. Do you really think it will produce something? You definitely do a good job of explaining one of the limitations an LLM faces compared to a general super-intelligence.
