I’m not the guy you’re arguing with, but it will be interesting to see if/how LLMs deal with novel Gödel sentences in a way that is more than just mimicry. I’m sure there is a way to set this up that is mathematically rigourous.

The idea is that there are some statements which are true but unprovable in a formal system. Since LLMs run on a computer they are technically a formal system of sorts. So it will be interesting to see if they can pinpoint the true yet unprovable sentences. If they can it will lead to a lot of interesting questions about how exactly they know it’s true without “proving” it in some roundabout way.

I’m not doing this issue justice. A lot has been written on this by Roger Penrose (you don’t have to agree with his stance to appreciate that he’s touching on an interesting problem, here).