The relevant primary passage is this:
I imagine someone asking my advice; he says: "I have constructed a
proposition (I will use 'P' to designate it) in Russell's symbolism,
and by means of certain definitions and transformations it can be so
interpreted that it says: 'P is not provable in Russell's system'.
Must I not say that this proposition on the one hand is true, and on
the other hand unprovable? For suppose it were false; then it is true
that it is provable. And that surely cannot be! And if it is proved,
then it is proved that it is not provable. Thus it can only be true,
but unprovable."
Just as we can ask, " 'Provable' in what system?", so we must also
ask, "'True' in what system?" "True in Russell's system" means, as was
said, proved in Russell's system, and "false" in Russell's system
means the opposite has been proved in Russell's system. —Now, what
does your "suppose it is false" mean? In the Russell sense it means,
"suppose the opposite is proved in Russell's system"; if that is your
assumption you will now presumably give up the interpretation that it
is unprovable. And by "this interpretation" I understand the
translation into this English sentence. —If you assume that the
proposition is provable in Russell's system, that means it is true in
the Russell sense, and the interpretation "P is not provable" again
has to be given up. If you assume that the proposition is true in the
Russell sense, the same thing follows. Further: if the proposition is
supposed to be false in some other than the Russell sense, then it
does not contradict this for it to be proved in Russell's system.
(What is called "losing" in chess may constitute winning in another
game.)
-- Ludwig Wittgenstein, Remarks on the Foundations of Mathematics, Blackwell, 1956. Part I, Appendix III, #8. Translated from the
original German by G.E.M. Anscombe.
In a letter to Karl Menger, G?del comments:
As far as my theorem about undecidable propositions is concerned it is
indeed clear from the passage that you cite that Wittgenstein did not
understand my Theorem (or that he pretended not to understand it). He
interprets it as a kind of logical paradox, while in fact it is just
the opposite, namely a mathematical theorem within an absolutely
uncontroversial part of mathematics (finitary number theory or
combinatorics).
-- Kurt G?del, Letter to Karl Menger, 1972. Collected Works. Volume V.
Correspondence H–Z. S. Feferman, J. W. Dawson, W. Goldfarb, C.
Parsons and W. Sieg (eds). Oxford University Press, 2003.
Many early commentators on the passage agreed with G?del that Wittgenstein had simply misunderstood the first incompleteness theorem. However, in subsequent years others have come to regard Wittgenstein as saying something more subtle. The fact that Wittgenstein summarises the theorem succintly suggests that he is not challenging its correctness. Some commentators have interpreted him as disagreeing with G?del’s platonistic interpretation of the philosophical significance of the theorem. Others have claimed that Wittgenstein is adverting to the possibility of non-standard models of arithmetic. Others that Wittgenstein is questioning the traditional notion of what constitutes mathematical truth. The speaker you quote is perhaps proposing the non-standard model option.
It is true to say that one can and should draw a distinction between the technical result that is G?del’s first incompleteness theorem and what we might call the philosophical significance of it. Many popular expositions of the theorem jump to the conclusion, "There are mathematical propositions that are true but unprovable". Statements like that, without qualification, are potentially misleading. We should rather speak of what is provable in specific recursively axiomatisable formal systems. And we should clarify what we mean by mathematical truth. There is room for philosophical disagreement. For example, intuitionists do not accept that it is intelligible to speak of mathematical propositions that are true but unprovable. Consequently, they have a distinctly different way of understanding the significance of the incompleteness theorems. I said something about this in my answer to this question.
Here are some references discussing the interpretation of the passage from Wittgenstein’s Remarks. There is also some useful material in the SEP article on Wittgenstein’s Philosophy of Mathematics, section 3.6.
Juliet Floyd, "Prose versus Proof: Wittgenstein on G?del, Tarski and Truth", Philosophia Mathematica, 9 (2001), pp 280-307.
Juliet Floyd and Hilary Putnam, "A note on Wittgeastein's notorious paragraph about the G?del theorem", Journal of Philosophy 97, (2000), 624-632.
Mark Steiner, "Wittgenstein as his Own Worst Enemy: The Case of G?del's Theorem", Philosophia Mathematica Vol. 9 (2001), pp. 257-279.
Timothy Bays, "On Floyd and Putnam on Wittgenstein on G?del", Journal of Philosophy, 101 (2004), pp. 197-210.
Charles Sayward, "Steiner versus Wittgenstein: Remarks on Differing Views of Mathematical Truth", Theoria 20 (3):347-352 (2005).