Yes it transpires actually that people are very easily duped. There's even a term for it - I cannot remember it off-hand but it's part of our psyche to attribute "human-ness" to non-human things, and when something actually pretends to BE human, then we get drawn in hook, line and sinker.
I think it may be a long time (decades?) before we really know whether these things GPT5, GPT6, Palm2, Deepmind Gemini etc have become conscious. We have no way of looking inside and figuring out what they are doing, and moreover even if we did, who's to say that some other non-human structures are actually thinking, even though they may not be doing it the same way humans do it.
But whilst that question is interesting, it's also a bit of a moot point. If it walks like a duck, talks like a duck and quacks like a duck, then effectively it is a duck. If we cannot tell whether its thinking like a human or not, does it matter whether it is or not?