Potter's a good manager but it's one thing doing well at Brighton, quite another to take the step up to a team like us or Chelsea. He's got a nice group of players at Brighton who aren't that well known and work hard for each other. What works for Brighton might not work for Chelsea.
Doing well at Brighton may get the players a better gig but it's a situation where the whole is greater than the sum of the parts. The players listen to him and have responded well. Potter and his players are reliant on each other but there's a load of players at Chelsea who probably feel they have made it and have little to learn, particularly from a man who's never won anything.
That's not to say he won't do well but expectations and egos are bigger plus the players at Chelsea simply may not suit the way he wants to play.