Yes - but I don't think the defender knew he'd moved it the first time, and I am certain he didn't see him move it the second time.
The keeper is called over because he is the only one who can see the angle properly, if he's said it's fine, and the defender doesn't know something has then changed, what can he do? If he overrides the keeper and moves the wall, he's giving the keeper more of the goal to cover. He's surely got to trust his keeper, especially if he's already asked him to double check?
Yeah, if he's turned and missed it happen, then it's not totally on him. Regardless though, I don't care if the keeper has said it's right - if the ball then gets moved it's no longer right, you have to adjust a touch.
If the guy on the end doesn't see it happen, someone in the wall must have, then they should push over.
Granted, it's possible that they tried and didn't quite have time. They could end with someone in the middle trying to move it but the guy on the end insisting it's right, unaware of the move.
Possibly difficult for them. Clever by Toney I'd say. It's all part of playing to the whistle and accepting the ref is always right - if you get away with it, well done.