Sounds good. We've done a few trips (including Geneva last year and Venice this year) where we fly out on the Saturday morning and return on the Sunday night.
Like you say, it is dependent on minimal flights delays, but so far it's always worked out OK for us.
I enjoyed Brussels - The Atomium...