The Tories have said that they would potentially leave the EU without a deal, so they can cost on that basis. If we then decide on a deal, the costs of that can be individually costed: assets + increased trade - bill = cost. I would hope that an acceptable deal would have a positive cost amount. By heading into the negotiations stating we won't leave without a deal, Labour are leaving that particular cost to the whims of the EU.
You are spot on - the EU can ask (demand) whatever they like and Labour will either pay it - so £85bn is their starting point - it will only go up with Labour bending over.
So have they costed in £85bn ? Does not seem so - so all their forecasts are built on sand
More likely though they would spend years being fucked over by the EU - with them continuously increasing their bill - and it will most likely end up with us withdrawing the A50 notice and staying - without a rebate, without opt-outs and tail firmly between our legs.