Stewards can eject you from the stadium or refuse entry but they cannot search you or confiscate anything unless you give it up to them.
They can ask to search your bag or you, if you refuse they can refuse entry but you can step back if you wish. Fans get caught up in the official stance both stewards take and what fans give them.
If a steward asks for your name and address you can also refuse, they have no real powers.
My advice would be to remain calm and leave the stadium if asked, do not give them any excuse to call the police over. If they do again remain calm, do not start an argument.
What is best, having a row and getting your card took off you or simply walking out to fight another day?