If you got the job in the first place mainly because you look nice, I can't see why you should keep it when you don't.