Being shorter or taller depends on details of the diet. Vegetarians can obtain a balanced diet, with sufficient complete protein, vitamins, minerals and micro-nutrients. It's trickier for vegans but it can be done.