In most sports they have a physical effect on your performance, in swimming only psychological. If you worry about what your rival is doing, you take your mind off what you are doing and so fail to concentrate on your performance.