It does make a physical difference how tall a swimmer is.
Taking a 50 m pool and assuming that a 150cm tall swimmer achieves propelling forward by half the body length and assuming that in the 3 strokes of freestyle, backstroke or butterfly a swimmer reaches the water surface after 10 m during the start it takes the 150cm tall swimmer 17.78 strokes to reach the end of the pool.
Assuming that a 180cm tall swimmers swims in a 50m pool and assuming that this taller swimmer comes up at 15m after the starting jump it takes the 180 cm tall swimmer 12.97 strokes to reach the end of the pool.
That is assuming that both swimmers have the same stroke efficiency. Hence we see that in most sprint events at the major international competitions the taller swimmers seem to dominate at the short distances.
I did not include breaststroke into the calculation because there is no rule that a swimmer has to surface after 15m neither at the start or after a turn, meaning that efficient kicking during the under-water phase can propel a swimmer half-way across the pool.
Yet it is quite interesting that in disciplines like the Individual Medley, we see it time and time again that those swimmers who are most proficient at the breaststroke also win the whole race that consists of a combination of all four strokes.