What are your opinions about the meaning of this saying? Does it mean that a garment is too big, or too small?

I have often heard it used to mean that an article of clothing was too tight, but this seems illogical when you analyse the phrase itself.

Yesterday I thought of this when trying on a suit in a shop. The skirt was miles too big (didn't touch anywhere, except perhaps the waistband on the hips) and the jacket only fitted (touched) in one place and stuck out everywhere else. Therefore, in such a situation, it would seem logical to say "it fits where it touches", as where it doesn't touch it is too big.

If, however, a garment is too tight, then logically it touches everywhere. So why do people use this expression to mean that a garment is too small?