I think the myth started over "usable" size as opposed to visable.

If a person is really over weight, they may not be able to use as many inches on their loved on. Hence, it grows (in the view of your mate) when you lose weight.