I think your pilot jets maybe clogged/gummed up...being a VT500. These are the ones that most have an effect on the "off idle" to main jet. They are teenie, tiny little suckers and they are hard to see. Use a toothbrush bristle or something like it....well maybe a straight pin to get ya started for cleaning.
I'd also check the rubber boots on the carb to manifold for tightness.