Wheel Back Space vs. Handling

I am looking for feedback from the experts. My current set of rims on my late model VD with narrow track suspension have back spacing of 3 1/2" in the front and back. If the back space increases to 5 1/2" (pulling the wheels inward), which results in a narrower track at the outside of the wheels, will this affect handling? Hopefully I explained this correctly.

Wheel Back Space vs. Handling