I better think of a faster workaround. I don't know what's best. Either decide to live with this layer scale/position popping, and re-scale in animation every time it happens, or ....
... scale each character to scale = 1, and rescale every single stinking vector shape.