since you assume that the fault most likely is caused by the additional gears of the annual calender, and then you suggest that the best time to set the watch is noon (which is when the annual mechanism adjusts the date from the hour wheel side (remember, its a dual driven date mechanism: both the hour wheel and the Valjoux's date mechanism feed in), I have troubles bringing this together.
All this in mind, I would think that early moning could be a good time to set the date on this particular watch.
Best,
Magnus