Wearing it for a few hours is not enough to fully wind it automatically. Depending on the movement, you might have to wear it for 10-12 hours a day for several days to get a full wind.
I recommend you manually wind it once a day if you're only wearing it for a few hours in the evening. You can't overwind it, and it will probably keep better time.