The story goes that cannonballs used to be stored aboard ship in piles on a brass frame or tray called a monkey. In very cold weather the brass would contract spilling the cannonballs. Hence very cold weather is cold enough to freeze the balls off a brass monkey.
There are several problems with this story.
• The first is that the term monkey is not otherwise recorded as the name for such an object.
• The second is that the rate of contraction of brass in cold temperatures is unlikely to be sufficient to cause the reputed effect.
• The third is that the phrase is actually first recorded as freeze the tail off a brass monkey which removes any essential connection with balls.
It therefore seems most likely that the phrase is simply a ribald allusion to the fact that metal figures will become very cold to the touch in cold weather (and some materials will become brittle).