This expression comes from the poet Alexander Pope's description of congenial conversation in Imitations of Horace : The feast of reason and the flow of soul.
feast or famine :
either too much of something or too little
a ghost at the feast = a spectre at the feast :
someone or something that brings gloom or sadness to an otherwise pleasant or celebratory occasion
The ghost or spectre of Banquo at the feast in Shakespeare's Macbeth is the most famous literary instance of this. There are other versions of the expression. A skeleton at the feast dates from the mid 19th century and probably refers to the ancient Egyptian practice of having the coffin of a dead person, adorned with a painted portrait of the deceased, present at a funeral banquet. A death's head at the feast alludes to the use of a death's head or skull as a memento mori (an object which serves as a reminder of death).
a movable feast :
an event which takes place at no regular time
In a religious context a movable feast is a feast day (especially Easter Day and the other Christian holy days whose dates are related to it) which does not occur on the same calendar date each year.