You dont need a keg fridge, however it will make life easier,
But if you have a spot that you can maintain a constant cool temp, it can be done. I manage most of the year OK needing a shelf chiller for the summer months.
It does involve a bit more active management of the co2 pressure and the natural level of condition can swing somewhat with the seasons meaning i can serve some fizzy bitters mid winter.
You can also use a bit more co2 as its always wise to vent a keg prior to pouring from it to counter any condition loss and overpressurisation of the keg.
shelf chillers cant be enclosed without overheating and can be quite noisy in a quiet home, so if your pockets are deep you could look at a flash chiller like the pygmys http://www.lindr.cz/products/pygmy-25k
These connect to a keg and are ready to serve chilled beer at the flick of a switch..
even with a party tap its a good idea to insert a length of restricting 3/16" microline before the tap if you intent to keep kegs at condition maintaining levels of pressure. without it you will need to reduce the keg pressure to nominal to serve foam free..