Here is a link to an excellent document written by an attorney specializing in literary and visual arts.  It outlines your general rights as a photographer and gives some advice on what to do in situations where you may be confronted about your location and/or subject.

http://www.krages.com/ThePhotographersRight.pdf

This might be a good thing to print and carry in your camera bag!