I consider old family photos one of the "must haves" of research. I've actually identified relatives through labelled photos. I couldn't figure out who they were, dig a little research, and surprise... I was looking at cousins.
Of course, not everyone is lucky enough to have inherited their family's photo trove. I've already mentioned a few resources for locating photos. I just discovered another one. Kimberly Powell provides a useful list of family photo websites on her About.com site. I wouldn't use every site (some are hard to use, others charge more than I'm comfortable with), but it's a nice starting point for research.