Also, when asked point-blank whether he believed in God, he answered "Yes, Spinoza's God," which would make him a pantheist. However, he also once said "I don't know that I can call myself a pantheist." So, really, it's up in the air. I'd be interested in finding out the chronological order of certain quotes, myself.
Actually it's really hard to define exactly what he was, Because he actually didn't believe in a personal god. I think what he called god was nature, he meant god in a poetic sense. In the strictest definition he was an atheist, I think he preferred the label agnostic because it doesn't have the knee jerk reaction that atheist seems to have to most people.