@AGB: I guess, though, that you don't translate the whole time when reading or listening to people, but more when speaking yourself? That's how it is with me, and I also find myself coming up with German phrases when speaking English.
@JNT: I think it's just down to time and exposure. The more you read and listen, the less you'll need to think about what it means. It also depends on how exciting the story is - with some extremely boring texts (letters from the tax advisor, etc) I still find that if I translate in my head I get a better picture, and remember what the message is, better.
On the other hand, I just finished reading "Das Parfum", which I found very gripping, and I was able to read that the same way I read an English book, without even being aware of the words, hardly, except when I came across a very nice phrase or word. There are plenty of things in that book which I understood perfectly but might not even know the English word for. This is after 15 years in Germany, mind! It's only in the last couple of years, really, that I've actually started being able to read in German in the full-steam-ahead way I read English books, and actually get lost in a book. It doesn't happen with every book; I still find I lose interest faster with German books, but it's happening more and more, I think.