It’s easy to compare the U.S. and the German higher education systems „from the outside“. When you’ve completed full degree programs and know both systems “from the inside”, comparison is much more difficult.
My personal feeling is that (as I believe the FAZ once put it) “Humboldt has moved to the United States”. The German Gymnasium is in no way comparable with a college education at one of the really top-notch 4-year colleges in the U.S. (the top 20-30, which, if taken together, would amount to 1-2 German universities in terms of total student population). At a college like Amherst, Williams or Swarthmore you are getting a generalist education at the very highest level, you are studying with a group of extremely bright, generally ambitions and usually exceptionally highly-motivated young students from the U.S. and abroad. Your teachers are often internationally recognized researchers and scholars (sometimes even winners of Nobel prizes), and in your final years at college you often get to know and work with them personally. You actually do get a significant degree of specialization (a “major”) during the last 2 years. There is really nothing comparable in Germany – there people have to be very intelligent and personally very diligent to avoid becoming “Fachidioten”. Graduate study at the top U.S. graduate and professional schools is, of course, outstanding… I would say that the only weakness is a general lack of knowledge of foreign languages which, from the perspective of people from the Anglo-American linguistic space, is understandable.
At lower-level institutions in the U.S. the story is quite different – in comparison with them the German system looks, in many ways, quite a bit better. There is one major caveat here, however: while the “state schools” used to be considered “inferior” to the Ivy League (Harvard, Yale, Princeton, Columbia, Cornell, etc.) and other top-notch private schools (e.g. Stanford, the University of Chicago, Johns Hopkins, etc.) I think that most people nowadays would say that some state universities (e.g. University of Michigan, University of California – not all, but many campuses -, University of Illinois, University of Texas, etc., etc.) are every bit as good as their best private counterparts. So in evaluating the US. System you really have to drill down into the details more than in the past, because the states (Bundesstaaten) have been doing a lot for education and often tend to specialize in areas that are important for their needs (e.g. agriculture or veterinary medicine, which can be even better at some state schools than at any of the private institutions).
German universities, on the other hand, which used to be the worldwide stars on the firmament of academic educations (Heidelberg, Tübingen, Göttingen, Munich, Berlin, etc.) in the 1920s have really gone downhill since the 1970s and have basically been degraded into “factories for producing qualified specialists for use in industry”. Allgemeinbilung is out (well, it never was in at German universities, because it was supposed to have been covered at the Gymnasium), and more and more German academics have some Allgemeinbilung only in a watered-down form, by virtue of osmosis from the media. As a result, their approach to their fields of specialization can often be somewhat uncreative. But while they have gone downhill, German universities have certainly not “gone to hell” – despite their mediocre showings in many international rankings, they are not all that bad. So it’s fair to say that an average German university is at or often well above the level of an average American university. What will happen as a result of the Bachelor-Master-Doctorate reform is really hard to say, but first indications are that this system does not fit the German system very well and may end up degrading it further.
It is true that good German students who have completed “Diplom” programs are well-respected in doctoral programs in the U.S., but I would say that this has more to do with the “cut-off points” of the various systems: a Diplom can in some cases be equivalent not to a master’s degree but to the so-called “PhD qualifiers”, i.e. the exams an American student takes before beginning to write his/her doctoral thesis (remember, in Germany there is, in general, no required coursework between a “Diplom” and a doctorate, any seminars would be optional, or on the recommendation of the student’s doctoral supervisor - his/her “BetreuerIn” or “Doktorvater”).
“Going back to school” in Germany is, of course, a much “bigger deal” than in America. That probably has to do with cultural factors, among other things.
Finally, the financial barrier to education should never be underestimated: despite all the financial aid in the U.S., the bottom line is that many students have to pay out-of-pocket or get loans. If one year can cost upwards of $30,000-$50,000 at one of the top U.S. professional schools, that’s NOT peanuts, so a degree at a German school can in many circumstances offer better “price/value”.