I don't think it's a Yiddish thing. I just found this interesting article from 2010 in the Chicago Trib (which sort of confirms my long-harbored suspicions):
Which brings us to "come with," probably the most common example of Midwest speak. "I'm going for coffee. You want to come with?" (Also works for "bring with," "go with" and most verbs indicating movement.)
John Spartz, assistant professor in the department of writing studies at the University of Minnesota Duluth, explored the famous phrase in his PhD dissertation: "Do You Want to Come With?: A Cross-dialectal, Multi-field, Variationist Investigation of With as a Particle Selected By Motion Verbs in The Minnesota Dialect of English."
He says "come with" can be traced to the upper Midwest's heavily German, Norwegian, Swedish and Dutch roots.