It's true that the most polite way to address a woman customer is 'ma'am' in AE, 'madam' in BE. 'Madam' is used sometimes in very formal AE, but only to address people with high official titles: Mr. President, Mr. Ambassador, Madam Secretary, Madam Prime Minister.
But in a restaurant, it's not often necessary to use such formal words. 'Thank you, ladies' or 'Thank you, gentlemen' is certainly possible, but if you don't want to use 'Thank you, ladies and gentlemen,' why not just say 'Thank you very much' with a friendly smile for the whole table? You can always add something like 'I enjoyed serving you' or 'We hope you will come back again soon.' The smile is the main thing.
Another thought: if the customers speak English, one person in the group will probably have paid with a credit card, so you could even say, if you like, 'Thank you very much, Mr. Smith' (having read his name on the card). (-;