FWIW "...have a bad conscience" sounds idiomatic to me.
"You constantly have a bad conscience for the civil job, the family, the children and the coaching assignment. Wherever you are, you believe you should be at ..." (Northumbria University)
"He remains unflustered by the flak. "I don't have a bad conscience. I know exactly who I am. I'm Jewish. I've never been conflicted about it, ...(The Guardian)
"These men, and many others in the government (only a quarter of the Cabinet voted for the Bill), have a bad conscience because they know the ...(The spectator)
"It's rather the political frameworks in developing countries which cause the environmental damage. So we shouldn't have a bad conscience." ..." (BBC)
from just the first page of Google UK results.