I found this, for the noun form here:http://www.docguide.com/dgc.nsf/html/English-...
1 : a fact or condition incompatible with another : OPPOSITE -- usually used with the
2 : one of a pair of opposites
- on the contrary : just the opposite
- to the contrary 1 : on the contrary 2 : NOTWITHSTANDING
However, the problem I see is that I am unfamiliar with this usage. I have never used "contrary" as a noun. In this sense I would only use "opposite".
There is another, completely different meaning for contrary: "unreasonable".
Example: "He's just being contrary for no reason at all."
Strangely I had to go to a BE site to find this definition. It may be primarily BE for this reason. Although I'm American, my father and his family grew up in England, so there are many words that seem quite natural to me that may not be standard here.