The downside to being a classical liberal (or libertarian, or whatever you wish to use) is that you find yourself grouped together with social conservatives by liberals who don't know any better and with socialists by conservatives who don't have a clue.
I am not a social conservative, at least not in most ways. I am definitely not a socialist.
Why people can't understand something in the middle is beyond me. Why do so many people the world to be "right/left" or "fundamental/atheist" in nature? What's wrong with complexity? Nuance? Questioning?