AnCap here. Unfortunately in our world there is no possible way to create an AnCap society, so you live within the bounds of what you have.
I’m of the mind that all of the forms of anarchism are utopian. They assume people will behave in certain ways which are contrary to reality. When people don’t behave in those expected ways the whole system breaks.
Game theory helps, as does training in computer security for similar reasons. Basically the idea is that you need to always assume people will try to break the system somehow if they think they can benefit from it.
That’s one of the reasons AnCaps believe in voluntary communities, that way those who didn’t advise but whatever the community’s rules are they can a) be punished based on the rules of the given community, or expelled.
There have been anarchist communities out there before. (Anarchist proper, not ancaps.) Most (though not all) of them have been destroyed by states who… well, not to put too fine a point on it but: states who hate freedom. I don’t necessarily know what people mean by “utopian”, i think anarchism is quite practical in a lot of cases but i think it also faces a number of significant obstacles.
I use the word utopian because it makes assumptions about people that aren’t true. Utopian systems can’t survive staunch and organized opposition.
This is a basic requirement for any system to survive. Look at the human body: the immune system is constantly fighting off threats. A person without an immune system (full blown AIDS) has a very difficult time surviving for very long.