So, Antman. I liked the movie, and it made me want to write about unstable villains because the movie’s main villain was unstable, and he was well done. (Incidentally, this is one case where a supervillain caused a super hero.)
One hard part in doing unstable villains is motive; it goes back to the fact that neither insanity nor evil is a motive. Instability isn’t a motive either. Where the instability should come into play is in making bad decisions. This can be in choice of motive, but more likely and importantly, in execution of achieving that motive. If the villain wants to gain fame, he will perhaps think that a dangerous and foolhardly plan, like kidnapping a celebrity and “convincing” them to include them in their show, or if they want a pie they gotta go rob a bakery, are good ideas. Or, if robbing a place, the villain may use way more explosive power than necessary because they didn’t bother to think through the consequences or risks of the actions.
[SPOILER]In Darren’s case, moments of true instability are like when he’s fighting Scott in the helicopter, just firing his lasers away. It doesn’t occur to him there are several allies he might hit – or if it does occur to him, he doesn’t care. Ditto on severely damaging the helicopter itself, which he’s currently in and needs to, y’know, not crash. Although when it does, he’s fine. His pilots, which he shot because he wasn’t worrying about them as he tried to destroy Antman, were not fine. But what does that matter? He’s unstable.
Also, it would take an unstable individual to trust Hydra. I mean, come on! Hydra are bad guys! To trust them, when you don’t believe in their ideology, is incredibly foolish…but they accomplished Darren’s goal of getting money and quickly spreading his technology, not to mention sticking it to Pym.[/SPOILER]
However, the biggest problem is in that poor decision making – that makes it easy for the villain to destroy himself. And as I’ve said before, that’s terrible for your story and your hero.
Which is why I liked Antman. Because they used the trope of Introduce The Forbidden Thing at the beginning of the film, the thing you should never do, and then to end the final conflict, completed the trope with Do The Thing. (I don’t feel bad about not tagging this as a spoiler; if you don’t know that as soon as the guy says “Never ever do this ever ever ever,” it’s going to happen before the end of the movie, you should.) That means that the defeat of Yellow Jacket – Darren – was all on Scott. Darren didn’t defeat himself. He did a lot of crazy unsafe stuff, but he was still smart – he still played the chess game with Pym, he wasn’t easy to take down. The defeat was based on the skills of the protagonists.
If you want your unstable villain to be scary, use the instability as a danger factor – and only a danger factor. Of course, I’m not opposed to the Portal Boss Defeat – you can’t defeat GLaDOS or Wheatly with just your portal gun. You have to use your brains and their bombs to stun them and then remove – or add – nodes. In other words, they do give you a necessary tool to defeat them: the bombs. In Wheatly’s case, you have to get the colorful sludges, too. There’s still puzzle work on your part.
And that’s the difference – if the unstable villain is laughing maniacally and zapping a laser without discretion, and the hero sees a mirror and some big thing hanging by a single support and tricks the villain into zapping the mirror, bouncing the laser, severing the support, and dropping the weight onto themselves, it’s still not exactly the villain defeating themselves. The hero still had to do something. Several things, actually – angle the mirror to hit the support (or know it was angled correctly), get the villain to actually aim at him, have great dexterity, enough to manage to dodge the laser, and keep the villain from realizing what the plan was (although that’s usually not hard with someone unstable.)
When you have an unstable villain, likely there is going to be some give on their part of making defeat easier for the hero due to bad decisions, but be very careful on just how much give there is, and always make up for the poor decisions allowing the villain’s defeat with several extra doses of danger. Someone who doesn’t care about the consequences will do much more to strike out at the hero – and if the hero isn’t okay with collateral damage, that can make an unstable villain all the worse.