Because this is such a broad topic, I’m certain that this discussion on the recurring villain is something to which we’ll…keep coming back. Just like the villain himself.
I personally always favored the Overlord type villain over a recurring villain, and I can’t say I was ever too fond of even the recurring minion of the overlord. In part because I always wonder why the Overlord keeps sending out the same stupid minion if he failed the first time. But an Overlord is not, of course, always an appropriate villain, and even when he is, sometimes the recurring minion is just appropriate. As long as there’s a clear reason why the same minion is being sent out, anyway. Let’s make the analysis and discussion of the recurring villain easier for our future encounters with him by breaking it down into types:
That guy who is always trying to beat the hero, perhaps sometimes succeeding, and usually nearly equal in power. I like The Rival because usually, at worst, The Rival is just a huge jerk; he is rarely actually evil. In fact, it’s usually better to just label a rival as an antagonist rather than a villain.
One problem with a recurring villain is the rate of villain growth versus the rate of hero growth. The Rival is a good way to make that whole process easy. Both protagonist and antagonist are usually striving for a similar goal – like in good old Pokemon Red/Blue, Red and Gary/Blue are both striving to be pokemon masters. Another common example are the two students both striving to be first chair in band or orchestra, or to…do whatever it is in sports that rivals would be doing…or to score the highest and become valedictorian. Progress is easy to see with The Rival and it makes sense that The Rival would be getting better alongside the protagonist.
This is a villain, whether minion or Big Bad or otherwise, that is following the heroes for one reason or another. A good example are the Nazguls. They’re minions, yes, but they leave Sauron and just follow Frodo until either they get thrown off the trail or distracted by something else (like, say, wars). There’s a similar Nazgul kind of creature in Terry Brook’s Sword of Shanara, called the Skullbearers. They also just follow Shea around until Gandal-er, Allanon kills them.
Usually, as with the Nazguls and their Brook’s knockoff, The Pursuit is way out of the heroes’ league. The heroes’ only choice, when confronted with The Pursuit, is to run or else die. Sometimes, perhaps often, the main hero doesn’t ever even defeat The Pursuit; instead, someone who is already established as way more badace has to take care of it, and far later in the story when they’ve racked up enough badace points.
With a bigger bad, to show up only as a final boss, the minion is the guy the Big Bad sends to stop the heroes. Again. And again. The Minion is different from The Pursuit in that The Minion periodically reports to his master and may be assigned to something besides getting at the heroes. Also, he is rarely stronger than the hero, or if he is, he scarcely is for long.
Like I mentioned more than a few times, I usually don’t get this guy because it doesn’t make sense for a good overlord or other Big Bad to send the same guy. But there are a few viable options. A recurring minion could be too strong until the hero beats him at the end. Although, it would almost have to be the hero that beats The Minion, since the presentation would likely show the hero as working to become strong enough to beat this recurring villain. It would just be rude to take that away from the hero. This is also different from The Pursuit in that usually the hero is just trying to get away from the villain, not overcome him. However, even with a minion that could kill the hero if he could only get his hands on him, it still makes little sense for the Overlord to send him out very much. Imagine the conversation:
“So did you get her?”
“Yes! Well, no. I was totally killing her to death and then this thing happened and she got away.”
“What, again? You said the last time it was a total fluke and it wouldn’t happen again!”
“Yeah…Yeah, I know. Look, she’s just really lucky!”
“…I needed her dead, Minion. Like, last week. When I first sent you.”
Why does the Overlord send even an otherwise very competent minion again and again? You’re probably going to have to change the game here. Maybe The Minion is just supposed to catch the hero. And he does. And she escapes. Again and again. Or maybe the Overlord isn’t worried about the hero but would like her dead and The Minion is told to kill her as a rite of passage.
Another option is to have a minion running the errands instead of trying to fight the heroes. The only problem with doing so, however, is that if the minion is consistently thwarted, he’s also going to have to be removed by the Overlord. So you might want to let the minion win on the less important, or at least less story-breaking, tasks.
This is that villain or at least antagonist that is just a part of the hero’s life to such a degree that of course they’re going to show up again, and again, and defeating them rarely/never means killing them. These are people like the terrible teacher, or the bully, or the evil ex. Most modern-setting non-action movies have this kind of recurring villain, especially if it’s a school setting. Fantasy school settings will definitely have these, too. Consider, after all, Severus Snape and Draco Malfoy.
Unpleasant Associate is different from The Rival because The Rival is specifically trying to best the protagonist in some way. Be the first to X, be the one to achieve Y. Unpleasant Associate, however, is just a part of life, one of those social interactions that are burdensome, a person who just makes life generally more difficult for the hero in some way every time they meet. Probably because Unpleasant Associate just plain hates the main character. And who knows? Maybe the main character is just as bad and tries to retaliate in kind…but if that’s the case, that does put them more in the rival category, since now the rivalry is about who can ruin the other’s life most.
And then, of course, there’s the:
Goldfish Poop Gang
I won’t describe them since it’s all in the link. Honestly, unless they’re a separate entity from the Big Bad, they’re just terrible minions who need to be terminated. These guys, if involved, had probably better be lovable in some way or else they’re just going to be really annoying. (Although with Team Rocket – they may be annoying but at least they mean extra exps and monies, which makes up for their probably not being lovable.) Be cautious using them in your story.
So how is a recurring villain different from the main villain?
I feel we should end with this as it’s an important enough of a question. What is the difference? I believe it has to do with how much the villain sits on the forefront of the mind of both the readers/viewers/players, and the hero. If the heroes, and perhaps subsequently the reader, are thinking heavily about the villain, they’re probably not a recurring villain. The recurring villain has to fade away, so when he shows up, it’s a reaction of, “aw, not this guy again!” or “Oh, no, it’s that guy again!”.
With some of the examples I already provided: rivals are a constant problem, but usually the hero, and the audience, is more worried about the hero’s own accomplishments. In Red/Blue, the player usually isn’t thinking about Gary. Maybe if a rival battle is coming up, the player might worry whether his pokemon are of the right type and level to beat him. But otherwise, the player is thinking about how to complete the pokedex or how to beat the gym leaders or Team Rocket. If The Rival is too present, whether physically or just mentally, The Rival kind of moves out of the realm of recurring villain and into main villain.
The Nazguls are a constant threat, yes, and one worries about them finding Frodo…but usually that fear is greatest when they’re actually around, or Frodo put on the ring again and they’re all attracted to him. Otherwise we’re usually more concerned with the orcs or what Sauramon’s doing. There’s so much else to worry about, the Nazguls are only a problem…when the Nazguls are a problem.
With The Minion, the thought is always on the Overlord. The Minion is usually just a reminder that the Big Bad has to be stopped. Soon. At all costs.
Finally, with Snape and Draco – they’re pretty persistent, as Unpleasant Associates tend to be, but they’re not constantly considered the huge threat. They’re annoying or even pawns or tools in the battle against the Big Bad, but they’re not the main focus. That’s Voldemort and Voldemort alone. With an evil ex, the sensation of “ugh not her” is very present, and the hope that, after she leaves, she will not show up again any time soon indicates that it’s now time to think of something else.
So there you have it. Four categories of recurring villains.