So here's my thoughts on the matter, specifically on the part of what makes and doesn't make an RPG.
Then I ramble.
In Which I Quibble About a Definition
Roleplaying is when actors in a game assume roles (knight) and motivations (protect the king) within a certain context (a battlefield). (The stuff in parenthesis can, so far, be applied to both chess and a story game.) The moment when a game becomes a roleplaying game is when choices (and player fun) starts to come from the established motivations (or story) of the game's actors, instead of from mechanical considerations.
You'll notice that this is a definition based on player motivations, and not on the game's rules. I am going to argue that what definese a "roleplaying game" is a result of gameplay, not of game design.
Chess can be a roleplaying game. If both players consent to play chess as a role-playing game, they can damn well play it as a roleplaying game. The black rook could court the white bishop, a pawn could agree to allow a knight past his defenses in exchange for surviving the war, etc. None of this requires any changes to the rules, it just requires the two players to agree that the fun comes from roleplaying, not from strictly pursuing a mechanical victory condition, then they are now playing the game as a roleplaying game.
Storygames can also be stripped, and can be played as if they weren't roleplaying games. Take a group of the most cliched, powergaming munchkins you can imagine and drop them into a storygame. They are only here to kill the evil wizard and rescue the princess. Everything else is just symbols for them to manipulate. A munchkin might (briefly) talk to the NPC king about nobility and romantic love, but only if they think that'll give them access to their next goal. Concepts like "nobility" and "romantic love" are just symbols that interact in a certain way, like choosing cards in Apples to Apples or Cards Against Humanity.
Another way to think about How to Take Roleplaying Out of a Roleplaying Game: Could a computer be taught to play it? If we built a AI ("Deep Blue Book") to play a roleplaying game, could it take part of some activities that look like roleplaying, while strictly maneuvering for a campaign's final victory condition? If it was 5e, the AI's character could have the flaw "rude to servants", and so whenever the DM mentions that there is a butler, maid, or linkboy nearby, the AI's character will say something rude to the servant. It looks like roleplaying, but it's a mechanical calculated action. There is no "role" there--it's just an internally meaningless manipulation of a symbol. The AI's flaw could be "spouts prime numbers when HP = a member of the Fibonacci series" and the AI wouldn't care. And if the goal-focused AI was playing with other goal-focused AIs, then the other players wouldn't care either.
Roleplaying among players comes about by agreement, moreso than rule design.
Yes, yes, of course some games are much better for roleplaying that others, according to how they are designed. But if you can roleplay chess and powergame through Lady Blackbird, it's clear that we're talking about a gradient that depends on how much the players want to roleplay. Actually, it's more complex than a simple gradient, since different players will react differently to different incentives to roleplaying.
Chess is tough to roleplay. All chess brings to the table is a bunch of agents (the nouns that verb with intention) with evocative labels like "knight" and "queen", but everything else needs to be supplied by the players. Checkers would be even harder to roleplay.
DnD is easier to roleplay because it gives you more context for the game's agents, so the players have to supply less. Not only do they have an evocative label like "wizard", but they presumably have a comprehensible context, like "defending a village from miniature dingos" or "shipwrecked on an island". Best of all, the humans playing the game have an understanding of all the symbols in the game, because the game's symbols mirror our own. You don't have to explain how the symbol for "fire" interacts with the symbol for "unlit torch" interacts with the symbol for "darkness".
So with a stronger context for the actors and their motivations, you can quickly move onto the fun stuff like moving, talking, discovering, and learning. Cool.
You can encourage roleplaying by a lack of tactical options, or by mechanically rewarding roleplaying.
Games like FATE or ICONS lack an abundance of crunchy tactical options. Everything is more free form, so there is less calculation involved when a player decides what to do. This leaves more room for the player's choices to be based on intuition, theme, or roleplaying concerns.
The opposite of FATE would be a game where each situation, after careful analysis, has one choice that is mechanically superior to all others, e.g. a situation where you'd always want to roll Intimidate instead of Diplomacy, which leaves less room in the decision making process for things like theme and character background.
There has been a trend in games to reward roleplaying with mechanical benefits, like XP, inspiration, hero points, etc. This is popular because it helps bridge the gap between powergamers (who are here to fuckin' win) and roleplayers (who are here to explore character and story and stuff). And it helps us, too, the people in the middle who want to both save the princess AND act as in-character as possible while doing it. It helps pad out the difference between choosing the most mechanically useful option and the most in-character option, by making the most in-character option more mechanically appealing. (Sometimes, anyway.) I've seen some GMs only award Inspiration when they chose something that wasn't tactically ideal, and that seems like an appropriate way to use it.
+Jack Mack wrote a pretty good essay about this same thing over here, with a somewhat different conclusion.
But on the other hand, it's possible to extend these rewards to make roleplaying (and by extension, social skills) all about the numbers on your character sheet. This leads to the rollplaying vs. roleplaying scab that people pick at every once in a while.
If people are only roleplaying because there are mechanical benefits to it, is it still roleplaying? Well, sometimes. Some players are doing what they think their character would do, and are enjoying mechanical benefits to that process. Other players see roleplaying just as another system to be engineered, and so will munchkin their way through every social encounter.
One of the reason that munchkins get so much hate is because they're interested in victory. It's like a chess game, where Black is interested in roleplaying and White is interested in victory. White will win every time, and be confused/angry as to why Black played so poorly. Black will be confused/angry as to why White kept shitting all over the story, because chess isn't about winning--it's about telling a story.
I'm not going to anything beyond that, but I just want to say that roleplaying tends to come out in social interaction, not during combat. A prince and a dirty bandit might both fight with the same type of swords in the same type of way, and no one cares about that. But we expect them to behave very differently in social situations, which is why any social mechanics need to be considered very carefully.
"Martial Artists Shouldn't Get a Bonus to Combat, so Why Should Articulate Players Get a Bonus to Roleplaying?"
Here's what I tell new players before I DM for them.
I don't care if you do funny voices. I don't care if you went to acting school. I don't care if you give hilarious speeches. Those things are all awesome and make the game more fun, but they won't help you when you're trying to seduce a dragon.
All I care about is (a) what your character is saying, and (b) how they are saying it. You can say: "I flatter the dragon and then ask the dragon if has a mate" OR you can say "Glorious exemplar of dragon-kind, your cavern is truly magnificent and more splendid than any mortal king's throne room, and yet your greatness exceeds even it! This humble human cannot help but wonder if this is the chamber of you and your mate, or yourself alone?"
To me as the GM, both of these approaches will be handled identically. I figure out if the dragon responds well to flattery and how it feels about being asked about its romantic situation. Usually I have notes that make this less of a "DM whimsy" and more of a "deterministic result of a previously-established facts". Then, based on that result, I will decide if the question automatically succeeds, automatically fails, or requires a Charisma roll (with or without a modifier).
I run social encounters this way because I believe that social skills are one of the things that DnD should test (which are distinct from acting ability). Empathy; recognizing pride, fear, and love in other creatures; learning how to exploit those emotions; figuring out when to be humble and when to be ostentatious; etc.
I also think DnD should test creativity by challenging players to solve problems with no obvious solution (i.e. good ol'-fashioned combat). I also think that DnD should challenge people's knowledge of the real world, but only a little bit. Like knowing that lamp oil floats and is flammable. I also think that DnD should challenge mechanical analysis of the situation; this involves math and tactics, and includes (but is not limited to) good ol'-fashioned combat. There's lots of ways that player skills can be expressed in a game.
Where does player skill come from in a roleplaying game? (Skill as that stuff that leads to victory/goals, as opposed to the stuff that leads to fun. Sometimes throwing a pie at the king is hella fun, but it rarely leads to victory.)
- Book knowledge of the rules? (e.g. knowing that flesh golems aren't undead, or that you get -4 to hit if prone)
- Real world knowledge? (e.g. lamp oil floats and is flammable)
- Mechanical analysis of the situation (i.e. stuff a chess computer could do)?
- Creativity? (i.e. thinking outside of character sheets and explicit mechanics?)
- Social skills (e.g. learning motives, interpreting social cues)?
You'll notice that roleplaying isn't on the list. This is because it's usually something that a player does for fun (not related to victory) or it's mechanically rewarded, and therefore part of the mechanical analysis bit. (Social skills are only part of roleplaying. A paladin choosing to abandon his god can be done for roleplaying reasons, but doesn't involve any talking.)
I'll be the first to admit that there is some overlap here between the last three things. Calculating the damage that burning oil would do to friends/foes and the subsequent chance of death is mechanical analysis. Thinking to go back for a barrel of oil is creativity, since the player is thinking outside of the character sheet. And getting the oil imp to give you his barrel quickly is roleplaying, since it depends on talking and understanding the imp's motivations and fears.
Anyway, different games and different DMs will reward/test different skills differently. I don't actually want to analyse this stuff (I want to go eat lunch), so I'll just leave it at that.
A FUN Question For DMs
If you think about DMing as giving a test, what skills would you be grading your players on?
Here's my answer.
Book Knowledge: 0% of the test.
Real World: 10%
Mechanical Analysis: 30%
Social Skills: 30%
The 0% book knowledge is probably the iffiest one. Can I really say that the PCs who've read the book ten times have no advantage over the ones who have never read it? Maybe. Maaaybe. It's a goal of mine, anyway.