When designing a dungeon, one question you should ask yourself is: What am I testing for?
Math tests challenge your math skills. Drinking contests challenge your liver and your brain. But what does your dungeon test?
How is the wheat separated from the chaff?
If you write a test that everyone succeeds at, you are merely writing a dungeon tour. The party walks through, witnesses all this cool shit, and has the same experience no matter how skilled or unskilled the players are. (Digression: I know at least one group that enjoys dungeons like this. Nothing wrong with that; they're having fun.)
If you write a test that everyone fails at, you are merely writing a doom cave. Rocks fall. Everyone dies. If no one, no matter how skilled or lucky, can progress through your dungeon, you've written a dead end.
So a good dungeon must lie somewhere in the middle. But what is the mechanism that sorts skilled groups from unskilled groups?
Does it test aggregate weight? Does the first room weigh the entire party, and if they weigh less than 1000 lbs, they may walk across to the treasure vault, while heavier parties fall down the pit straight to hell?
Does it test the character's combat skills? There are different kinds of combat tests. Does it test their character builds? I.e. did they build a strong enough character, numerically and mechanically, with enough synergy and clever combos that they can wear down difficult foes? Does it test their levels? Does it test their tactics, with lots of fights that can be deadly unless the party retreats to a choke point? Or the party's insight, where they must realize that the shrine guardian can only be killed when it is outside of the shrine?
Does it test their resourcefulness when trying to solve OSR-style problemshttp://goblinpunch.blogspot.com/2016/03/1d135-osr-style-challenges.html? Must they solve riddles? Or find hidden doors? Do they use inductive logic to solve things, such as realizing that the cure for the fire snake's venom is the ice snake's venom?
Does it test their role-playing abilities? Does it, god forbid, test a player's acting ability? Does it test the parties ability to discern the motives of NPCs? Does it test their discretion, in choosing which NPCs to trust? Does it test their ability for subterfuge and lies? Does it test their scheming ability?
Does it test system mastery, where players have an advantage if they know protection from evil also protects against possession? Does it test setting knowlege, where a player has an advantage if they realize that the guy in a mantis mask is a high level assassin?
Does it test luck? If you use dice, you are testing luck, but to what degree?
You probably read through this list and shook your head at some entries, and nodded approvingly at others. That is good. You are thinking about it.
I have seen dungeons test all of these things, to large and to small degrees. Suffice to say, they are all adjustable. An organic chemistry exam can have 60% of its questions test redox reaction, or it can have 0%.
For example, if you write a dreamlands dungeon where a character's ability scores are all mirrors of their Charisma score, and all of their abilities are replaced with randomized dream abilities, you have negated any influence character build has on that player's success or failure.
And there is no right answer. Some groups will love one thing; others will loathe the same.
Narrow and Broad Tests
When you have an obstacle, how many ways are there to overcome it? Combat, the quintessential obstacle, usually has a bunch of ways: diplomacy, fighting, bribery, running away, treachery, etc. There are many paths to success in a broad obstacle.
Some tests are narrow. A door that can only be opened with the crystal scepter. A hidden room that can only be found with magic, not ingenuity. There are few paths to success in a narrow obstacle.
Generally, I like things to have at least 3 solutions. A locked door is good example. It has three or four solutions. It tests a character's build, by allowing a thief to pick it. It tests a party's willingness to risk a random encounter, by allowing anyone to kick it down noisily. It tests a party's thoroughness, by allowing them to find the key in room 22 and open it. It tests a character's build, by allowing the wizard to cast knock on it.
Personal philosophy: if you are going to trap the party in a dungeon, give them at least 3 ways out. Don't make any of them easy.
Rule of Thumb: Don't use narrow tests.
At their worst, they're just pixel-bitching. Narrow tests are cool only when they are optional. A clue that only be found if all of the spider webs are burned away. A small treasure vault that can only be opened if the party answers a riddle correctly.
There is a wrong way to use riddles, and it involves putting the riddle in the middle of main path through the dungeon. If the party can't answer the riddle, they can't progress in the dungeon. Shitty.
If you are going to have a riddle block a main path of the dungeon, there should be other ways around it (it should be a broad test). If a sphinx blocks a path, you should be able to kill it, bribe it, or go around it.
Alternatively, you can use a riddle to block off a small side-branch of the dungeon, as mentioned above. Maybe just an alcove.
Pro-tip: What to do when the smartest player is playing the dumbest character? You obviously can't have the Int 5 guy answer all the riddles. Just transmigrate the answer over to the Int 18 wizard, and let it come out of his mouth. Fiction is preserved, the smart player gets to have fun being smart at the table, and honestly it probably takes a table full of guys to simulate Int 18.
Discriminatory Ability and Secret Areas
Remember when I said a test that fails everyone or passes everyone can't discriminate between skilled parties and incompetent ones? That's still true.
If you want to have a test that discriminates between skill and incompetent, you need to have challenges with a variable difficulty.
Rule of thumb: Some treasure should be easy to find. Some treasure should be moderately difficult to find. And some treasure should be damn near impossible to find.
The reason for this is to test (and therefore reward) players who search.
If the group just runs through the dungeon, they'll get the obvious treasure, just sitting out in the open. Maybe they're low on health, or maybe they're just careless.
If the group is clever or thorough, they'll find more stuff. Chests hidden under beds, keys in the pockets of dead guards, secret passages hidden behind easily-moved bookshelves, etc.
If the group is clever and tenacious, they'll find everything, including the really hard to find stuff. Treasure at the bottom of the outhouse. Secret alcove that can only be discovered if you topple the statue of Nyarlathotep (which takes time, and makes noise). Some parties won't be able to do this, because they're low on HP, or time, or too low-level to risk more wandering encounters.
The reason I write my dungeons like this is because I want to reward cleverness and tenacity. With treasure.
If finding all of the treasure in your dungeon requires a moderate effort (not automatic, not extremely difficult), they you might not be offering enough rewards to players who run quickly through your dungeon (perhaps they're dumb, perhaps they're too low-level) and you might not be offering enough rewards to players who are both clever and tenacious.
Two Rabbit Tattoos Talk About Hidden Rooms
RRT: It sounds like you're advocating putting treasure in rooms that most parties won't ever find.
RRT: Why waste time creating content that most players will never get to enjoy? Surely, it is better to spend your time creating treasure and rooms that everyone will get to enjoy. Take those impossible-to-find treasure vaults and stick them somewhere obvious, like on the main dungeon path.
LRT: I think one thing about old-school play that a lot of newcomers don't grok is that there is a lot more emphasis on exploration, and less emphasis on straight-up combat.
When you do well in combat, you survive with less damage, or none at all. There's a variable degree of success in it.
When you do well in exploration, you should find more treasure. There should be a variable degree of success in that, as well.
RRT: Do what you want, but be aware that the DMs who follow your advice will write up 10 rooms and their players will only find 8 of them, on average. That can be frustrating for a DM.
LRT: You know what? I'm okay with that. It's more important to me that my dungeon is a more discriminatory test of exploration ability.