One of my players is a demonologist and requested a cantrip related to the field. So I took a crack at it and came up with the following thinking that an attack on the mind was fitting:

Three-Word Curse: Nails

Enchantment cantrip

Casting Time: 1 action

Range: 60 feet

Components: V, M (a fragment of ram horn)

Duration: Concentration, up to 1 minute

You speak in a demonic tongue slowly cursing the mind of a creature you can see within range. If the target can hear you (though it need not understand you), it must succeed a Wisdom saving throw to end the spell. On each of your turns for the duration, you may use your action to have the target attempt the Wisdom saving throw again. The spell ends if you use your action to do anything else. On the third failed attempt by the target, it experiences the feeling of several nails piercing their skull simultaneously and takes 5d8 psychic damage, and the spell then ends.

The spell's damage increases by 1d8 when you reach 5th level (6d8), 11th level (7d8), and 17th level (8d8).

Does the risk of wasting the invested time balance the high damage reward?

The risk/cost is too high

The cost is:

• Concentration


• Three Actions

This is a rather high investment to start with, but there is also the risk:

• The target needs to fail 3 saves


• These have to be consecutive
  

Below level 5 the damage potential of this spell (5d8) is higher than other cantrips (3d12 with toll the dead is the next), but on higher levels it seriously starts to lag behind. Other cantrips also get +1dX, but they can be used 3x as often.

Add to this the aforementioned risk. Requiring 3 saves already gives this a low chance to succeed, but to achieve that on 3 consecutive rolls plunges the chance of success even deeper. The effective time/action investment will be way over 3 because of needing to restart on any successful save. This spell will extremely rarely do anything, especially considering that most combats do not last for even 10 turns (usual is 5-8 in my experience).

All in all this spell is so risky I would dare to call it near useless and the damage progression is miscalculated.

Recommendations

If you want to preserve the "three word" aspect, imitate spells like contagion and do not require the failed saves to be consecutive, but end the spell with 3 successful ones (also not necessarily consecutive). A mechanic resembling death saves. And either do not require any further action investment (like other such spells) or make it a bonus action. Also, I would advise you to make it a higher level spell, not a cantrip.

If you want to just make a more thematically appropriate cantrip, reskinning an existing one would be much easier to do and to balance.

The other answers give a good picture on why the spell is, from a mechanical perspective, too weak. I will touch on that too, but also a couple of game design points that might help yiu understand why this spell won't work very well.

From a purely mechanical point of view, "succeed on more rolls to inflict more damage" is a workable concept. The expected damage output should be very high compared to normal cantrips to offset the fact that for normal cantrips, two successes (for three castings) inflict 2/3 of the damage one would inflict on three hits, while the "all-or-nothing" approach of the three word curse would inflict no damage unless all rolls succeeded. (Or failed, from the saving creature's POV)

However, mechanical balance isn't enough to make a cantrip good. I forecast problems relating to the use of this spell in tables.

It should be used early...

Suppose you're fighting a group of nasties and choose to use TWC. Three rounds later, you may have a chance to deal 5d8 damage ...if the combat is still going on. Many encounters last for a surprisingly small number of rounds, so TWC: Nails would probably find itself used mostly as an opener, because it might not get a chance to fire later on and 5d8 against an enemy weakened by the action already would be more likely to be an overkill.

...but early turns are too valuable to waste

Removing enemies from the encounter during the first turns is very valuable in DnD, strategically, because every removed enemy shifts the action economy to favor the PCs. Removing enemies during the first turns is something TWC: Nails can't do, but since it has little use later in the combat too, it's an inadvisable spell to use regardless.

Inaction is boring

This is the worst part to me. TWC: Nails will have the player essentially skipping three turns in order to have a chance to deal relatively massive damage. Assuming the spell dealt enough damage to be mechanically appealing, you'd be promoting an extremely boring style of gameplay to your players --- don't do that. As a general rule, make fun choices strong and strong choices fun, so players don't feel tempted to min-max themselves into something boring.

Poor party synergy

Especially assuming the cantrip gets the damage buff to make it mechanically competitive, another problem sets in --- it's overkill against many types of enemies. To avoid wasting that sweet damage, it encourages the players to not target the enemy targeted with TWC. Reserving enemies for oneself gets stale fast and can cause a lot of tension among the players, and can also result in very swingy outcomes: if the spell fails, you're gonna wish you had put in some damage instead of trusting the dice.

Bookkeeping

The spell forces an additional bookkeeping need for the players or the GM. It's a minor issue, but still one more thing I'd change.

Overall

The spell seems like something that might be a workable concept in a computer game where turns can be executed in mere seconds. However, in DnD, I think even the part where you have to idle during combat to have a chance to deal damage is unacceptable. The spell's concept is neat, but ultimately just a gimmick that can't be the sole justification of its existence.

If you want to preserve the "three strikes", I recommend the following changes:
- make the spell deal some damage on every failed save
- don't require the saves to be consecutive
- the third failed save deals extra damage
- no concentration needed

It's still not my favorite kind of spell, but it's more flexible in its use than before and far less swingy.

Way too weak.

Most foes will be dead before it does anything. And even if the target fails every save, it is 3 actions for 22.5 damage or 7.5 damage/action.

Unlike most attacks, you have to hit 3 times for it to do anything.

If the foe fails 60% of the time that is a 21.6% chance it does anything. If they fail 80% of the time, your chance of it doing damage is about 50-50. This is horrible.

3 turn delay, horrible accuracy, mediocre effect, bad scaling.

Three-Word Curse: Nails

Enchantment cantrip

Casting Time: 1 action

Range: 60 feet

Components: V, M (a fragment of ram horn)

Duration: Concentration, up to 1 minute

You speak a word in a demonic tongue slowly cursing the mind of a creature you can see within range who can hear you. The target creature must make a Wisdom saving throw or suffer the curse of this spell. Creatures under the curse feel nails surrounding their skull in all directions.

Whenever a creature fails any saving throw against this spell, they suffer disadvantage on their next attack before the start of your next turn.

While a creature is under the effects of this curse, you may spend an action to utter the second or third word of the curse.

When the 2nd word is uttered, the creature must make a Wisdom saving throw or suffer 1d8 psychic damage as they feel nails being tapped into their skull on all sides. Passing a save against the 2nd word does not end the spell; you may try the 2nd word again.

After the 3rd word is uttered the curse ends. The target must make a Wisdom save or suffer 4d8 psychic damage and feel nails slam into their skull from all sides.

You may only utter the 3rd word after the cresture under thr curse has failed a save against the 2nd word during this curse.

The spell's damage increases by 1d8 when you reach 5th level, 11th and 17th level. This applies to the damage from both the 2nd and 3rd word.

I did a few things.

1. I added disadvantage on attacks. Like viscious mockery.




2. I moved some damage up to the 2nd curse word.




3. I doubled the rate of damage scaling (by having it apply twice).




4. You can now utter the first word of the curse and hold it in reserve with concentration.




5. Succeeding in a save against the 2nd word doesn't remove the curse. The caster can try again and again until it takes hold.

The curse now has an effect right away (disadvantage). If you hit "twice" you get damage similar to Viscious Mockery, and 3 times to deal decent damage.

Ignoring concentration, this ability is now stronger than Viscious Mockery. Add in concentration and it looks more reasonable.

Probability

Your proposed mechanism requires 3 consecutive failed saves starting with the first. As such, the 1 minute duration is irrelevant - it will all be over in at most 3 rounds.

D&D aims to have a "hit rate" (including failed saving throws) of about 60%. assuming that your target has this chance of failing their saving throw, chaining 3 of these together gives a 21.6% success rate.

In addition, you are proposing that this requires concentration. So, unlike every other damaging cantrip it is not certain that it will "go off" and it means you can't be concentrating on anything else.

Damaging cantrips

Damaging cantrips are for a spellcaster what making an attack is for a martial class - the default action to take when you have no better options. Specifically:

• They don't use up resources (e.g. spell slots)


• They use up 1 action


• They are all or nothing in terms of damage


• They scale with character level, either by increasing in damage or adding additional targets

Saving throw cantrips

There are 4 cantrips in the PHB that have a saving throw:

$$begin{array}{|c|c|c|c|c|c|}
hline
textbf{Spell}& textbf{Range} & textbf{Damage} & textbf{Expected*} & textbf{Save} &textbf{Extra} \ hline
textit{Acid Splash} & text{60} & text{1d6 acid} & 2.1text{ or }4.2 & text{Dex} & text{Up to 2 targets within 5 feet} \ hline
textit{Poison Spray} & text{10} & text{1d12 poison} & 3.9 & text{Con} & text{} \ hline
textit{Sacred Flame} & text{60} & text{1d8 radiant} & 2.7 & text{Dex} & text{} \ hline
textit{Vicious Mockery} & text{60} & text{1d4 psychic} & 1.5 & text{Wis} & text{Disadvantage on next attack roll} \ hline
textit{TWC: Nails} & text{60} & text{5d8 psychic} & 1.62 & text{Wis} & text{Takes 3 rounds w/- concentration} \ hline
end{array}$$

• Expected damage assuming the target needs to roll an 13 or better to save (60% success rate) and dividing by 3 for your cantrip because it takes 3 rounds.

So, given these comparisons, would you chose this cantrip? By the way, this is at levels 1-4 - the spell only gets worse after than in comparison.

Use one of these as a model

Make it the same as Sacred Flame only with psychic damage and your flavour text.

Or reduce the range and do more damage like Poison Spray or reduce the damage and allow it to affect more people like Acid Splash or reduce the damage and give it an effect like Vicious Mockery.

Whatever you do, you need to drop the multi-round nature of the effect - that's not how cantrips work.