Failure rate clustering at high completion percentages

[pehamm]

Not sure if this belongs here, but I thought when in doubt best put it here. I did not want to open a PR before hearing some input from others how to best handle this issue as the possible approaches are very different depending on what was intended in the first place. This also interferes with how astronomically large some failure probabilities are after #48673, and changes to the success formula may necessitate changes to the solution here.

One thing I noticed during the discussion on failure rates in came with the updated failure rates in #48673 was that crafting failures clusters at high completion percentages due to how failure points are set. For illustration, this is the progress a debug character made crafting a leather funnel with 8 intelligence, 6 tailoring, 1 survival, missing principles of leatherworking proficiency:

Failure Point	Progress Loss
48%	6%
67%	13%
84%	15%
95%	12%
90%	14%
88%	10%
92%	11%
94%	12%
92%	24%
85%	14%
83%	16%
85%	14%
84%	11%
87%	10%

On the level of code, the game works by setting a failure point (roughly normally distributed) on the progress counter. Once that point is reached, it rolls from the same probability distribution for the remaining progress. That means, failures are not equally distributed, starting at 0% has the same probability to success without failure as starting at 99% after a crafting failure. The failure probability per progress percentage is not constant but dependent on the last failure point, which is extremely unintuitive.

From the perspective of the player, this is frustrating as it feels possible to complete the craft when starting out, but suddenly we fail every few percentage point while feeling close to the goal. This also means that to practice proficiencies, it is optimal to start a new crafting object after a failure to maximize crafting time while controlling for ressource destruction. It also seems unrealistic that we fail more towards the end for most crafts. Usually whenever I try to build something, I mess up catastrophically in the middle and not when putting on the finishing touches.

The simplest solution would be to no longer scale the next failure point with the percentage left until completion. This ensures that the failure probability is the same regardless of when we last failed. This would have the side effect of boosting success probabilities massively for low-probability crafts compared to how they are now.

A cleaner solution would be to roll failure points or number of failures up front, but to do this I would need to know how the probabilities were supposed to be in the first place, or if #48673 is indeed correct.

Thankful for any input. I considered just removing the scaling which is a one-line fix, but I do not want to break anything without a second opinion after the effects from 48673 were so massive.

[actual-nh]

Ping: @eltank, @kevingranade? EDIT: You may wish to change the title to "Failure rate clustering at high completion percentages", BTW.

[pehamm]

You are right, thanks!

[eltank]

I don't have much to add other than yes, I have observed this as well. In the current state of the game when grinding proficiencies it is usually more efficient (in terms of material and time) to give up on failure, throw away the in-progress craft and start again from scratch.

I'm willing to volunteer my time to implement a better crafting failure mechanism in the C++ code but I don't feel qualified to design (or review a proposed design) on my own because I don't fully understand the intended behavior or the game balance requirements. I'm hoping that can be done by the core devs or with their guidance.

[eltank]

The current crafting failure algorithm was implemented in #30460 (2 years ago) as part of the #29210 crafting overhaul. There is some additional context to be found there.

[eltank]

The dice roll formula for crafting success is at least 8 years old:

https://github.com/CleverRaven/Cataclysm-DDA/blame/7f58d558e332caa6739e8109020b7e9bd54d9f3f/src/crafting.cpp

4*skill dice with 16+int sides vs 4*difficulty dice with 24 sides

But back then (and until #30460) it was an all-or-nothing one time check on craft completion.

Perhaps this should be the "overall success probability" that we should aim to preserve.

[eltank]

Based on my read of the code and its history I would agree that the issue described here stems from scaling the next success roll after a failure, specifically this line in crafting.cpp:

const int failure_point_delta = crafter.crafting_success_roll( get_making() ) * percent_left;

The closer a player gets to the end the less progress they can make per skill-check attempt.

However, prior to #30460 the only alternative to failure was to redo the craft from scratch. Now, you can pay a fraction of the cost and spend a fraction of the time to try again -- but you must still roll a skill/difficulty dice ratio >= 1 to succeed. It's still much better than it used to be.

One could argue that the clustering of failures close to the end is a natural consequence of requiring a passing skill check in order to actually finish, and that this is "working as intended". That's something the core devs should decide, but the question can be expressed in the form of a PR (which can either be merged or not). The PR could be as simple as removing * percent_left from the formula, or could be more complicated by also adjusting other aspects such as the formula for lost progress. But it would need justification from someone who has done the simulations.
@pehamm, do you think you could make that PR yourself?

[pehamm]

Thanks for the dive in the code history! I did some checks yesterday and removing the scaling by percent_left indeed fixes the egregious clustering (we still have to multiply by 10000000 or we fail instantly). The adjustment for lost progress is a simple exponential distribution that scales with progress, it works just as well with the new formula. I think your approach is correct, if the core devs give additional feedback or just express their intentions and ideas I can see whether I can put them into distributions. Maybe my advanced stochastics courses in grad school were not in vain after all^^

I will make the PR, thank you for your help!

[pehamm]

Looks like there already a rewrite in the works in #46153, just on hold until i-am-erk comes back.

[Anaglyphic]

I love the graph work. Since so many smart people are here.

> 4skill dice with 16+int sides vs 4difficulty dice with 24 sides

So many have worked on so much in the last ~3 years I feel like a jerk offering stuff you didn't ask for, still I'm going to.

Why not an old-fashioned lookup table roll, AD&D style?

Then instead of constant variables value-checked in how much you % work you lose, it's a table value that's pulled from threshold ranges? Then say you percentile RNG every so often, we'll say 10 times per item, every 10% for this example - that's why the the standard lookup table method is "better" (well), and you can just pass threshold modifiers as you will, each time, not hard-coded...

Anyway, the fictitious 'making' call, weighted by proficiencies mastery and overall stats:

99 - nn | InAGroove! - Exceed expectation, +10% production, skip next check. 
nn - nn | Success -  continue.
nn - nn | Delay - no progress, no loss
nn - nn | Setback - 10% progress lost
nn - nn | Frustration - no progress, increased exhaustion +2hr worth
nn - nn | Fail - damage object being created 1 level
nn - nn | Hard Fail - damage tool 1 level
nn - nn | Nasty Fail - break object, lose components, start over
nn - nn | Oh Shit - broke important tool, work stops
nn - nn | Catastrophe! - break object+ 1-3  tools, cannot build again <24h, no morale
nn - 00 | FUMBLE! - Lose all object / components, 25% health loss to 2 limbs, heavy bleeding, broken bone

There would be no clustering possibilities as the weighting values for success or failure would just shift the threshold points more likely <> less likely.

Hey why not actual injury, even chance of death? If you are doing something stupid and dangerous that's ## levels above your skill(s) (maybe additive/cumulative?) and it's a weapon, explosive, fuel-related, etc? Welding on a gas tank with fuel in it? OK but if you roll a 00 and fumble out of a percentile roll, and a 2nd fumble for punishment calc? Well dummy, you ignite the fuel, your vehicle, yourself, likely lose the vehicle if you don't have an extinguisher handy, maybe a section of woods or a building/block if you're against flammable stuff, xx% of items stored in the trunk or cargo storage, or all of it, and if it burns a nearby NPC they'll open fire on you as though you attacked them. Fun!

"it's a thought" Anaglyphic says, quite frequently.

[pehamm]

The clustering would still happen regardless of this, just the visible effects would change, unless you rewrite the whole code basically from scratch. I believe crafting is too much of a core aspect of the game to throw in these kinds of punishments, because most characters craft often enough to hit a catastrophic outcome eventually even with great skills. These kinds of things are fun the first three times, then become annoying as you craft your 200th item. This sounds more like something from nethack, which follows a very different philosophy. Imo.

[actual-nh]

The balance between "reflect that something can always go wrong" and "don't have routine tasks turn into catastrophes that should only happen once in a million years" can be a difficult one, and it's common in a lot of paper-and-pencil systems (critical failures). The above actually sounds more like something from one of the NetHack variants - NetHack itself has things like 100% resistances being possible, which not all of them do.

[anothersimulacrum]

Crafting failure chances have been overhauled since #63886