Default setting for skipping households on simulation failure

[jpn--]

We are about to implement in #1023 a great new feature: the ability to allow a limited number of failed choices in a model component, which will warn and log errors, and remove failed households from further processing, but not stop a model run. There is some debate over whether the default setting for this feature should be True or False. How we implement this has been solved by @ActivitySim/engineering but whether the default setting should be True or False needs guidance from @ActivitySim/product

if True:

• This great new feature will be active by default.
• Users who previously built models that rely on overflow protection will have results from those models now change.  The most likely scenario here is that overflow protection is masking places where the utility function is trying to make all alternatives unavailable by applying large negative values.  Currently the system is forcing a choice on those cases, typically an alternative with the smallest large negative penalty.  This will now manifest as a failed choice.
• Some models will drop households. This could really confuse folks and cause outputs to not make sense. Imagine, for example, a planning practitioner is trying to run a parking cost scenario -- they update the parking costs but miss a zone, that zone propagates NAs, and all of a sudden, they have 10% less households in the output file than the input file. Run says it succeeded and they think all is groovy. But their revenue forecast is now off by 10% and they don't even know!

if False:

• ActivitySim will give the same results as previously unless the user takes the initiative to turn on the new feature.
• Since users will need to take action to enable this feature, they perhaps will be more informed about the consequences.

[DavidOry]

@jpn--
Is there currently a switch to turn the overflow protection on/off? If so, what is the default for that?

This approach to allowing model runs to continue with errors present strikes me as better than the overflow protection. So in the long run it would be good to remove the overflow protection. Is that on the roadmap? (Either literal or in our heads?)

Do we have any sense as to the number of users who are currently relying on overflow protection and the scale of the problems this is covering over?

In your hypothetical, you say turning this feature on could result in 10 percent fewer households. If that's true, doesn't that mean there's some error now being covered up by overflow protection that is resulting in bad forecasts? It's not obvious to me that the bad forecasts being covered up by the overflow protection are worse that the forecast being "off by 10 percent", as in your hypothetical. Is it just worse because the current error is the status quo?

Your hypothetical also implies that ActivitySim is not taking any action to re-scale the households up, either via replication of successful records or by returning an updated sample size parameter to the user. Why not?

My preference would be:

1. Help users solve their problems that is causing the overflow. Given the CT-RAMP1 design, this should be straightforward, shouldn't it? Do we have an example where these errors are not clear and obvious?
2. Remove the overflow protection functionality
3. Set the default for this feature to True
4. Have the software return an updated sampling rate parameter that users can use to scale their trip tables
5. Allow the user to set a threshold of dropped households that triggers the model run to fail (this may already be a feature)

[i-am-sijia]

@DavidOry 1,3,4,5 of your preference are implemented along with the new feature. Under this feature, ActivitySim prints out debug trace files for the failed choice. It updates the sampling rate in the output household file and scales the trip tables. It allows uses to set a threshold of dropped households in the settings.yaml.

I responded in the PR and I'm just copy it here:

1. Enabling skip_failed_choices could cause existing model run results to change, because it would surface choice failures that are currently masked by the overflow_protection feature. I recognize the value of preserving agencies' existing implementations.
2. When skip_failed_choices is set to False, the 10% households in the hypothetical still make a choice because overflow_protection kicks in and masks the failure. Those households will very likely not be responsive to parking cost changes. The practitioner wouldn't know about this behavior either, and may be confused about why the model appears insensitive. This is a tradeoff worth considering.
3. It is right that the output household is fewer than the input, but that's the intended design, just like how ActivitySim runs on a sample of households. This is reflected in the adjusted sample_rate column in the output households, and the resulting trip tables and matrices are scaled up accordingly.

[jpn--]

The current default for overflow protection is "True". It is switchable, and as implemented in the PR you cannot have both overflow protection and failed household skipping active simultaneously.

The PR for skipping households does in fact upscale sample weights to account for skipped households, so that the total weight remains the same regardless of skipping. Obviously if the skipping is systematic and significant there will be bias created, but that is unavoidable (other than by fixing the model to have less skipping).

I think everyone agrees that significant amounts of fail households are bad. But also having a model run just stop midstream is bad. The consensus view is that having the model run just stop is the worse outcome, that's why we keep building solutions here. We do want to still have the model stop if there are "too many" problems, as discussed e.g. on this call.

We are smoothing over problems by (a) forcing some choice via overflow/underflow protection, or (b) skipping the household completely. Who is to say which one is worse? (Product, that's who.)

In my personal opinion, I would argue that we should make skipping failed households the default, but also the default thresholds for failing a model run should be quite conservative. The current default in the PR is 10% but I think maybe it should be 0.1%, i.e. allowing failures on the order of only 1 in 1,000 households at most. More than this is reflective of a fundamental model deficiency that should be corrected.