Getting Started with E2P Simulator

How to use E2P Simulator

E2P Simulator is designed to be intuitive and interactive. You can explore different scenarios by adjusting effect sizes, measurement reliability, base rates, and decision threshold, and immediately see how these changes impact predictive performance through various visualizations and metrics. Still, in this section we will highlight and clarify some of the key features and assumptions of the simulator.

The image above provides an overview of all E2P Simulator's interactive components.

Binary vs. Continuous Outcomes

E2P Simulator provides two analysis modes that cover the two most common research scenarios:

• Binary Mode: Considers dichotomous outcomes such as diagnostic categories (e.g., cases vs. controls) or discrete states (e.g., success vs. failure). All metric calculations and conversions in this mode are completely analytical and follow the formulas provided on the page.
• Continuous Mode: Considers continuous measurements such as symptom severity or performance scores that may need to be categorized (e.g., responders vs. non-responders or performers vs. non-performers) for practical decisions. This mode is based on actual data simulations rather than analytical solutions, hence it may be slower to react.

Measurement Reliability and True vs. Observed Effects

Measurement reliability attenuates observed effect sizes, which in turn reduces predictive performance. The simulator allows you to specify reliability using the Intraclass Correlation Coefficient (ICC) for continuous measurements and Cohen's kappa (κ) for binary classifications. These typically correspond to test-retest reliability and inter-rater reliability, respectively. You can toggle between "true" effect sizes (what would be observed with perfect measurement) and "observed" effect sizes (what we actually see given imperfect reliability); see Karvelis & Diaconescu, 2025 for more details on how reliability attenuates individual and group differences.

Note that the simulator does not account for sample size limitations, which can introduce additional uncertainty around the true effect size through sampling error.

Base Rates

Base rate (or prevalence) refers to the proportion of individuals in the population who have the outcome of interest before considering any predictors or test results (in Bayesian terms, this is the prior probability of the outcome). To estimate real-world predictive utility, the base rate should be set to reflect the population where your predictor or model will actually be used — not the composition of your study sample. This distinction is crucial because research studies often use case-control designs with balanced sampling (e.g., 50% cases, 50% controls) that do not reflect real-world prevalence. This is one of the most commonly overlooked problems in evaluating prediction models (Barbec et al., 2020), as the base rate directly affects multiple metrics used for model evaluation (see Understanding Predictive Metrics).

For instance, if you are developing a model for a rare disorder that affects 2% of the general population, the base rate should be set to 2%, even if your training dataset contains equal numbers of cases and controls. However, if your model will be used in a pre-screened high-risk population where the disorder prevalence is 20%, then 20% becomes the relevant base rate (however, in this scenario, the effect size should also reflect the difference between cases and high-risk controls rather than general population controls).

Multivariable Simulators

Both binary and continuous outcomes analysis modes include simulators that help estimate how many predictors need to be combined to achieve a desired level of real-world predictive utility. The main metric for this is PR-AUC - it accounts for the base rate and is threshold-independent. For binary classification, the simulator also displays Mahalanobis D, a multivariate generalization of Cohen's d, and for continuous outcomes, it displays total variance explained R².

The multivariable simulators can help approximate the expected performance of multivariate models without having to train the full models and thus help with research planning and model development. They also help gain intuition about how multicollinearity undermines predictive performance and leads to diminishing returns when adding more predictors.

Assumptions and Limitations

The multivariable simulators are based on several simplifying assumptions:

• Average effects and correlations: The simulators use single values to represent the average effect size across predictors and average correlation among them, which can provide useful approximations even when individual predictors vary in strength
• Linear effects: The formulas assume predictors contribute additively without interactions (where one predictor's effect depends on another). This assumption is supported by research showing that in clinical prediction, complex non-linear models generally do not outperform simple linear logistic regression (Christodoulou et al., 2019). In general, more complex machine learning models excel at capturing non-linear relationships that we expect to see in the real world, but they also require more data and are more prone to overfitting.
• Normality: The underlying variables are assumed to be normally distributed. This is consistent with the assumptions of input metrics like Cohen's d and Pearson's r, although in practice these are often computed despite normality violations.

Even though real-world predictors will often not be normally distributed and will vary in their individual strengths and collinearity, the general trends (such as diminishing returns and the impact of shared variance among the predictors) remain informative for understanding multivariate relationships and estimating expected model performance.

Understanding Predictive Metrics

Classification Outcomes and Metrics

When using a predictor to classify cases into two groups, there are four possible outcomes: True Positives (TP), False Positives (FP), False Negatives (FN), and True Negatives (TN). These form the basis for all predictive metrics.

The image above illustrates how these four outcomes relate to various classification metrics. On the left, you can see how negative (e.g., controls) and positive (e.g., cases) distributions overlap and how a classification threshold (red line) creates these four outcomes. On the right, you'll find the confusion matrix and the formulas for key metrics derived from it. Note how some metrics have multiple names (e.g., Sensitivity/Recall/TPR, Precision/PPV) - this reflects how the same concepts are referred to differently across fields like medicine, cognitive science, and machine learning.

Threshold-Independent Metrics

Some metrics evaluate performance across all possible thresholds and can serve as a better summary of the overall model performance. These include:

• ROC-AUC (Area Under the Receiver Operating Characteristic Curve): Visualizes how well a model balances true positives (Sensitivity) and false positives (1-Specificity) across all possible thresholds, capturing the trade-off between the two. An ROC-AUC of 0.5 means the model is no better than flipping a coin, while 1.0 means perfect separation between groups.
• PR-AUC (Area Under the Precision-Recall Curve): Shows how well a model can maintain both precision (PPV) and Recall (Sensitivity) together, which is particularly informative in the context of rare outcomes, as the base rate affects Precision. A larger PR-AUC means the model can achieve high Precision without sacrificing Recall (or vice versa), indicating a smaller trade-off between finding all positive cases and avoiding false alarms.

Both ROC-AUC and PR-AUC represent areas under their respective curves and are mathematically expressed as integrals:

These integrals are computed using trapezoidal numerical integration.

Decision Curve Analysis (DCA)

Decision Curve Analysis (Vickers & Elkin, 2006) evaluates the clinical utility of a predictive model or a single predictor by explicitly balancing the costs of false positives against the benefits of true positives.

A DCA plot typically includes three key curves:

• Model: Shows the net benefit of using the model (or a single predictor) at different threshold probabilities.
• All: Represents the strategy of intervening for everyone regardless of their predicted risk. This strategy maximizes sensitivity (no false negatives) but results in many unnecessary interventions (false positives).
• None: Represents the strategy of intervening for no one, which always yields zero net benefit but avoids all intervention-related harms.

The net benefit formula accounts for both the benefits of true positives and the costs of false positives:

Where N_total is the total sample size, and p_t is the threshold probability — the minimum predicted probability at which you would decide to intervene (e.g., diagnose, treat, or refer for further assessment). For example, if p_t = 0.10, you would intervene for anyone with a predicted risk ≥ 10%. In practical terms, p_t represents the trade-off between benefits and harms: for every 1/(1-p_t) - 1 unnecessary interventions you're willing to accept to prevent one adverse outcome. The ratio p_t/(1-p_t) in the net benefit formula captures this relative weighting of false positives compared to true positives. p_t itself can be calculated as:

Where C_FP is the cost of a false positive (unnecessary intervention) and C_FN is the cost of a false negative (missed positive case). p_t can also be estimated through expert surveys, stakeholder preferences, or established guidelines.

For population screening, p_t is typically set low because missing true cases is costlier than unnecessary follow-ups, so more false positives are acceptable. For diagnostic confirmation (e.g., before initiating high-risk treatment), p_t is set higher to avoid false positives, reflecting a preference for specificity. As a rule of thumb, screening scenarios may use p_t in the 1–10% range, whereas diagnostic decisions often warrant much higher p_t (for example 30–70% or more), depending on harms and preferences.

What we often want to know is not the absolute NB, but added value. ΔNB (Delta Net Benefit) measures this additional utility by comparing the model against the better of the two simple strategies (either All or None) at a specific threshold probability:

At each threshold probability, the model's net benefit is compared against whichever simple strategy performs better at that threshold. This provides a more conservative and meaningful assessment of the model's added value. A positive ΔNB indicates that the predictive model offers genuine improvement over the best simple strategy, while values near zero suggest that simple strategies may be equally effective.

DCA is particularly valuable because it:

• Incorporates the decision-making context through threshold probabilities that reflect real-world scenarios
• Accounts for the relative costs of different types of errors (false positives vs. false negatives)
• Provides actionable insights about when a model should or should not be used
• Facilitates comparison between different models or strategies across various contexts

For more information about DCA, visit https://mskcc-epi-bio.github.io/decisioncurveanalysis.

Quick Start Examples

To further clarify how the tool can be used and to demonstrate its utility we provide some specific examples.

Example 1: Diagnostic Prediction

Can we predict depression diagnosis using a cognitive biomarker?

1. In the Binary outcome mode:
   • Set base rate to 8% (the prevalence of depression in the population; Shorey et al., 2022)
   • Set the grouping reliability to 0.28 (depression diagnosis reliability based on DSM-5 field trials; Regier et al., 2013)
   • Set the predictor reliability for both groups to 0.6 (an average reliability for cognitive measures; Karvelis et al., 2023)
   • Set the observed effect size to d = 0.8 (a large effect size that is optimistic and rarely seen in practice)
2. This will yield ROC-AUC = 0.71 and PR-AUC = 0.19. Even with the optimistic effect size of 0.8, the predictive utility remains modest, particularly in balancing recall and precision (as shown by the low PR-AUC). The DCA plot also suggests limited clinical utility: assuming a threshold probability (p_t) of 0.1, which would be suitable for a screening scenario, we get ΔNB ≈ 0.018, which means that relying on this biomarker would result in 1.8 additional true positives per 100 patients (after accounting for the harm of false positives at that threshold).
3. Note that with the low reliability values, this observed effect corresponds to a much larger true effect, d = 1.58, and much better predictive utility, ROC-AUC = 0.87, PR-AUC = 0.46, and ΔNB ≈ 0.042. This demonstrates the importance of measurement reliability in predictive modeling.
4. Now let's say we are serious about precision psychiatry and we want to achieve a PR-AUC of 0.8. Using the tool, we can find that it would require d = 2.55. It would be rather unrealistic to expect a single biomarker to achieve this effect size. Using the Mahalanobis D calculator, you can explore how many predictors with smaller d values would be required to achieve D = 2.55.

Explore this example yourself →

Example 2: Treatment Response Prediction

Can we predict who will respond to antidepressant treatment using task-based brain activity measures?

1. Select Continuous outcome mode:
• Set base rate to 15% (the rate of response to antidepressant treatment beyond placebo; Stone et al., 2022)
• Set predictor reliability to 0.4 (an average reliability for task-fMRI measures; Elliott et al., 2020)
• Set outcome reliability to 0.94 (Hamilton Depression Rating Scale (HAMD) reliability; Trajković et al., 2011)
• Adjust effect size such that R² = 0.2 (average multivariate R² from recent research; Karvelis et al., 2022)
2. This will yield AUC = 0.73 and PR-AUC = 0.32, indicating rather modest predictive performance, as shown by the low PR-AUC. At p_t = 0.30, which is a reasonable threshold for accounting for the harms of unnecesarily prescribing antidepressants, we get ΔNB ≈ 0.013. Improving measurement reliability could increase performance up to AUC = 0.87, PR-AUC = 0.57, and ΔNB ≈ 0.054.
3. If we want to once again be serious about precision psychiatry and aim for PR-AUC of 0.8, we will find it requires r = 0.9 (R² = 0.81). These are rather extremely ambitious values (requiring to explain 81% of variance in symptom improvement). This helps demonstrate the inherent limitations of dichotomizing continuous outcomes for evaluating treatment response prediction - doing so leads to a loss of valuable information. On the other hand, it does reflect the binary nature of decision-making in psychiatry (to prescribe the treatment or not).

Explore this example yourself →

Example 3: Risk Prediction

Can we predict who will attempt suicide using a combination of risk factors?

Most suicide prediction models (which include a variety of risk factors such as psychopathology, history of suicidal behavior, socio-demographics, etc.) achieve ROC-AUC > 0.7, with a handful exceeding ROC-AUC = 0.9 for predicting suicide attempts (Pigoni et al., 2024). Let's ignore potential overfitting concerns and consider how ROC-AUC = 0.9 would translate to real-world predictive utility.

1. In the Binary outcome mode:
   • Set base rate to 0.3% (the 12-month prevalence of suicide attempts; Borges et al., 2011)
   • Set the grouping reliability to 0.8 (The Columbia Suicide Severity Rating Scale (C-SSRS) supporting evidence)
   • Set the predictor reliability for both groups to 0.8 (a rough estimate for the average reliability across the risk factors)
   • Set the observed effect size to d = 2.04, which corresponds to AUC = 0.9
2. This will yield PR-AUC = 0.08, indicating extremely abysmal predictive performance, meaning that depending on where we place the threshold, we would either predict a lot of false positives (low precision, high recall) or a lot of false negatives (high precision, low recall). When the threshold is set to maximize the F1 score (0.15), we get precision = 0.15 and recall = 0.15. Which means that we would miss 85% of actual attempters and 85% of the predicted attempters would not attempt suicide. Not surprisingly, this gives us extremely low clinical utility (ΔNB ≈ 0.0004 at p_t = 0.15). At threshold of p_t = 0.05, which would be reasonable given how costly false negatives (missing actual attempters) are, ΔNB would be very low (≈ 0.0004), illustrating the limited utility of prediction in this context.
3. Considering that we are serious about precision psychiatry and want to aim for PR-AUC of 0.8, we will find it requires observed d = 3.76, which means we have a very long way to go. Note that because the reliability is already quite high, improving it further would not make much of a difference. What we need to do is to find better predictors. Alternatively, it may be more effective to simply focus on universal suicide prevention strategies rather than trying to predict individual cases (e.g.,Large 2018).

Explore this example yourself →

Sample Size Calculations for Prediction Models

E2P Simulator includes sample size calculators that implement evidence-based criteria (Riley et al., 2020) to ensure your multivariable prediction models have sufficient data to avoid overfitting and ensure low prediction error. These go beyond simple rules of thumb like "10 events per predictor."

How to Use

Specify your number of predictors, realistically expected R² (based on prior research or pilot data), and outcome prevalence (for binary outcomes only). Use R²_CS for binary outcomes and standard R² for continuous outcomes. Note, for binary outcomes with a single continuous predictor, R²_CS equals eta-squared (η²), which is already displayed in the main E2P simulator dashboard. The final recommendation uses the maximum across all criteria to ensure all performance targets are met.

The sample size calculators complement the main E2P simulators in study planning: the E2P simulators explore relationships between effect sizes and predictive utility (both what you need for desired performance and what utility to expect from realistic effects), while the sample size calculators determine adequate sample size based on realistic R² estimates from prior research. For sample size planning, always use conservative, realistic R² estimates based on prior research, not idealized target values.

Single Predictor Models and Power Analysis

When developing a model with just one predictor (p = 1), the sample size calculations relate directly to traditional power analysis for detecting that predictor's effect. However, the criteria still provide value beyond simple power calculations:

• Power analysis: Focuses on detecting a statistically significant effect (p < 0.05)
• Prediction-focused criteria: Ensure the model will be robust and generalizable, with stable coefficients and accurate predictions

For single predictors, the shrinkage and optimism criteria often require larger sample sizes than traditional power analysis, reflecting the higher standards needed for reliable prediction versus mere statistical significance.

Feedback and Contributions

E2P Simulator is an open-source project - feedback, bug reports, and suggestions for improvement are welcome. The easiest way to do so is through the GitHub Issues page.

You can view the source code, track development, and contribute directly at the project's GitHub repository.

For other inquiries, you can find my contact information here.