Getting Started with E2P Simulator

How to use E2P Simulator

E2P Simulator is designed to be intuitive and interactive. You can explore different scenarios by adjusting parameters like effect sizes, measurement reliability, base rates, and decision threshold, and immediately see how these changes impact predictive performance through various visualizations and metrics. Still, in this section we will highlight and clarify some of the key features and assumptions of the simulator.

Binary vs. Continuous Outcomes

E2P Simulator provides two analysis modes that cover the two most common research scenarios:

• Binary Mode: Use this mode for analyzing dichotomous outcomes like diagnostic categories (e.g., cases vs. controls) or discrete states (e.g., success vs. failure). All metric calculations and conversions in this mode are completely analytical and follow the formulas provided on the page.
• Continuous Mode: Use this mode for analyzing continuous measurements like symptom severity or performance scores that may need to be categorized (e.g., responders vs. non-responders or performers vs. non-performers) for practical decisions. This mode is based on actual data simulations and the affects on all the metrics are calculated from the data itself.

Measurement Reliability and True vs. Observed Effects

Measurement reliability attenuates the observed effect sizes, and in turn attenuates the predictive performance. The simulator allows you to specify reliability using appropriate metrics - the Intraclass Correlation Coefficient (ICC) for continuous measurements and Cohen's kappa (κ) for binary classifications. These typically correspond to test-retest reliability and inter-rater reliability, respectively. You can toggle between "true" effect sizes (what would be observed with perfect measurement) and "observed" effect sizes (what we actually see given imperfect reliability).

Note that the simulator does not account for sample size limitations, which can introduce additional uncertainty around the true effect size through sampling error.

Base Rates

The base rate refers to the prevalence of the outcome in the studied population (rather than the sample used to calculate the effect size). It is important to make sure that the effect size in question and the base rate correspond to the same population. For example, if the effect size characterizes the differences between some disease/disorder group and a healthy control group, the relevant base rate is the prevalence of the disease/disorder in the general population. In a different scenario, if we have identified a high-risk group, and want to use the disease/disorder prevalence within this group, then we need to know the effect size comparing disease/disorder vs. the rest of the high-risk group, as it may not be the same as when comparing with healthy controls.

The base rate primarily affects the tradeoff between precision and recall, which is reflected in PR-AUC, MCC, and F-1 score (see Understanding Predictive Metrics).

Multivariate Effects Calculators

Both binary and continuous outcomes analysis modes include calculators that help explore how multiple predictors can be combined to achieve stronger effects:

• Mahalanobis D Calculator (Binary mode)
• Multivariate R² Calculator (Continuous mode)

The calculators can help approximate the expected performance of multivariate models without having to train the full models and thus help with research planning and model development. More specifically, they illustrate:

• How the number of predictors affects combined effect size
• The diminishing returns of adding more predictors
• How collinearity (correlation among predictors) reduces their combined effectiveness
• The trade-offs between using fewer strong predictors versus more moderate ones

Calculator Assumptions and Limitations

Both calculators correspond to fundamental predictive models in statistics: the Mahalanobis D Calculator approximates Linear Discriminant Analysis and logistic regression, while the Multivariate R² Calculator aligns with multiple linear regression.

Like the statistical methods they approximate, the calculators operate under several simplifying assumptions:

• Average effects and correlations: The calculators use single values to represent the average effect size across predictors and average correlation among them, which can provide useful approximations even when individual predictors vary in strength
• Linear effects: The formulas assume predictors contribute additively without interactions (where one predictor's effect depends on another). However, research shows that in clinical prediction, complex non-linear models generally do not outperform simple linear logistic regression (Christodoulou et al., 2019)
• Normality: Variables are assumed to be normally distributed

Despite these limitations, these calculators serve as valuable tools for building intuition about how multiple predictors combine to achieve stronger effects. Even though real-world predictors will vary in their individual strengths and collinearity, the overall patterns demonstrated (such as diminishing returns and the impact of shared variance) remain informative for understanding multivariate relationships.

Quick Start Examples

To further clarify how the tool can be used and to demonstrate its utility we provide some specific examples.

Example 1: Diagnostic Prediction

Can we predict depression diagnosis using a cognitive biomarker?

1. In the Binary outcome mode:
   • Set base rate to 8% (the prevalence of depression in the population; Shorey et al., 2021)
   • Set the grouping reliability to 0.28 (depression diagnosis reliability based on DSM-5 field trials; Regier et al., 2013)
   • Set the predictor reliability for both groups to 0.6 (an average reliability for cognitive measures; Karvelis et al., 2023)
   • Set the observed effect size to d = 0.8 (a large effect size that is optimistic and rarely seen in practice)
2. This will yield AUC = 0.71 and PR-AUC = 0.19. So, even with the optimistic effect size of 0.8, the predictive utility remains very modest, especially when it comes to the tradeoff between recall and precision (as shown by the low PR-AUC).
3. Note that with the low reliability values, this observed effect corresponds to a much larger true effect, d = 1.58, which, while not being very realistic in practice, highlights how much information is lost due to lack of measurement reliability.
4. Now let's say we are serious about precision psychiatry and we want to achieve a PR-AUC of 0.8. Using the tool, we can find that it would require d = 2.55. It would be totally unrealistic to expect a single biomarker to achieve this effect size. Using the Mahalanobis D calculator, you can explore how many predictors with smaller d values would be required to achieve D = 2.55.

Explore this example yourself →

Example 2: Treatment Response Prediction

Can we predict who will respond to antidepressant treatment using a cognitive biomarker?

1. Select Continuous outcome mode:
• Set base rate to 15% (the rate of response to antidepressant treatment beyond placebo; Stone et al., 2022)
• Set predictor reliability to 0.6 (an average reliability for cognitive measures; Karvelis et al., 2023)
• Set outcome reliability to 0.94 (Hamilton Depression Rating Scale (HAMD)) reliability; Trajković et al., 2011)
• Adjust effect size such that R² = 0.2 (average multivariate R² from recent research; Karvelis et al., 2022)
2. This will yield AUC = 0.73 and PR-AUC = 0.32, indicating rather modest predictive performance, as shown by the low PR-AUC. Note that the limiting factor in this scenario is not so much the reliability but the effect size.
3. If we want to once again be serious about precision psychiatry and aim for PR-AUC of 0.8, we will find it requires r = 0.9 (R² = 0.81). These are rather extremely ambitious values (requiring to explain 81% of variance in symptom improvement). This helps demonstrate the inherent limitations of dichotomizing continuous outcomes for evaluating treatment response prediction - doing so leads to a loss of valuable information. On the other hand, it does reflect the nature of binary decision-making in psychiatry (to prescribe the treatment or not).

Explore this example yourself →

Example 3: Risk Prediction

Can we predict who will attempt suicide using a combination of risk factors?

Here we will rely motly on Borges et al., 2011, who analyzed the data of 108,705 adults from 21 countries, and using a logistic regression model found that combining a range of risk factors (socio-demographics, parent psychopathology, childhood adversities, DSM-IV disorders, and history of suicidal behavior) led to AUC = 0.74-0.8 discrimination performance between those who did and did not attempt suicide.

1. In the Binary outcome mode:
   • Set base rate to 0.3% (the 12-month prevalence of suicide attempts; Borges et al., 2011)
   • Set the grouping reliability to 0.8 (The Columbia Suicide Severity Rating Scale (C-SSRS) supporting evidence)
   • Set the predictor reliability for both groups to 0.8 (a rough estimate for the average reliability across the risk factors)
   • Set the observed effect size to d = 1.18, which corresponds to AUC = 0.8
2. This will yield PR-AUC = 0.02, indicating extremely abysmal predictive performance, meaning that depending on where we place the threshold, we would either predict a lot of false positives (low precision, high recall) or a lot of false negatives (high precision, low recall). When the threshold is set to maximize the F1 score (0.06), we get precision = 0.06 and recall = 0.06. Which means that we would miss 94% of actual attempters and 94% of the predicted attempters would not attempt suicide.
3. Considering that we are serious about precision psychiatry and want to aim for PR-AUC of 0.8, we will find it requires observed d = 3.76, which means we have a very long way to go. Note that because the reliability is already quite high, improving it further would not make much of a difference. What we need to do is to find better predictors. Alternatively, it may be more effective to simply focus on universal suicide prevention strategies rather than trying to predict individual cases (e.g.,Large 2018).

Explore this example yourself →

Understanding Predictive Metrics

Classification Outcomes and Metrics

When using a predictor to classify cases into two groups, there are four possible outcomes: True Positives (TP), False Positives (FP), False Negatives (FN), and True Negatives (TN). These form the basis for all predictive metrics.

The image above illustrates how these four outcomes relate to various classification metrics. On the left, you can see how negative (e.g., controls) and positive (e.g., cases) distributions overlap and how a classification threshold (red line) creates these four outcomes. On the right, you'll find the confusion matrix and the formulas for key metrics derived from it. Note how some metrics have multiple names (e.g., Sensitivity/Recall/TPR, Precision/PPV) - this reflects how the same concepts are referred to differently across fields like medicine, cognitive science, and machine learning.

Key Metrics for Different Contexts

Depending on your research context, certain metrics may be more relevant than others:

• When false negatives are costly (e.g., missing a disease diagnosis): Focus on Sensitivity/Recall. In clinical settings where missing a diagnosis could be life-threatening, maximizing Sensitivity ensures fewer cases are missed, even if it means more false alarms.
• When false positives are costly (e.g., unnecessary treatments): Focus on Specificity. When treatments have significant side effects or costs, high Specificity ensures fewer healthy individuals receive unnecessary interventions.
• When dealing with low base rate: Focus on Precision (PPV), NPV, F1 score, MCC, and PR-AUC. These metrics are sensitive to the base rate and thus provide a more accurate assessment of the model's performance in the real world.

Threshold-Independent Metrics

Some metrics evaluate performance across all possible thresholds and, as such, can serve as a better summary of the overall model performance. These include:

• AUC (Area Under the ROC Curve): Visualizes how well a model balances true positives (Sensitivity) and false positives (1-Specificity) across all possible thresholds, capturing the trade-off between the two. An AUC of 0.5 means the model is no better than flipping a coin, while 1.0 means perfect separation between groups.
• PR-AUC (Area Under the Precision-Recall Curve): Shows how well a model can maintain both precision (PPV) and Recall (Sensitivity) together, which is particularly informative in the context of rare outcomes, as the base rate affects Precision. A larger PR-AUC means the model can achieve high Precision without sacrificing Recall (or vice versa), indicating a smaller trade-off between finding all positive cases and avoiding false alarms.

Feedback and Contributions

E2P Simulator is an open-source project - feedback, bug reports, and suggestions for improvement are welcome. The easiest way to do so is through the GitHub Issues page.

You can view the source code, track development, and contribute directly at the project's GitHub repository.

For other inquiries, you can find contact information on my personal website.