Machine learning algorithms are widely used for decision making in societally high-stakes settings from child welfare and criminal justice to healthcare and consumer lending. Recent history has illuminated numerous examples where these algorithms proved unreliable or inequitable. This talk will show how causal inference enables us to more reliably evaluate such algorithms’ performance and equity implications. In the first part of the talk, it will be demonstrated that standard evaluation procedures fail to address missing data and as a result, often produce invalid assessments of algorithmic performance. A new evaluation framework is proposed that addresses missing data by using counterfactual techniques to estimate unknown outcomes. Using this framework, we propose counterfactual analogues of common predictive performance and algorithmic fairness metrics that are tailored to decision-making settings. We provide double machine learning-style estimators for these metrics that achieve