Workshop

Nonsense associations can arise when an exposure and an outcome of interest exhibit similar patterns of dependence. Confounding is present when potential outcomes are not independent of treatment. This talk will describe how confusion about these two phenomena results in shortcomings in popular methods in three areas: causal inference with multiple treatments and unmeasured confounding, causal and statistical inference with social network data, and spatial confounding. For each of these areas I will demonstrate the flaws in existing methods and describe new methods that were inspired by careful consideration of dependence and confounding.

Mediation analysis assesses the extent to which the treatment affects the outcome through a mediator and the extent to which it operates through other pathways. As one of the most cited methods in empirical mediation analysis, the classic Baron–Kenny approach allows us to estimate the indirect and direct effects of the treatment on the outcome in linear structural equation models. However, when the treatment and the mediator are not randomized, the estimates of the direct and indirect effects from the Baron–Kenny approach may be biased due to unmeasured confounding among the treatment, mediator, and outcome. Building on Cinelli & Hazlett (2020), we provide a sharp and interpretable sensitivity analysis method for the Baron–Kenny approach to mediation in the presence of unmeasured confounding. We first generalize Cinelli & Hazlett (2020)’s sensitivity analysis method for linear regression to allow for heteroskedasticity and model misspecification. We then apply the general result to develop a sensitivity analysis method for the Baron–Kenny approach. Importantly, we express the sensitivity parameters in terms of the partial R^2s that correspond to the natural factorization of the joint distribution of the direct acyclic graph. Thus, they are interpretable as the proportions of variability explained by unmeasured confounding given the observed covariates. Moreover, we extend the method to deal with multiple mediators, based on a novel matrix version of the partial R^2. We prove that all our sensitivity bounds are attainable and thus sharp.

We consider the problem of predicting a response Y from a set of covariates X when test and training distributions differ. We consider a setting where such differences have causal explanations and the test distributions emerge from interventions. Causal models minimize the worst-case risk under arbitrary interventions on the covariates but may not always be identifiable from observational or interventional data. In this talk, we argue that underidentification and distribution generalization are closely connected. We propose to consider most predictive invariant models and discuss some of their properties. We also present limits of distribution generalization.

One hopes that data analyses will be used to make beneficial decisions regarding people's health, finances, and well-being. But the data fed to an analysis may systematically differ from the data where these decisions are ultimately applied. For instance, suppose we analyze data in one country and conclude that microcredit is effective at alleviating poverty; based on this analysis, we decide to distribute microcredit in other locations and in future years. We might then ask: can we trust our conclusion to apply under new conditions? If we found that a very small percentage of the original data was instrumental in determining the original conclusion, we might expect the conclusion to be unstable under new conditions. So we propose a method to assess the sensitivity of data analyses to the removal of a very small fraction of the data set. Analyzing all possible data subsets of a certain size is computationally prohibitive, so we provide an approximation. We call our resulting method the Approximate Maximum Influence Perturbation. Our approximation is automatically computable, theoretically supported, and works for common estimators --- including (but not limited to) OLS, IV, GMM, MLE, MAP, and variational Bayes. We show that any non-robustness our metric finds is conclusive. Empirics demonstrate that while some applications are robust, in others the sign of a treatment effect can be changed by dropping less than 0.1% of the data --- even in simple models and even when standard errors are small.

We introduce Kernel Thinning-Compress++ (KT-Compress++), an algorithm based on two new procedures for compressing a distribution P nearly optimally and more effectively than i.i.d. sampling or standard thinning. Given a suitable reproducing kernel k and O(n log^3 n) time, KT-Compress++ compresses an n-point approximation to P into a sqrt(n)-point approximation with better than Monte Carlo integration error rates for functions in the associated reproducing kernel Hilbert space (RKHS). First we show that for any fixed function KT-Compress++ provides dimension-free guarantees for any kernel, any distribution, and any fixed function in the RKHS. Second, we show that with high probability, the maximum discrepancy in integration error is O_d(n^{-1/2} sqrt(log n)) for compactly supported P and O_d(n^{-1/2} (log n)^{(d+1)/2} sqrt(loglog n)) for sub-exponential P on R^d. In contrast, an equal-sized i.i.d. sample from P suffers at least n^{-1/4} integration error. Our sub-exponential guarantees nearly match the known lower bounds for several settings, and while resembling the classical quasi-Monte Carlo error rates for uniform P on [0,1]^d they apply to general distributions on R^d and a wide range of commonly used universal kernels. En route, we introduce a new simple meta-procedure Compress++, that can speed up any thinning algorithm while suffering at most a factor of 4 n error. In particular, Compress++ enjoys a near-linear runtime given any quadratic-time input and reduces the runtime of super-quadratic algorithms by a square-root factor. We also use our results to derive explicit non-asymptotic maximum mean discrepancy bounds for a range of kernels including Gaussian, Matern, Laplace, and B-spline kernels. Finally, we present several vignettes illustrating the practical benefits of KT-Compress++ over i.i.d. sampling and standard Markov chain Monte Carlo thinning with challenging differential equation posteriors in dimensions d = 2 to 100.

Embedding nodes of a large network into a metric (e.g., Euclidean) space has become an area of active research in statistical machine learning, which has found applications in natural and social sciences. Generally, a representation of a network object is learned in Euclidean geometry and is then used for subsequent tasks regarding the nodes and/or edges of the network, such as community detection, node classification, and link prediction. Network embedding algorithms have been proposed in multiple disciplines, often with domain-specific notations and details. In addition, different measures and tools have been adopted to evaluate and compare the methods proposed under different settings, often dependent of the downstream tasks. As a result, it is challenging to study these algorithms in the literature systematically. Motivated by the recently proposed Veridical Data Science (VDS) framework, we propose a framework for network embedding algorithms and discuss how the principles of predictability, computability, and stability apply in this context. The utilization of this framework in network embedding holds the potential to motivate and point to new directions for future research.

Bayesian nonparametric models are a popular and effective tool for inferring the heterogeneous effects of interventions. I will discuss how to carefully specify models and prior distributions to apply judicious regularization of heterogeneous effects. I will also discuss how to extract answers to scientific and policy questions from a fitted nonparametric model using posterior summarization to avoid problems incurred by using competing or incompatible model specifications for targeting different estimands. Together these tools provide a general recipe for obtaining stable, generalizable and transferrable insights about heterogeneous effects.

We consider the problem of transferring robotic control from simulation to the real world. In particular, we consider two important sub-problems that are often faced: model size and visual robustness.

To decrease the size of our trained models, we develop a one shot pruning technique for recurrent and time series models that significantly reduces our model footprint while maintaining accuracy. This smaller model size is crucial for overcoming hardware limitations in robotics. For our vision system, we develop a new a statistical process, Invariance Through Inference, for adapting visual systems from simulation into the real world. This process shows how we can use statistical inference at test time to extract robust visual features that are constant across simulated and real world models.

We will consider a metric---the "Projection Norm"---that predicts a model's performance on out-of-distribution (OOD) data, without access to ground truth labels. Projection Norm first uses model predictions to pseudo-label test samples and then trains a new model on the pseudo-labels. The more the new model's parameters differ from an in-distribution model, the greater the predicted OOD error. Empirically, this outperforms existing methods on both image and text classification tasks and across different network architectures. Theoretically, we connect our approach to a bound on the test error for overparameterized linear models. Furthermore, we find that Projection Norm is the only approach that achieves non-trivial detection performance on adversarial examples.

Joint work with Yaodong Yu, Zitong Yang, Alex Wei, and Yi Ma. https://arxiv.org/abs/2202.05834