understanding black box predictions via influence functions

However, in a lower Data-trained predictive models see widespread use, but for the most part they are used as black boxes which output a prediction or score. Uses cases Roadmap 2 Reviving an "old technique" from Robust statistics: Influence function We try to understand the effects they have on the dynamics and identify some gotchas in building deep learning systems. Understanding Black-box Predictions via Influence Functions International Conference on Machine Learning (ICML), 2017. In. Jaeckel, L. A. After all, the optimization landscape is nonconvex, highly nonlinear, and high-dimensional, so why are we able to train these networks? I am grateful to my supervisor Tasnim Azad Abir sir, for his . In this paper, we use influence functions a classic technique from robust statistics to trace a models prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. Bilevel optimization refers to optimization problems where the cost function is defined in terms of the optimal solution to another optimization problem. The dict structure looks similiar to this: Harmful is a list of numbers, which are the IDs of the training data samples On the importance of initialization and momentum in deep learning. below is divided into parameters affecting the calculation and parameters << In this paper, we use influence functions -- a classic technique from robust statistics -- to trace a model's prediction through the learning algorithm and back to its training data, thereby . Overwhelmed? Class will be held synchronously online every week, including lectures and occasionally tutorials. %PDF-1.5 If there are n samples, it can be interpreted as 1/n. (a) train loss, Hessian, train_loss + Hessian . In this paper, we use influence functions a classic technique from robust statistics to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. Linearization is one of our most important tools for understanding nonlinear systems. on the final predictions is straight forward. International conference on machine learning, 1885-1894, 2017. Understanding black-box predictions via influence functions Computing methodologies Machine learning Recommendations On second-order group influence functions for black-box predictions With the rapid adoption of machine learning systems in sensitive applications, there is an increasing need to make black-box models explainable. Agarwal, N., Bullins, B., and Hazan, E. Second order stochastic optimization in linear time. lehman2019inferringE. In this paper, we use influence functions a classic technique from robust statistics to trace a models prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. In this paper, we use influence functions -- a classic technique from robust statistics -- to trace a model's prediction through . While these topics had consumed much of the machine learning research community's attention when it came to simpler models, the attitude of the neural nets community was to train first and ask questions later. The ACM Digital Library is published by the Association for Computing Machinery. Visualised, the output can look like this: The test image on the top left is test image for which the influences were we develop a simple, efficient implementation that requires only oracle access to gradients Which optimization techniques are useful at which batch sizes? Neural tangent kernel: Convergence and generalization in neural networks. Limitations of the empirical Fisher approximation for natural gradient descent. This code replicates the experiments from the following paper: Pang Wei Koh and Percy Liang Understanding Black-box Predictions via Influence Functions International Conference on Machine Learning (ICML), 2017. Understanding Black-box Predictions via Influence Functions Background information ICML 2017 best paper Stanford Pang Wei Koh CourseraStanfordNIPS 2019influence function Percy Liang11Michael Jordan Abstract Jianxin Ma, Peng Cui, Kun Kuang, Xin Wang, and Wenwu Zhu. James Tu, Yangjun Ruan, and Jonah Philion. Adaptive Gradient Methods, Normalization, and Weight Decay [Slides]. A. M. Saxe, J. L. McClelland, and S. Ganguli. to trace a model's prediction through the learning algorithm and back to its training data, In. Cook, R. D. Assessment of local influence. Liu, D. C. and Nocedal, J. It is known that in a high complexity class such as exponential time, one can convert worst-case hardness into average-case hardness. Despite its simplicity, linear regression provides a surprising amount of insight into neural net training. config is a dict which contains the parameters used to calculate the Debruyne, M., Hubert, M., and Suykens, J. ICML 2017 best paperStanfordPang Wei KohPercy liang, x_{test} y_{test} label x_{test} , n z_1z_n z_i=(x_i,y_i) L(z,\theta) z \theta , \hat{\theta}=argmin_{\theta}\frac{1}{n}\Sigma_{i=1}^{n}L(z_i,\theta), z z \epsilon ERM, \hat{\theta}_{\epsilon,z}=argmin_{\theta}\frac{1}{n}\Sigma_{i=1}^{n}L(z_i,\theta)+\epsilon L(z,\theta), influence function, \mathcal{I}_{up,params}(z)={\frac{d\hat{\theta}_{\epsilon,z}}{d\epsilon}}|_{\epsilon=0}=-H_{\hat{\theta}}^{-1}\nabla_{\theta}L(z,\hat{\theta}), H_{\hat\theta}=\frac{1}{n}\Sigma_{i=1}^{n}\nabla_\theta^{2} L(z_i,\hat\theta) Hessien, \begin{equation} \begin{aligned} \mathcal{I}_{up,loss}(z,z_{test})&=\frac{dL(z_{test},\hat\theta_{\epsilon,z})}{d\epsilon}|_{\epsilon=0} \\&=\nabla_\theta L(z_{test},\hat\theta)^T {\frac{d\hat{\theta}_{\epsilon,z}}{d\epsilon}}|_{\epsilon=0} \\&=\nabla_\theta L(z_{test},\hat\theta)^T\mathcal{I}_{up,params}(z)\\&=-\nabla_\theta L(z_{test},\hat\theta)^T H^{-1}_{\hat\theta}\nabla_\theta L(z,\hat\theta) \end{aligned} \end{equation}, lossNLPer, influence function, logistic regression p(y|x)=\sigma (y \theta^Tx) \sigma sigmoid z_{test} loss z \mathcal{I}_{up,loss}(z,z_{test}) , -y_{test}y \cdot \sigma(-y_{test}\theta^Tx_{test}) \cdot \sigma(-y\theta^Tx) \cdot x^{T}_{test} H^{-1}_{\hat\theta}x, \sigma(-y\theta^Tx) outlieroutlier, x^{T}_{test} x H^{-1}_{\hat\theta} Hessian \mathcal{I}_{up,loss}(z,z_{test}) resistencevariation, \mathcal{I}_{up,loss}(z,z_{test})=-\nabla_\theta L(z_{test},\hat\theta)^T H^{-1}_{\hat\theta}\nabla_\theta L(z,\hat\theta), Hessian H_{\hat\theta} O(np^2+p^3) n p z_i , conjugate gradientstochastic estimationHessian-vector productsHVP H_{\hat\theta} s_{test}=H^{-1}_{\hat\theta}\nabla_\theta L(z_{test},\hat\theta) \mathcal{I}_{up,loss}(z,z_{test})=-s_{test} \cdot \nabla_{\theta}L(z,\hat\theta) , H_{\hat\theta}^{-1}v=argmin_{t}\frac{1}{2}t^TH_{\hat\theta}t-v^Tt, HVPCG O(np) , H^{-1} , (I-H)^i,i=1,2,\dots,n H 1 j , S_j=\frac{I-(I-H)^j}{I-(I-H)}=\frac{I-(I-H)^j}{H}, \lim_{j \to \infty}S_j z_i \nabla_\theta^{2} L(z_i,\hat\theta) H , HVP S_i S_i \cdot \nabla_\theta L(z_{test},\hat\theta) , NMIST H loss , ImageNetInceptionRBF SVM, RBF SVMRBF SVM, InceptionInception, Inception, , Inception591/60059133557%, check \mathcal{I}_{up,loss}(z_i,z_i) z_i , 10% \mathcal{I}_{up,loss}(z_i,z_i) , H_{\hat\theta}=\frac{1}{n}\Sigma_{i=1}^{n}\nabla_\theta^{2} L(z_i,\hat\theta), s_{test}=H^{-1}_{\hat\theta}\nabla_\theta L(z_{test},\hat\theta), \mathcal{I}_{up,loss}(z,z_{test})=-s_{test} \cdot \nabla_{\theta}L(z,\hat\theta), S_i \cdot \nabla_\theta L(z_{test},\hat\theta). The algorithm moves then Understanding Black-box Predictions via Influence Functions. Rethinking the Inception architecture for computer vision. Fortunately, influence functions give us an efficient approximation. All Holdings within the ACM Digital Library. Depending what you're trying to do, you have several options: You are welcome to use whatever language and framework you like for the final project. So far, we've assumed gradient descent optimization, but we can get faster convergence by considering more general dynamics, in particular momentum. This paper applies influence functions to ANNs taking advantage of the accessibility of their gradients. Three mechanisms of weight decay regularization. Here, we used CIFAR-10 as dataset. Appendix: Understanding Black-box Predictions via Inuence Functions Pang Wei Koh1Percy Liang1 Deriving the inuence functionIup,params For completeness, we provide a standard derivation of theinuence functionIup,params in the context of loss minimiza-tion (M-estimation). A spherical analysis of Adam with batch normalization. Gradient-based hyperparameter optimization through reversible learning. Understanding Black-box Predictions via Influence Functions. Fast convergence of natural gradient descent for overparameterized neural networks. , . In this paper, we use influence functions --- a classic technique from robust statistics --- to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. J. Cohen, S. Kaur, Y. Li, J. Model selection in kernel based regression using the influence function. This site last compiled Wed, 08 Feb 2023 10:43:27 +0000. Influence functions are a classic technique from robust statistics to identify the training points most responsible for a given prediction. How can we explain the predictions of a black-box model? Interacting with predictions: Visual inspection of black-box machine learning models. To manage your alert preferences, click on the button below. On linear models and convolutional neural networks, Here, we plot I up,loss against variants that are missing these terms and show that they are necessary for picking up the truly inuential training points. Infinite Limits and Overparameterization [Slides]. 2018. Understanding black-box predictions via influence functions. That can increase prediction accuracy, reduce calculate which training images had the largest result on the classification LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. We'll use linear regression to understand two neural net training phenomena: why it's a good idea to normalize the inputs, and the double descent phenomenon whereby increasing dimensionality can reduce overfitting. Or we might just train a flexible architecture on lots of data and find that it has surprising reasoning abilities, as happened with GPT3. influence-instance. can speed up the calculation significantly as no duplicate calculations take ICML 2017 best paperStanfordPang Wei KohCourseraStanfordNIPS 2019influence functionPercy Liang11Michael Jordan, , \hat{\theta}_{\epsilon, z} \stackrel{\text { def }}{=} \arg \min _{\theta \in \Theta} \frac{1}{n} \sum_{i=1}^{n} L\left(z_{i}, \theta\right)+\epsilon L(z, \theta), \left.\mathcal{I}_{\text {up, params }}(z) \stackrel{\text { def }}{=} \frac{d \hat{\theta}_{\epsilon, z}}{d \epsilon}\right|_{\epsilon=0}=-H_{\tilde{\theta}}^{-1} \nabla_{\theta} L(z, \hat{\theta}), , loss, \begin{aligned} \mathcal{I}_{\text {up, loss }}\left(z, z_{\text {test }}\right) &\left.\stackrel{\text { def }}{=} \frac{d L\left(z_{\text {test }}, \hat{\theta}_{\epsilon, z}\right)}{d \epsilon}\right|_{\epsilon=0} \\ &=\left.\nabla_{\theta} L\left(z_{\text {test }}, \hat{\theta}\right)^{\top} \frac{d \hat{\theta}_{\epsilon, z}}{d \epsilon}\right|_{\epsilon=0} \\ &=-\nabla_{\theta} L\left(z_{\text {test }}, \hat{\theta}\right)^{\top} H_{\hat{\theta}}^{-1} \nabla_{\theta} L(z, \hat{\theta}) \end{aligned}, \varepsilon=-1/n , z=(x,y) \\ z_{\delta} \stackrel{\text { def }}{=}(x+\delta, y), \hat{\theta}_{\epsilon, z_{\delta},-z} \stackrel{\text { def }}{=}\arg \min _{\theta \in \Theta} \frac{1}{n} \sum_{i=1}^{n} L\left(z_{i}, \theta\right)+\epsilon L\left(z_{\delta}, \theta\right)-\epsilon L(z, \theta), \begin{aligned}\left.\frac{d \hat{\theta}_{\epsilon, z_{\delta},-z}}{d \epsilon}\right|_{\epsilon=0} &=\mathcal{I}_{\text {up params }}\left(z_{\delta}\right)-\mathcal{I}_{\text {up, params }}(z) \\ &=-H_{\hat{\theta}}^{-1}\left(\nabla_{\theta} L(z_{\delta}, \hat{\theta})-\nabla_{\theta} L(z, \hat{\theta})\right) \end{aligned}, \varepsilon \delta \deltaloss, \left.\frac{d \hat{\theta}_{\epsilon, z_{\delta},-z}}{d \epsilon}\right|_{\epsilon=0} \approx-H_{\hat{\theta}}^{-1}\left[\nabla_{x} \nabla_{\theta} L(z, \hat{\theta})\right] \delta, \hat{\theta}_{z_{i},-z}-\hat{\theta} \approx-\frac{1}{n} H_{\hat{\theta}}^{-1}\left[\nabla_{x} \nabla_{\theta} L(z, \hat{\theta})\right] \delta, \begin{aligned} \mathcal{I}_{\text {pert,loss }}\left(z, z_{\text {test }}\right)^{\top} &\left.\stackrel{\text { def }}{=} \nabla_{\delta} L\left(z_{\text {test }}, \hat{\theta}_{z_{\delta},-z}\right)^{\top}\right|_{\delta=0} \\ &=-\nabla_{\theta} L\left(z_{\text {test }}, \hat{\theta}\right)^{\top} H_{\hat{\theta}}^{-1} \nabla_{x} \nabla_{\theta} L(z, \hat{\theta}) \end{aligned}, train lossH \mathcal{I}_{\text {up, loss }}\left(z, z_{\text {test }}\right) , -y_{\text {test }} y \cdot \sigma\left(-y_{\text {test }} \theta^{\top} x_{\text {test }}\right) \cdot \sigma\left(-y \theta^{\top} x\right) \cdot x_{\text {test }}^{\top} H_{\hat{\theta}}^{-1} x, influence functiondebug training datatraining point \mathcal{I}_{\text {up, loss }}\left(z, z_{\text {test }}\right) losstraining pointtraining point, Stochastic estimationHHHTFO(np)np, ImageNetdogfish900Inception v3SVM with RBF kernel, poisoning attackinfluence function59157%77%10590/591, attackRelated worktraining set attackadversarial example, influence functionbad case debug, labelinfluence function, \mathcal{I}_{\text {up,loss }}\left(z_{i}, z_{i}\right) , 10%labelinfluence functiontrain lossrandom, \mathcal{I}_{\text {up, loss }}\left(z, z_{\text {test }}\right), \mathcal{I}_{\text {up,loss }}\left(z_{i}, z_{i}\right), \mathcal{I}_{\text {pert,loss }}\left(z, z_{\text {test }}\right)^{\top}, H_{\hat{\theta}}^{-1} \nabla_{x} \nabla_{\theta} L(z, \hat{\theta}), Less Is Better: Unweighted Data Subsampling via Influence Function, influence functionleave-one-out retraining, 0.86H, SVMhinge loss0.95, straightforwardbest paper, influence functionloss. Up to now, we've assumed networks were trained to minimize a single cost function. Things get more complicated when there are multiple networks being trained simultaneously to different cost functions. Terry Taewoong Um (terry.t.um@gmail.com) University of Waterloo Department of Electrical & Computer Engineering Terry T. Um UNDERSTANDING BLACK-BOX PRED -ICTION VIA INFLUENCE FUNCTIONS 1 A. S. Benjamin, D. Rolnick, and K. P. Kording. While influence estimates align well with leave-one-out. The previous lecture treated stochasticity as a curse; this one treats it as a blessing. We look at what additional failures can arise in the multi-agent setting, such as rotation dynamics, and ways to deal with them. Li, B., Wang, Y., Singh, A., and Vorobeychik, Y. Understanding black-box predictions via influence functions. Thus, in the calc_img_wise mode, we throw away all grad_z I recommend you to change the following parameters to your liking. Dependencies: Numpy/Scipy/Scikit-learn/Pandas the algorithm will then calculate the influence functions for all images by For the final project, you will carry out a small research project relating to the course content. In. We'll consider the two most common techniques for bilevel optimization: implicit differentiation, and unrolling. Amershi, S., Chickering, M., Drucker, S. M., Lee, B., Simard, P., and Suh, J. Modeltracker: Redesigning performance analysis tools for machine learning. Approach Consider a prediction problem from some input space X (e.g., images) to an output space Y(e.g., labels). numbers above the images show the actual influence value which was calculated. How can we explain the predictions of a black-box model? calculated. (a) What is the effect of the training loss and H 1 ^ terms in I up,loss? ; Liang, Percy. Not just a black box: Learning important features through propagating activation differences. the training dataset were the most helpful, whereas the Harmful images were the We have 3 hours scheduled for lecture and/or tutorial. place. A Dockerfile with these dependencies can be found here: https://hub.docker.com/r/pangwei/tf1.1/. Training test 7, Training 1, test 7 . In Proceedings of the international conference on machine learning (ICML). prediction outcome of the processed test samples. To scale up influence functions to modern machine learning This could be because we explicitly build optimization into the architecture, as in MAML or Deep Equilibrium Models. In many cases, the distance between two neural nets can be more profitably defined in terms of the distance between the functions they represent, rather than the distance between weight vectors. above, keeping the grad_zs only makes sense if they can be loaded faster/ How can we explain the predictions of a black-box model? Understanding black-box predictions via influence functions. To scale up influence functions to modern machine learning settings, we develop a simple, efficient implementation that requires only oracle access to gradients and Hessian-vector products. Existing influence functions tackle this problem by using first-order approximations of the effect of removing a sample from the training set on model . How can we explain the predictions of a black-box model? We have two ways of measuring influence: Our first option is to delete the instance from the training data, retrain the model on the reduced training dataset and observe the difference in the model parameters or predictions (either individually or over the complete dataset). Tasha Nagamine, . Most weeks we will be targeting 2 hours of class time, but we have extra time allocated in case presentations run over. stream multilayer perceptrons), you can use straight-up JAX so that you understand everything that's going on. Strack, B., DeShazo, J. P., Gennings, C., Olmo, J. L., Ventura, S., Cios, K. J., and Clore, J. N. Impact of HbA1c measurement on hospital readmission rates: analysis of 70,000 clinical database patient records. We'll use the Hessian to diagnose slow convergence and interpret the dependence of a network's predictions on the training data. 2017. Highly overparameterized models can behave very differently from more traditional underparameterized ones. prediction outcome of the processed test samples. It is individual work. . Is a dict/json containting the influences calculated of all training data We show that even on non-convex and non-differentiable models where the theory breaks down, approximations to influence functions can still provide valuable information. The second mode is called calc_all_grad_then_test and influences. To get the correct test outcome of ship, the Helpful images from Are you sure you want to create this branch? Koh P, Liang P, 2017. Students are encouraged to attend synchronous lectures to ask questions, but may also attend office hours or use Piazza. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I. J., Harp, A., Irving, G., Isard, M., Jia, Y., Jzefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Man, D., Monga, R., Moore, S., Murray, D. G., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P. A., Vanhoucke, V., Vasudevan, V., Vigas, F. B., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., and Zheng, X. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. Y. LeCun, L. Bottou, G. B. Orr, and K.-R. Muller. /Length 5088 Differentiable Games (Lecture by Guodong Zhang) [Slides]. Understanding black-box predictions via influence functions. Deep learning via hessian-free optimization. Please download or close your previous search result export first before starting a new bulk export. We'll then consider how the gradient noise in SGD optimization can contribute an implicit regularization effect, Bayesian or non-Bayesian. We motivate second-order optimization of neural nets from several perspectives: minimizing second-order Taylor approximations, preconditioning, invariance, and proximal optimization. How can we explain the predictions of a black-box model? We have a reproducible, executable, and Dockerized version of these scripts on Codalab. The more recent Neural Tangent Kernel gives an elegant way to understand gradient descent dynamics in function space. ( , ?) You can get the default config by calling ptif.get_default_config(). the original paper linked here. Lage, E. Chen, J. , . Reconciling modern machine-learning practice and the classical bias-variance tradeoff. Programming languages & software engineering, Programming languages and software engineering, Designing AI Systems with Steerable Long-Term Dynamics, Using platform models responsibly: Developer tools with human-AI partnership at the center, [ICSE'22] TOGA: A Neural Method for Test Oracle Generation, Characterizing and Predicting Engagement of Blind and Low-Vision People with an Audio-Based Navigation App [Pre-recorded CHI 2022 presentation], Provably correct, asymptotically efficient, higher-order reverse-mode automatic differentiation [video], Closing remarks: Empowering software developers and mathematicians with next-generation AI, Research talks: AI for software development, MDETR: Modulated Detection for End-to-End Multi-Modal Understanding, Introducing Retiarii: A deep learning exploratory-training framework on NNI, Platform for Situated Intelligence Workshop | Day 2. Cook, R. D. and Weisberg, S. Characterizations of an empirical influence function for detecting influential cases in regression. The meta-optimizer has to confront many of the same challenges we've been dealing with in this course, so we can apply the insights to reverse engineer the solutions it picks. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. On linear models and convolutional neural networks, we demonstrate that influence functions are useful for multiple purposes: understanding model behavior, debugging models, detecting dataset errors, and even creating visually-indistinguishable training-set attacks. Imagenet classification with deep convolutional neural networks. SVM , .

Yankee Announcers Salaries, 25 Guineas In 1966 Worth Today, Why Did They Kill Quentin On One Tree Hill, What Does An Unexcused Absence Mean At School, Articles U