CSCI 6900: Advanced Data Analytics: Statistical Learning and Optimization

Overview: The last decade has witnessed an increasing amount of data generated by modern applications, such as daily photos, videos, and medical image scans. There is a great need for techniques to learn from this data. In this course, we will discuss advanced topics in data analysis, with an emphasis on statistical learning and related optimization problems. In particular, we will cover learning methods (e.g., image regression, dictionary learning, random forests, and deep learning), numerical optimization approaches (e.g., the adjoint method, coordinate descent, and stochastic gradient descent), and their connections. The applications include prediction, classification, segmentation, and other tasks in image analysis. This course is targeted towards graduate students, and the lectures will be based on articles from journals and conferences in the field of computer vision and medical image analysis.

Course Information

• Class meetings: TR 02:00pm – 03:15pm @ LIFE SCI C112, W 02:30 pm – 03:20 pm @ Boyd 0208

• Instructor: Yi Hong (yihong -at- cs.uga.edu, office: Boyd 616)

• Office hours: W 3:30pm - 4:30pm, R 11am - 12pm, or by appointment

• Course webpage: http://cs.uga.edu/~yihong/CSCI6900-Fall2016.html

Topics

• Image and shape regression: cross-sectional and longitudinal studies

• Sparse representation, dictionary learning, and low-rank approximation

• Random forests: classification forests and regression forests

• Deep learning: convolutional neural networks (CNNs) and recurrent neural networks (RNNs)

Prerequisites

No prior experience in computer vision or medical image analysis is required, while some exposure to image analysis, machine learning, or numerical computation is highly recommended.

Grading

This course will mix lectures with reading and discussion of related research papers and projects. Students should present their selected papers and work on a project related to the topics discussed in this course. In particular, the grading will be based on

• Project (40%), including proposal (5%), update (5%), presentation (15%), and write-up (15%)

• Paper presentation (20%) and leading the discussion (10%)

• Paper summaries (20%)

• Participation (10%)

There is no exam.

Tentative Schedule

Reading List

Papers will be assigned on a first-come-first-serve basis. You may also propose a paper that is not listed, but you must get it approved.

Image and Shape Regression

1. Zitova and Flusser, Image registration methods: a survey, Image and Vision Computing 2003.
2. Beg et al., Computing large deformation metric mappings via geodesic flows of diffeomorphisms, IJCV 2005.
3. Hart et al., An optimal control approach for deformable registration, CVPR Workshop 2009.

Geodesic regression

1. Niethammer et al., Geodesic regression for image time-series, MICCAI 2011.
2. Fletcher, T., Geodesic regression and the theory of least squares on Riemannian manifolds, IJCV 2013.
3. Singh et al., A vector momenta formulation of diffeomorphisms for improved geodesic regression and atlas construction, ISBI 2013.
4. Fishbaugh et al., Geodesic shape regression in the framework of currents, IPMI 2013.

Higher-order models

1. Hinkle et al., Intrinsic polynomials for regression on Riemannian manifolds, JMIV 2014.
2. Singh et al., Splines for diffeomorphisms, MedIA 2015.
3. Hong et al., Parametric regression on the Grassmannian, TPAMI 2016.

Longitudinal studies

1. Singh et al., A hierarchical geodesic model for diffeomorphic longitudinal shape analysis, IPMI 2013.
2. Durrleman et al., Toward a comprehensive framework for the spatiotemporal statistical analysis of longitudinal shape data, IJCV 2013.
3. Schiratti et al., A mixed effect model with time reparametrization for longitudinal univariate manifold-valued data, IPMI 2015.

Kernel regression

1. Davis et al., Population shape regression from random design data, ICCV 2007.
2. Banerjee et al., A nonlinear regression technique for manifold valued data with applications to medical image analysis, CVPR 2016.

Sparse Representation, Dictionary Learning, and Low-Rand Approximation

Sparse coding & Dictionary learning

1. Aharon et al., K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation, TSP 2006.
2. Elad and Aharon, Image denoising via sparse and redundant representation over learned dictionaries, TIP 2006.
3. Mairal et al., Online dictionary learning for sparse coding, ICML 2009.
4. Wright et al., Robust face recognition via sparse representation, TPAMI 2009.
5. Yang et al., Image super-resolution via sparse representation, TIP 2010.
6. Wang et al. Semi-coupled dictionary learning with application to image super-resolution and photo-sketch synthesis, CVPR 2012.
7. Mairal et al. Task-driven dictionary learning, TPAMI 2012.
8. Gangeh et al., Supervised dictionary learning and sparse representation - a review, arXiv:1052.05928 2015.

Low rank approximation

1. Wright et al., Robust principal component analysis: Exact recovery of corrupted low-rank matrices by convex optimization, NIPS 2009.
2. Cabral et al., Matrix completion for weakly-supervised multi-label image classification, TPAMI 2012.
3. Zhang et al., Learning structured low-rank representation for image classification, CVPR 2013.
4. Liu et al., Low-rank atlas image analyzes in the presence of pathologies, TMI 2015.

Random Forests

Local optimization

1. Dollar and Zitnick, Structured forests for fast edge detection, ICCV 2013.
2. Konukoglu et al., Neighborhood approximation using randomized forests, MedIA 2013.
3. Fanello et al., Filter forests for learning data-dependent convolutional kernels, CVPR 2014.
4. Schulter et al., Fast and accurate image upscaling with super-resolution forests, CVPR 2015.

Global optimization

1. Schulter et al., Alternating decision forest, CVPR 2013.
2. Schulter et al., Alternating regression forests for object detection and pose estimation, ICCV 2013.
3. Ren et al., Global refinement of random forest, CVPR 2015.
4. Kontschieder et al., Deep neural decision forest, ICCV 2015.

Understanding random forests

1. Denil et al., Narrowing the gap: Random forests in theory and in practice, ICML 2014.
2. Scornet et al., Consistency of random forests, The Annals of Statistics 2015.
3. Scornet et al., On the asymptotics of random forests, Journal of Multivariate Analysis 2016.

Deep Learning

Overview

1. LeCun et al., Deep learning, Nature 2015.
2. Schmidhuber et al., Deep learning in neural networks: An overview, Neural Networks, 2015.
3. Lipton et al., A critical review of recurrent neural networks for sequence learning, arXiv:1506.00019 2015.

Convolutional neural networks

1. Alex et al., ImageNet classification with deep convolutional neural networks, NIPS 2012.
2. Szegedy et al., Going deeper with convolutions, CVPR 2015.
3. He et al., Deep residual learning for image recognition, CVPR 2016.

Recurrent neural network

1. Gregor et al., DRAW: A recurrent neural network for image generation, arXiv:1502.04623 2015.
2. Srivastava et al., Unsupervised learning of video representations using LSTMs, ICML 2015.

Fully Convolutional Networks

1. Long et al., Fully convolutional networks for semantic segmentation, CVPR 2015.
2. Wulfmeier et al., Maximum entropy deep inverse reinforcement learning, arxiv:1507.04888v3 2016.

Autoencoder

1. Vincent et al., Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion, JMLR 2010.
2. Le et al., Building high-level features using large scale unsupervised learning, ICML 2012.
3. Kingma et al., Auto-encoding variational Bayes, ICLR 2014.
4. Burda et al., Importance weighted autoencoders, ICLR 2016.

Deep structured models

1. Schwing and Urtasun, Fully connected deep structured networks, arXiv:1503.02351 2015.
2. Chen et al., Learning deep structured models, ICML 2015.
3. Zheng et al., Conditional random fields as recurrent neural networks, ICCV 2015.

Generative model

1. Goodfellow et al., Generative adversarial nets, NIPS 2014.

Training deep neural networks

1. Srivastava et al., Dropout: A simple way to prevent neural networks from overfitting, JMLR 2014.
2. Ioffe et al., Batch normalization: Accelerating deep network training by reducing internal covariate shift, arXiv:1502.03167 2015.

Understanding deep learning

1. Szegedy et al., Intriguing properties of neural networks, arXiv:1312.6199 2013.
2. Goodfellow et al., Explaining and harnessing adversarial examples, arXiv preprint arXiv:1412.6572 2014.
3. Zeiler et al., Visualizing and understanding convolutional networks, ECCV 2014.
4. Zhou et al., Learning deep feature for scene recognition using places database, NIPS 2014.
5. Nguyen et al., Deep neural networks are easily fooled: High confidence predictions for unrecognizable images, CVPR 2015.
6. Yosinski et al., Understanding neural networks through deep visualization, arXiv:1506.06579 2015.

Reference Books and Resources

• Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Second Edition, Springer.

• Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. In preparation for MIT Press. http://www.deeplearningbook.org.

• Julien Mairal, Francis Bach, and Jean Ponce. Sparse Modeling for Image and Vision Processing. Now Publishers, 2014.

• Antonio Criminisi, Jamie Shotton, and Ender Konukoglu. Decision Forests for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning. Microsoft Research technical report TR-2011-114.

• Antonio Criminisi and Jamie Shotton. Decision Forests for Computer Vision and Medical Image Analysis. Springer 2013.

Useful Links

Dectionary Learning and Low-Rank Approximation

• SPAMS

• Dictionary Learning Tools for Matlab

• Semi-Coupled Dictionary Learning

• LRSLibrary

• Low-Rank Matrix Recovery and Completion

• Low-Rank Atlas-to-Image Registration

Random Forests

• Sherwood C and C# code library for decision forests

• Regression tree package by Pierre Geurts

• Scikit-learn

• CloudForest

Deep Learning

• Caffe

• Torch

• Theano

• Tensorflow

• Keras (High-level wrapper for Theano and Tensorflow)

Abbreviations

• Journals – IJCV: International Journal of Computer Vision,   TPAMI: IEEE Transactions on Pattern Analysis and Machine Intelligence,   TSP: IEEE Transactions on Signal Processing,   JMIV: Journal of Mathematical Imaging and Vision,   TIP: IEEE Transactions on Image Processing,   TMI: IEEE Transactions on Medical Imaging,   MedIA: Medical Image Analysis,   JMLR: Journal of Machine Learning Research.

• Conferences – MICCAI: Medical Image Computing and Computer Assisted Intervention,   IPMI: Information Processing in Medical Imaging,   ISBI: International Symposium on Biomedical Imaging,   ICML: International Conference on Machine Learning, CVPR: IEEE Conference on Computer Vision and Pattern Recognition,   ECCV: European conference on Computer Vision,   ICCV: International Conference on Computer Vision,   NIPS: Neural Information Processing Systems,   ICLR: International Conference on Learning Representations.

Disclaimer

The instructor reserves the right to make changes to the syllabus, including project due dates. These changes will be announced as early as possible.