Background Estimation in Videos
Background estimation is a fundamental task of computer vision. Indeed, in problems such as tracking or object recognition in videos, a good background estimation is usually a prerequisite. This is because these subsequent tasks are much easier if we know where to look for interesting objects, in particular if we know where the foreground is (when we track objects, we are generally interested in foreground objects).
Publications in this research area
- Low-Rank Spatio-Temporal Video Segmentation, A. Newson, M. Tepper, G. Sapiro, BMVC 2015
- Multi-temporal Foreground Detection in Videos, M. Tepper, A. Newson, P. Sprechmann, G. Sapiro, ICIP 2015 — Project webpage
Here is an example of background/foreground estimation (thanks to Jose Lezama for appearing in this video):
Robust Principal Component Analysis (RPCA)
Candès et al. introduced an approach which is widely known as Robust Principal Component Analysis [1] (even if other works had previously proposed robustified PCAs [2]), which could be used for tasks such as background estimation. The central idea of this work is to robustify traditional Principal Component Analysis, a data analysis method which is very popular, but can break down in the presence of outliers. This is done by decomposing an input matrix (the video, in our case) into the sum of a low-rank and a sparse component. More precisely, we solve the following convex optimisation problem:
minL,S ‖X − L − S‖F2 + λ* ‖L‖* + λ1 ‖S‖1
where X is the input matrix (our video), L is the low-rank background component and S is the sparse foreground component. The norms ‖·‖F, ‖·‖* and ‖·‖1 correspond to the Frobenius matrix norm, the nuclear norm and the ℓ1 norm, respectively. The two parameters λ* and λ1 are tuning parameters.