Organizing Egocentric Videos
of Daily Living Activities

Accepted at Pattern Recognition, Elsevier


DOI information: 10.1016/j.patcog.2017.07.010

A. Ortis, G. M. Farinella, S. Battiato
Image Processing Laboratory - University of Catania
{ortis, gfarinella, battiato}@unict.it

V. D'Amico, L. Addesso, G. Torrisi
JOL WAVE - Telecom Italia
{valeria1.damico, luca.addesso, giovanni.torrisi}@telecomitalia.it

Highlights:

Abstract:

Egocentric videos are becoming popular because give the possibility to observe the scene ow from the user's point of view (First Person Vision). Among the assistive applications in which those videos can be exploited is the daily living monitoring of a user that is wearing the camera. In this paper we propose a system useful for the automatic organization of the egocentric videos acquired by a user over different days. The system is able to perform an unsuper vised segmentation of each egocentric video in chapters by considering visual content. The video segments related to the different days are hence linked to produce graphs which are coherent with respect to the context in which the user acts. Experiments on two different datasets demonstrate the effectiveness of the proposed approach which outperforms with a good margin the state of the art both in accuracy and computational time.

Download input video sequences

This archive contains both the input egocentric videos and their segmentation Ground Truth.
Download the egocentric video dataset (2016)


Daily Living Monitoring - Visual Results

Working Day Home Day Office Day
Working Day Home Day Office Day

Daily Living Monitorig - Experimental Results

Intrafow performances for wearable devices, computed using [1] and the proposed approach. Each test is evaluated considering the accuracy of the segmentation (Q), the computation time (T) and the number of the scenes detected by the algorithm (S). The accuracy is measured as the percentage of well classified frames with respect to the Ground Truth. The measured time includes the feature extraction process.

Intraflow proposed in [1] Proposed Intraflow Approach Proposed Intraflow Approach
with Segmentation Refinement
Video Scenes Q S T Q S Q S T
HomeDay1 3 62,5% 8 20'45'' 77,5% 4 92,5% 3 1'23''
HomeDay2 3 71,6% 3 20'18'' 80,3% 4 94,5% 3 1'46''
HomeDay3 3 64,3% 5 19'03'' 79,7% 5 94,3% 3 1'21''
HomeDay4 3 84,4% 3 8'36'' 91,8% 3 85,4% 2 36''
WorkingDay1 4 95,7% 5 16'16'' 98,4% 5 99,5% 5 1'22''
WorkingDay2 4 82,5% 5 15'15'' 98,9% 5 100% 4 1'08''
WorkingDay3 5 98,7% 6 19'02'' 99,2% 6 99,4% 5 1'29''
OfficeDay1 3 23,0% 5 24'8'' 55,3% 19 66,9% 2 2'39''
OfficeDay2 2 59,7% 2 10'25'' 90,0% 3 98,7% 2 1'26''
OfficeDay3 3 57,2% 4 13'28'' 83,6% 10 96,3% 3 1'49''
OfficeDay4 3 52,0% 4 11'37'' 79,5% 5 84,1% 4 1'41''
OfficeDay5 3 70,7% 3 8'35'' 86,7% 4 95,9% 3 1'21''
OfficeDay6 3 78.8% 3 9'33'' 61,5% 5 94,5% 4 1'34''
Average 69,3% 15'9'' 83,3% 91,8% 1'31''



Popularity Estimation - Visual Results

Considering the improvements achieved by the proposed method in the context of egocentric videos on both, intraflow and between flow analysis, we have tested the approach to solve the popularity estimation problem defined in [1].


Foosball Room Meeting SAgata
Foosball Room Meeting SAgata

Popularity Estimation - Experimental Results

To evaluate the performances of the compared methods, we compute three measures. Two of them are measures proposed in [1]. Specifically, for each clustering step we compute:

From the above scores, the weighted mean of the ratios Pa/Pr and Pg/Pr over all the clustering steps are computed in [1]. Thus, the ratio Pa/Pr provides a score for the popularty estimation, whereas the ratio Pg/Pr assesses the visual content of the videos in the popular cluster. We introduced a third evaluation measure that takes into account the number of outliers in the most popular cluster. Let Po be the number of wrong videos in the popular cluster (outliers). From this score, we compute the weighted mean of the ratio Po/Pr over all the clustering steps, where the weights are given by the length of the segmented blocks (i.e., the weights are computed as suggested in [1]). This value can be considered as a percentage of the presence of outliers in the most popular cluster inferred by the system. The aim is to have this value as lower as possible.

The obtained results with respect to the different scenarios are reported in the Table below.

[1] Proposed method
Scenario Devices Models Pa/Pr Pg/Pr Po/Pr Pa/Pr Pg/Pr Po/Pr
Foosball 4 2 1.02 1 0.023 1 1 0
Meeting 2 2 1.01 0.99 0.020 0.99 0.99 0.018
Meeting 4 4 0.99 0.95 0.033 0.93 0.93 0
Meeting 5 5 0.89 0.76 0.131 0.70 0.70 0.006
SAgata 7 6 1.05 1 0.050 0.99 0.99 0


References

[1] A. Ortis, G.M. Farinella, V. D'Amico, G. Torrisi, L. Addesso, and S. Battiato, RECfusion: Automatic Video Curation Driven by Visual Content Popularity, Proc. ACM Multimedia, pg. 1179-1182 (2015).