What’s new

Overview

Introduction

The Video­Com­ple­tion pro­ject in­tro­duces the first bench­mark for video-com­ple­tion meth­ods. We pre­sent re­sults for dif­fer­ent meth­ods on a range of di­verse test se­quences which are avail­able for view­ing on a player equipped with a mov­able zoom re­gion. Ad­di­tion­ally, we pro­vide the re­sults of an ob­jec­tive analy­sis us­ing qual­ity met­rics that were care­fully se­lected in our study on video-com­ple­tion per­cep­tual qual­ity. We be­lieve that our work can help rank ex­ist­ing meth­ods and as­sist de­vel­op­ers of new gen­eral-pur­pose video-com­ple­tion meth­ods.

Data set

Our cur­rent data set con­sists of 7 video se­quences with ground-truth com­ple­tion re­sults. We con­sider ob­ject re­moval, so the test se­quences are con­structed by com­pos­ing var­i­ous fore­ground ob­jects over a set of back­ground videos. Some of these back­ground videos in­clude left-view se­quences from the stereo­scopic-video data set RMIT3dv [1 ]. As fore­ground ob­jects we use those em­ployed in the video-mat­ting bench­mark [2 ] as well as sev­eral 3D mod­els. To seam­lessly in­sert a 3D model in a back­ground video we use Blender [3 ] mo­tion-track­ing tools. Each video-com­ple­tion method takes the com­pos­ited se­quence and the cor­re­spond­ing ob­ject mask as in­put.

Evaluation Methodology

Video-com­ple­tion re­sults are sel­dom ex­plic­itly ex­pected to ad­here to ground truth and are usu­ally judged only by their plau­si­bil­ity, which is as­sessed by a hu­man ob­server. It makes ob­jec­tive qual­ity as­sess­ment of video com­ple­tion an in­her­ent prob­lem. How­ever, by re­lax­ing the re­quire­ment of com­plete ad­her­ence to ground truth we can in­crease cor­re­la­tion with per­cep­tual com­ple­tion qual­ity. This bench­mark em­ploys four qual­ity met­rics: MS-DSSIM, MS-DSSIMdt, MS-C_DSSIM, MS-C_DSSIMdt. Thor­ough de­scrip­tion and com­par­a­tive analy­sis of these and other met­rics can be found in our pa­per (to be pub­lished soon).

MS-DSSIM met­ric mea­sures ad­her­ence of com­ple­tion re­sult V to ground truth video V_ref in a multi-scale fash­ion with scale weights de­ter­mined us­ing per­cep­tual qual­ity data. It is based on the struc­tural sim­i­lar­ity in­dex (SSIM) [4 ] val­ues com­puted for all ×ばつ9 lu­mi­nance patches P(x) within the spa­tio-tem­po­ral hole .

Here su­per­script i de­notes the level of the Gauss­ian pyra­mid—that is, V_ref⁰ is the orig­i­nal ground-truth video, and V_ref¹ is the video blurred and sub­sam­pled by a fac­tor of two in both spa­tial di­men­sions.

MS-DSSIMdt met­ric cap­tures tem­po­ral co­herency along ground-truth op­ti­cal-flow vec­tors .

MS-C_DSSIM re­lies on the as­sump­tion that com­ple­tion re­sult should be lo­cally sim­i­lar to the ground truth—that is, each patch P(x) within the spa­tio-tem­po­ral hole should have a sim­i­lar ground-truth patch P_ref(y).

MS-C_DSSIMdt is a tem­po­ral sta­bil­ity met­ric that uses the same as­sump­tions as MS-C_DSSIM. Es­sen­tially it cap­tures the changes in patch ap­pear­ance from frame to frame, as op­posed to eval­u­at­ing con­sis­tency with ground-truth op­ti­cal flow us­ing MS-DSSIMdt. To do so we find for a given patch the most sim­i­lar patch from the pre­vi­ous frame within a cer­tain win­dow, com­pute the dis­tances from these patches to the most sim­i­lar ground-truth patches and then com­pare the re­spec­tive dis­tances.

Here de­notes a square win­dow of pix­els (we use equal to 1/​10th of the frame width) spa­tially cen­tered at and lo­cated in the pre­vi­ous frame.

Ex­act com­pu­ta­tion of MS-C_DSSIM and MS-C_DSSIMdt quickly be­comes im­prac­ti­cal for larger spa­tio-tem­po­ral holes, so we re­sort to ap­prox­i­mate so­lu­tions based on the Patch­Match [5 ] al­go­rithm.

Participate

We in­vite de­vel­op­ers of video-com­ple­tion meth­ods to use our bench­mark. We can eval­u­ate the sub­mit­ted data and re­port qual­ity scores to the de­vel­oper. In cases where the de­vel­oper specif­i­cally grants per­mis­sion, we will pub­lish the re­sults on our site. The test se­quences with the re­spec­tive com­ple­tion masks are avail­able for down­load: Deck, Li­brary, Foun­tain, Wires, Tower, Sky­scrap­ers, Sign.

For eval­u­a­tion re­quests or if you have any ques­tions or sug­ges­tions please feel free to con­tact us by email: abokov@graph­ics.cs.msu.ru.

Evaluation

Overall Plots

References