本文为美国肯塔基大学（作者：Vijay Venkatesh Mahalingam）的博士论文，共128页。

数字修复是利用周围区域的信息来填充图像或视频中缺失区域的技术。这项技术在错误恢复、多媒体编辑和视频隐私保护等领域有着广泛的应用。

本论文主要探讨三种与现有及新兴修补演算法及应用相关的重大挑战。影响的三个关键领域是1）图像修复算法的结构，2）快速有效的基于对象的视频修复框架，3）大面积图像修复算法的感知评估。

现有图像修复算法修复缺失信息的主要方法之一是遵循两个阶段的过程。通过结构完成步骤，勾勒孔洞区域的边界，然后在纹理完成过程中采用先进的纹理合成方法。虽然纹理合成阶段很重要，但可以说结构完成步骤是提高感知图像修复质量的重要组成部分。为此，我们提出了一种以对称性为主要特征的全局结构补全算法。虽然现有的对称性完成方法需要先验信息，但我们的方法采用非参数方法，利用曲率的不变性来完成缺失的边界。

将我们的注意力从图像转向视频修复，我们很容易观察到现有的视频修复技术已经发展成为图像修复技术的延伸。结果，它们遭受了各种各样的缺点，其中包括不能处理大面积的丢失的时空区域、显著地慢执行时间使得其不适用于交互使用以及时空伪影的存在。为了解决这些主要问题，我们提出了一种基于对象框架的改进视频修复算法性能的方法。本文提出了一种模块化的修复方案，首先利用获取的背景模型将视频分割成多个对象，然后对静态背景区域和动态前景区域进行修复。对于静态背景区域的修复，我们使用简单的背景替换和偶尔的图像修复。为了修复动态移动的前景区域，我们在动态规划框架下引入了一种新的基于滑动窗口的相异度度量方法。该技术可以有效地修复大面积的遮挡区域，修复多帧内完全缺失的物体，改变物体的大小和姿态，并且具有最小的模糊和运动伪影。

最后，我们将重点放在大面积图像修复算法感知质量评价的实验研究上。大面积修复技术的感知在本质上是一个主观的过程，但以往的研究还没有从人类视觉系统的主观本质出发。我们使用24名受试者的眼球追踪装置进行主观实验，分析修复对人类注视的影响。实验表明，修复伪影的存在直接影响了无偏观察者的注视，这实际上直接影响了观察者的主观评价。具体地说，当修复伪影很明显时，修复图像空洞区域的注视能量表现出明显的偏离正常行为。

Digital inpainting is the technique of filling in the missing regions of an image or a video using information from surrounding area. This technique has found widespread use in applications such as restoration, error recovery, multimedia editing, and video privacy protection. This dissertation addresses three significant challenges associated with the existing and emerging inpainting algorithms and applications. The three key areas of impact are 1) Structure completion for image inpainting algorithms, 2) Fast and efficient object based video inpainting framework and 3) Perceptual evaluation of large area image inpainting algorithms. One of the main approach of existing image inpainting algorithms in completing the missing information is to follow a two stage process. A structure completion step, to complete the boundaries of regions in the hole area, followed by texture completion process using advanced texture synthesis methods. While the texture synthesis stage is important, it can be argued that structure completion aspect is a vital component in improving the perceptual image inpainting quality. To this end, we introduce a global structure completion algorithm for completion of missing boundaries using symmetry as the key feature. While existing methods for symmetry completion require a-priori information, our method takes a non-parametric approach by utilizing the invariant nature of curvature to complete missing boundaries. Turning our attention from image to video inpainting, we readily observe that existing video inpainting techniques have evolved as an extension of image inpainting techniques. As a result, they suffer from various shortcoming including, among others, inability to handle large missing spatio-temporal regions, significantly slow execution time making it impractical for interactive use and presence of temporaland spatial artifacts. To address these major challenges, we propose a fundamentally different method based on object based framework for improving the performance of video inpainting algorithms. We introduce a modular inpainting scheme in which we first segment the video into constituent objects by using acquired background models followed by inpainting of static background regions and dynamic foreground regions. For static background region inpainting, we use a simple background replacement and occasional image inpainting. To inpaint dynamic moving foreground regions, we introduce a novel sliding-window based dissimilarity measure in a dynamic programming framework. This technique can effectively inpaint large regions of occlusions, inpaint objects that are completely missing for several frames, change in size and pose and has minimal blurring and motion artifacts. Finally we direct our focus on experimental studies related to perceptual quality evaluation of large area image inpainting algorithms. The perceptual quality of large area inpainting technique is inherently a subjective process and yet no previous research has been carried out by taking the subjective nature of the Human Visual System (HVS). We perform subjective experiments using eye-tracking device involving 24 subjects to analyze the effect of inpainting on human gaze. We experimentally show that the presence of inpainting artifacts directly impacts the gaze of an unbiased observer and this in effect has a direct bearing on the subjective rating of the observer. Specifically, we show that the gaze energy in the hole regions of an inpainted image show marked deviations from normal behavior when the inpainting artifacts are readily apparent.

1.       引言

2. 文献回顾

3. 对称完成与全局图像修补

4. 高效的基于对象的视频修复方法

5. 图像修复的感知评估

6. 结论与展望

更多精彩文章请关注公众号：