Deep Image Prior

1 minute read

<span class="timestamp-wrapper"><span class="timestamp">[2019-05-20 Mon] </span></span> Paper: Deep Image Prior
  • https://arxiv.org/abs/1711.10925
    • Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky
  • Webpage for the project, with short summary and example images
  • Much longer version with more images
  • Supplementary material
  • A bit of a mind-fuck, i.e. interesting
  • Talks about the impedance of the network and how is has to be matched to the image it is trying to restore
  • "we show that a randomly-initialized neural network can be used as a handcrafted prior with excellent results in standard inverse problems such as denoising, superresolution, and inpainting."
  • "generalization requires the structure of the network to “resonate” with the structure of the data."
  • How it works, denoising example
    • A NN is given as input pure noise: z
    • Adjust the weights of the network: θ
    • Minimize the difference between the output of the network x = fθ(z) and the image to be denoised x0
    • Stop the network after a few thousand iterations (minutes on GPU)
  • In short
    • The NN tries to create an image that is identical to the corrupted one x0 – given only noise as input and the freedom to change the weights in the convolutional kernels
    • But, given finite time, and without skip connections, it ends up creating a denoised, upsampled, or de-inpainted image x*
  • The architecture of the network is, effectively the prior, and because it is a deep network it is a deep prior
  • The network is not trained in advance and has not see any other images
  • When upsampling it cannot hallucinate fine details because it hasn't see those in it's life
  • Elegant, because it does not need data
  • I think of it as running a NN in passive mode, unharged, as a slave – normally the NN is first trained, charged, and will spit out things that look like what it has been trained on, will hallucinate
  • Every architecture will have it's own characteristic impedance
  • What works best for restoration seems to be U-nets without skip connections