Back to overview

Sketch-Based Image Synthesis

English title Sketch-Based Image Synthesis
Applicant Favaro Paolo
Number 156253
Funding scheme Project funding (Div. I-III)
Research institution Institut für Informatik Universität Bern
Institution of higher education University of Berne - BE
Main discipline Information Technology
Start/End 01.02.2015 - 31.01.2019
Approved amount 367'488.00
Show all

All Disciplines (2)

Information Technology

Keywords (4)

image rendering; sketch rectification; categorization; sketch parsing

Lay Summary (Italian)

Questa ricerca si prefigge di generate immagini realistiche a partire da uno sketch.Il sistema proposto dovrebbe anche correggere inaccuratezze dello sketch iniziale (per es. asimmetrie, prospettive incorrette, ripetizioni distorte di pattern ecc.).
Lay summary

L'abilita' di esprimerci o di essere creativi e' talvolta limitata dalla nostra stessa abilita' tecnica. Un modo molto efficace per illustrare idee e' di rappresentarle tramite un'immagine. Sfortunatamente pero' non tutti possono farlo e produrre un risultato convincente dell'idea originale. Noi quindi proponiamo di sviluppare uno strumento computazionale che puo' assistere un autore nella realizzazione del suo concetto. Il nostro sistema usera' uno sketch iniziale (anche inaccurato) fornito dall'autore e automaticamente produrra' un'immagine realistica dello stesso. Inoltre il sistema introdurra' aggiustamenti per rendere lo sketch iniziale realistico nel caso vi siano distorsioni o inesattezze.

Direct link to Lay Summary Last update: 14.03.2019

Lay Summary (English)

This research aims at generating realistic pictures from sketches. The proposed systemshould also correct for inaccuracies in the initial sketch (such as asymmetries, incorrectperspective, distorted repeating patterns etc).
Lay summary

The ability to express ourselves or to be creative is sometimes limited by our own technical skills. One very powerful way to illustrate ideas is through images. Unfortunately, however, not all of us can do so and produce a convincing rendering of the original idea. We therefore propose the development of a computational tool that can aid authors in implementing their concepts. Our tool will take an inaccurate initial sketch of the concept and then automatically produce a realistic rendering of that sketch. The tool will also introduce adjustments to make the rendering realistic if the original sketch had distortions or flaws.

Direct link to Lay Summary Last update: 14.03.2019

Responsible applicant and co-applicants



Scientific events

Active participation

Title Type of contribution Title of article or contribution Date Place Persons involved
Seminar at the Department of Engineering Science, University of Oxford Individual talk Towards Unsupervised Learning 23.11.2018 Oxford, Great Britain and Northern Ireland Favaro Paolo;
The Rank Prize Funds - Symposium on Geometry and Uncertainty in Deep Learning for Computer Vision Individual talk Beyond Supervised Learning 20.08.2018 Grasmere, Great Britain and Northern Ireland Favaro Paolo;
SIGGRAPH 2018 Talk given at a conference FaceShop: Deep Sketch-based Image Editing 12.08.2018 Vancouver, Canada Zwicker Matthias;
Workshop on Beyond Supervised Learning - CVPR 2018 Talk given at a conference Unsupervised Learning and Data Redundancy 22.06.2018 Salt Lake City, Utah, United States of America Favaro Paolo;
computer vision and pattern recognition (CVPR) conference 2018 Poster Disentangling Factors of Variation by Mixing Them 18.06.2018 Salt Lake City Utah, United States of America Favaro Paolo; Hu Qiyang;
computer vision and pattern recognition (CVPR) conference 2018 Poster Understanding Degeneracies and Ambiguities in Attribute Transfer 18.06.2018 Salt Lake City, Utah, United States of America Hu Qiyang; Favaro Paolo;
Expressive, Eurographics Conference 2017 Talk given at a conference SmartSketcher: Sketch-based Image Retrieval with Dynamic Semantic Re-ranking 29.07.2017 Los Angeles, California, United States of America Portenier Tiziano;
Vision, Modeling, and Visualization Conference Poster 3D Face Reconstruction with Silhouette Constraints 10.10.2016 BAYREUTH, Germany Hu Qiyang;

Associated projects

Number Title Start Funding scheme


In this proposal we investigate the task of generating realistic pictures from an abstract concept. Currently, this is a very lengthy and frustrating process. Concepts can be illustrated via sketches, but sketches are rarely enough to portray the exact vision of the author. Only the final rendering of the sketch into a realistic image (typically, via a labour-intensive manual process or costly computer graphics modeling) can convey the correct impression. Therefore, this creative process can become a slow, expensive, and tedious feedback cycle. Fortunately, nowadays we have access to a vast array of real images, and it is plausible to think that some of these images contain parts that can be efficiently used to build a realistic rendering of a sketch. Indeed, there exists recent work that exploits this idea by retrieving, cutting, and pasting objects from real images to then compose a convincing result. These methods rely on finding image content that can be used ``as is'' in the final rendering. However, this assumption is quite debatable, even with current large data collections, because of the high dimensionality of the space of all possible views, deformations, and illuminations of each object instance. One can instead more conceivably find a close match of just one aspect of the sketch, i.e., either the specific object instance (e.g., a smiling face), or the specific viewpoint, or the specific illumination. Unluckily, as of now, there does not exist a system that can accurately modify the content found in real images both geometrically and photometrically to match a sketch.We propose to develop novel solutions for generating realistic images from sketches. We envision an interactive system that finds and carefully matches image content to a sketch, by using a combination of geometric and photometric deformations that are low-order globally and piecewise smooth locally. Moreover, to limit the magnitude of the geometric deformations, our solution exploits a hierarchical matching of sketch parts at different scales. Our assumption is that content in real images can be matched closely to the sketch at some scale. For example, it might not be possible to find a real image of a specific viewpoint of a smiling face, but it might be more likely to find real images of a specific viewpoint of parts of the face, e.g., the nose, the eyes, the mouth and so on. Because we strongly rely on the input sketch, we also introduce an important novel step in its analysis. To assist the author with creating an accurate sketch, our proposed system will automatically detect symmetries, repeating patterns, and perspective effects, and allow the user to amend the sketch by enforcing exact regularity of these properties. Moreover, the system will use these properties in the matching to real images to determine viewpoints, textural patterns and dealing with occlusions.