Project

Back to overview

Data-driven Modeling of 3D Objects with Functional Parts

English title Data-driven Modeling of 3D Objects with Functional Parts
Applicant Zwicker Matthias
Number 169151
Funding scheme Project funding (Div. I-III)
Research institution Department of Computer Science University of Maryland
Institution of higher education University of Berne - BE
Main discipline Information Technology
Start/End 01.01.2017 - 31.12.2019
Approved amount 346'926.00
Show all

Keywords (4)

3D Reconstruction; Data-driven Modeling; Computer Science; Computer Graphics

Lay Summary (German)

Lead
Mit heutigen Werkzeugen ist das Design und die Konstruktion von 3D-Computergrafik Inhalten wie 3D-Formen und Animationen kompliziert und teuer. Ziel dieses Projektes ist es, den Design-Prozess durch die Erfassung und Verarbeitung von 3D-Daten aus der physischen Welt zu vereinfachen. Wir entwickeln mathematische und algorithmische Werkzeuge, die Benutzern ermöglichen, einfach und schnell neue 3D-Formen und Animationen mit 3D-Objekten mit funktionalen Teilen zu erstellen.
Lay summary

3D-Modellierung mit heutigen Computergrafik Werkzeugen ist sehr mühsam, und erfordert Expertenwissen und
spezielle Ausbildung. Beispielsweise ist die Erzeugung von Computergrafikbildern für die Filmproduktion sehr anspruchsvoll. Es erfodert immer noch viel Handarbeit von hochqualifizierten und spezialisierten Künstlern, um die in diesen Anwendungen benötigten 3D-Objekte zu modellieren und animieren. Dieses zeitraubende und
kostspielige Verfahren beschränkt den Einsatz von Computergrafiktechniken oft auf große Unternehmen mit ausreichender Finanzierung.  Die Verwendung von 3D-Computergrafik in anderen Arten von visuellen Medien mit kleineren Produktionsbudgets, wie Bildungsmedien, Blogs oder Journalismus, ist viel weniger verbreitet.

Ziel des vorgeschlagenen Projektes ist es daher, den Modellierungsprozess für 3D-Computergrafikinhalte zu vereinfachen. Wir verfolgen dabei einen "data-driven" Ansatz, der darauf beruht, existierenden Daten zu durchsuchen, bearbeiten und verändern. Weiter sollen Daten direkt aus der physichen Welt aufgenommen und aufbereitet werden. Wir entwickeln dann mathematische und algorithmische Methoden, welche die Repräsentation und Bearbeitung dieser Daten ermöglichen. Das Ziel ist es, einfache Werkzeuge und Schnittstellen zu entwerfen, so dass schliesslich Benutzer effektiv mit diesen Daten arbeiten und ihre Design-Vorstellungen umsetzen können. Insbesondere wird dieses Projekt ermöglichen, 3D-Objekte mit funktionalen Teilen zu erstellen.
Direct link to Lay Summary Last update: 02.12.2016

Lay Summary (English)

Lead
This project addresses the modeling bottleneck in Computer Graphics. Today, the design and construction of 3D Computer Graphics content such as 3D shapes and animations is complicated and expensive. The goal of this project is to simplify the content creation process by acquiring and processing 3D data from the physical world. We will develop sophisticated mathematical and algorithmic tools to allow users to create novel shapes and animations including 3D objects with functional parts.
Lay summary

3D content creation with today’s computer graphics tools is highly laborious and requires expert knowledge and
training. For example, generating computer graphics imagery (CGI) for movie production involves sophisticated
and complex content creation pipelines. Highly skilled and specialized artists perform significant amounts of manual
labor to model, texture, and animate the 3D objects required in these applications. This time consuming and
costly process often limits the use of computer graphics techniques to large companies with sufficient funding
sources. The proliferation of effective computer graphics applications, however, is severely limited by this modeling
bottleneck. Usage of CGI is much less widespread in many other types of visual media with smaller production
budgets, such as educational media, blogs, or journalism.

Hence, the objective of the proposed project is to simplify the modeling process for computer graphics content.
Ultimately, the outcomes of this project will make visual media production based on computer graphics available
to non-specialists, fostering the development and proliferation of new types of visual media, and making visual
storytelling using 3D computer graphics widely accessible. Our approach will leverage the concept of data-driven
modeling. This means that users can browse, retrieve, edit, and recombine content stored in rich databases in
intuitive ways, instead of modeling desired objects from scratch. In general, the challenges for data-driven modeling include data acquisition, the development of mathematical representations that expose the variability and thedegrees of freedom inherent in the data, and the design of intuitive, interactive editing tools.

 

Direct link to Lay Summary Last update: 02.12.2016

Responsible applicant and co-applicants

Employees

Publications

Publication
Point2Sequence: Learning the Shape Representation of 3D Point Clouds with an Attention-Based Sequence to Sequence Network
Liu Xinhai, Han Zhizhong, Liu Yu-Shen, Zwicker Matthias (2019), Point2Sequence: Learning the Shape Representation of 3D Point Clouds with an Attention-Based Sequence to Sequence Network, in Proceedings of the AAAI Conference on Artificial Intelligence, 33, 8778-8785.
View Inter-Prediction GAN: Unsupervised Representation Learning for 3D Shapes by Learning Global Shape Memories to Support Local View Predictions
Han Zhizhong, Shang Mingyang, Liu Yu-Shen, Zwicker Matthias (2019), View Inter-Prediction GAN: Unsupervised Representation Learning for 3D Shapes by Learning Global Shape Memories to Support Local View Predictions, in Proceedings of the AAAI Conference on Artificial Intelligence, 33, 8376-8384.
Y2Seq2Seq: Cross-Modal Representation Learning for 3D Shape and Text by Joint Reconstruction and Prediction of View and Word Sequences
Han Zhizhong, Shang Mingyang, Wang Xiyang, Liu Yu-Shen, Zwicker Matthias (2019), Y2Seq2Seq: Cross-Modal Representation Learning for 3D Shape and Text by Joint Reconstruction and Prediction of View and Word Sequences, in Proceedings of the AAAI Conference on Artificial Intelligence, 33, 126-133.
SeqViews2SeqLabels: Learning 3D Global Features via Aggregating Sequential Views by RNN With Attention
Han Zhizhong, Shang Mingyang, Liu Zhenbao, Vong Chi-Man, Liu Yu-Shen, Zwicker Matthias, Han Junwei, Chen C. L. Philip (2019), SeqViews2SeqLabels: Learning 3D Global Features via Aggregating Sequential Views by RNN With Attention, in IEEE Transactions on Image Processing, 28(2), 658-672.
Structure-Aware Data Consolidation
Wu Shihao, Bertholet Peter, Huang Hui, Cohen-Or Daniel, Gong Minglun, Zwicker Matthias (2018), Structure-Aware Data Consolidation, in IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(10), 2529-2537.
Temporally Consistent Motion Segmentation From RGB-D VideoTemporally Consistent RGB-D Motion Segmentation
Bertholet P., Ichim A.E., Zwicker M. (2018), Temporally Consistent Motion Segmentation From RGB-D VideoTemporally Consistent RGB-D Motion Segmentation, in Computer Graphics Forum, 37(6), 118-134.

Scientific events

Active participation

Title Type of contribution Title of article or contribution Date Place Persons involved
AAAI 2019 Poster View inter-prediction GAN: unsupervised representation learning for 3d shapes by learning global shape memories to support local view predictions 27.01.2019 Hawaii, United States of America Zwicker Matthias;
AAAI 2019 Poster Y2seq2seq: Cross-modal representation learning for 3d shape and text by joint reconstruction and prediction of view and word sequences 27.01.2019 Hawaii, United States of America Zwicker Matthias;
AAAI 2019 Poster Point2sequence: Learning the shape representation of 3d point clouds with an attention-based sequence to sequence network 27.01.2019 Hawaii, United States of America Zwicker Matthias;
ECCV 2018 Poster Specular-to-diffuse translation for multi-view reconstruction 08.09.2018 Munich, Germany Wu Shihao; Zwicker Matthias;


Use-inspired outputs

Abstract

3D content creation with today's computer graphics tools is highly laborious and requires expert knowledge and training. For example, generating computer graphics imagery (CGI) for movie production involves sophisticated and complex content creation pipelines. Highly skilled and specialized artists perform significant amounts of manual labor to model, texture, and animate the 3D objects required in these applications. This time consuming and costly process often limits the use of computer graphics techniques to large companies with sufficient funding sources. The proliferation of effective computer graphics applications, however, is severely limited by this modeling bottleneck. Usage of CGI is much less widespread in many other types of visual media with smaller production budgets, such as educational media, blogs, or journalism.Hence, the objective of the proposed project is to simplify the modeling process for computer graphics content. Ultimately, the outcomes of this project will make visual media production based on computer graphics available to non-specialists, fostering the development and proliferation of new types of visual media, and making visual storytelling using 3D computer graphics widely accessible. Our approach will leverage the concept of data-driven modeling. This means that users can browse, retrieve, edit, and recombine content stored in rich databases in intuitive ways, instead of modeling desired objects from scratch. In general, the challenges for data-driven modeling are data acquisition, the development of mathematical representations that expose the variability and the degrees of freedom inherent in the data, and the design of intuitive, interactive editing tools. In this project, we will focus on data-driven modeling of the geometry and appearance of 3D objects with functional parts. Specifically, we will pursue the following goals: First, we will develop methods to acquire real-world data for computer graphics modeling of dynamic, functional part-based 3D objects. While there is a large amount of research in the computer graphics literature on scanning static, monolithic 3D objects, the problem of acquiring models of real world objects including their parts and part relationships has gained much less attention. A key idea is to obtain this data by capturing users interacting with physical objects with RGB-D cameras. Our goal is to deploy such a system to a wide user base via the Internet to obtain a large amount of data. Next, we will develop mathematical models to represent the entirety of the acquired data and capture its variability and degrees of freedom. The representation will be suitable for modeling operations such as retrieval, editing, or combining of data. In a second part of this project we focus on acquiring and modeling the appearance of 3D models, based on the captured data. Our ultimate goal is to obtain a generative model that can be used to synthesize textures and appearance for 3D objects. Leveraging recent progress in deep learning, we propose to learn this model using generative adversarial networks (GANs). Finally, we will develop user interfaces and interactive modeling tools that leverage the databases and underlying data representations, which we will develop in this project, to provide effective content creation tools.Over the last two decades computer graphics technology has fundamentally changed visual media. Because of the difficulty in modeling suitable content, however, computer graphics has a long way to go to reach its full potential. In this project we will develop fundamental new tools for data-driven modeling in computer graphics, which will eventually help to unlock a wide range of new application scenarios from personal communication to blogs, short stories, daily news, educational media, and so on. Databases of computer graphics assets together with suitable mathematical representations and editing tools will become ubiquitous similar to image and video databases, but they will provide much richer tools for creative story telling. While databases of scanned and manually modeled 3D objects exist today, they represent 3D objects in a monolithic manner, without capturing parts or part relationships. This would be crucial to use them in many applications, however, where the objects should exhibit functionality and users should be able to manipulate them as in the real world. The outcomes of this project will address this issue, and provide tools for users to capture objects with these properties easily from real world examples.
-