Project

Back to overview

Multimodal machine learning for remote sensing information fusion

English title Multimodal machine learning for remote sensing information fusion
Applicant Tuia Devis
Number 150593
Funding scheme SNSF Professorships
Research institution
Institution of higher education Institution abroad - IACH
Main discipline Other disciplines of Environmental Sciences
Start/End 01.12.2014 - 30.11.2018
Approved amount 1'269'344.00
Show all

All Disciplines (2)

Discipline
Other disciplines of Environmental Sciences
Other disciplines of Engineering Sciences

Keywords (9)

Multisource; Multimodal; Very-high resolution; Data fusion; Machine learning; Uncertainty estimation; Hyperspectral imaging; Remote sensing; Crowdsourcing

Lay Summary (French)

Lead
La télédétection est une importante source d’information pour l’observation des processus à la surface de la terre, car elle permet de couvrir de grandes étendues avec une haute périodicité et de manière non intrusive.Récemment, ceci est devenu encore plus intéressant par la multitude de senseurs disponibles: des informations sur la couleur, les propriétés thermiques ou la position tridimensionnelle des objets peuvent être utilisées et constituent différentes vues sur le territoire. De plus, l’information sur le Web constitue une nouvelle source d’information à disposition pour améliorer les modèles obtenus par télédétection.Bien que prometteuses, ces sources de données (ou modalités) ne sont pas utilisables sans des routines de traitement qui puissent les combiner et les intégrer de manière efficace pour en extraire l’information recherchée par l’utilisateur.
Lay summary

Ce projet vise à développer ces routines, sous le nom de « télédétection multimodale ». Pour ce faire, nous travaillerons à l’interface entre la télédétection et l’apprentissage statistique: l’un apportera le savoir sectoriel et les contraintes physiques du domaine, alors que l’autre contribuera par les avancées méthodologiques en traitement de grandes masses de données.

Quatre domaines de recherche seront abordés: 1) fusion multiresolution de données optiques ; 2) intégration de données entre senseurs optiques et radar ; 3) intégration de sources discontinues et incertaines (issues du Web) et 4) community remote sensing, où le savoir des utilisateurs est intégré dans des modèles interactifs.

Les résultats du projet serviront à améliorer les applications en monitoring environnemental : utiliser plusieurs modalités va fournir des outputs plus réalistes. Ces résultats vont bénéficier les organisations qui font de l’intervention post-catastrophe (et qui doivent cartographier avec ce qui est disponible) et les fournisseurs de données qui préparent les missions futures. 

 

Direct link to Lay Summary Last update: 27.11.2014

Responsible applicant and co-applicants

Employees

Publications

Publication
Land cover mapping at very high resolution with rotation equivariant CNNs: Towards small yet accurate models
Marcos Diego, Marcos Diego, Volpi Michele, Kellenberger Benjamin, Tuia Devis, Tuia Devis (2018), Land cover mapping at very high resolution with rotation equivariant CNNs: Towards small yet accurate models, in ISPRS Journal of Photogrammetry and Remote Sensing, 145, 96-107.
Deep multi-task learning for a geographically-regularized semantic segmentation of aerial images
Volpi Michele, Tuia Devis (2018), Deep multi-task learning for a geographically-regularized semantic segmentation of aerial images, in ISPRS Journal of Photogrammetry and Remote Sensing, 144, 48-60.
Detecting mammals in UAV images: Best practices to address a substantially imbalanced dataset with deep learning
Kellenberger Benjamin, Marcos Diego, Tuia Devis (2018), Detecting mammals in UAV images: Best practices to address a substantially imbalanced dataset with deep learning, in Remote Sensing of Environment, 216, 139-153.
Decision Fusion with Multiple Spatial Supports by Conditional Random Fields
Tuia Devis, Volpi Michele, Volpi Michele, Moser Gabriele (2018), Decision Fusion with Multiple Spatial Supports by Conditional Random Fields, in IEEE Transactions on Geoscience and Remote Sensing, 56(6), 3277-3289.
Fine-grained landuse characterization using ground-based pictures: a deep learning solution based on globally available data
Srivastava Shivangi, Vargas Muñoz John E., Lobry Sylvain, Tuia Devis (2017), Fine-grained landuse characterization using ground-based pictures: a deep learning solution based on globally available data, in International Journal of Geographical Information Science.
Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources
Zhu Xiao Xiang, Tuia Devis, Mou Lichao, Xia Gui Song, Zhang Liangpei, Xu Feng, Fraundorfer Friedrich (2017), Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources, in IEEE Geoscience and Remote Sensing Magazine, 5(4), 8-36.
Detecting animals in African Savanna with UAVs and the crowds
Rey Nicolas, Volpi Michele, Joost Stéphane, Tuia Devis (2017), Detecting animals in African Savanna with UAVs and the crowds, in Remote Sensing of Environment, 200, 341-351.
Toward Seamless Multiview Scene Analysis from Satellite to Street Level
Lefevre Sebastien, Tuia Devis, Wegner Jan Dirk, Produit Timothee, Nassaar Ahmed Samy (2017), Toward Seamless Multiview Scene Analysis from Satellite to Street Level, in Proceedings of the IEEE, 105(10), 1884-1899.
Optimal Transport for Domain Adaptation
Courty Nicolas, Flamary Remi, Tuia Devis, Rakotomamonjy Alain (2017), Optimal Transport for Domain Adaptation, in IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(9), 1853-1865.
Dense semantic labeling of subdecimeter resolution images with convolutional neural networks
Volpi Michele, Tuia Devis (2017), Dense semantic labeling of subdecimeter resolution images with convolutional neural networks, in IEEE Transactions on Geoscience and Remote Sensing, 55(2), 881-893.
Processing of Extremely High Resolution LiDAR and RGB Data: Outcome of the 2015 IEEE GRSS Data Fusion Contest-Part B: 3-D Contest
Vo A. V., Truong-Hong L., Laefer D. F., Tiede D., Doleire-Oltmanns S., Baraldi A., Shimoni M., Moser G., Tuia D. (2016), Processing of Extremely High Resolution LiDAR and RGB Data: Outcome of the 2015 IEEE GRSS Data Fusion Contest-Part B: 3-D Contest, in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 9(12), 5560-5575.
Processing of Extremely High-Resolution LiDAR and RGB Data: Outcome of the 2015 IEEE GRSS Data Fusion Contest-Part A: 2-D Contest
Campos-Taberner Manuel, Romero-Soriano Adriana, Gatta Carlo, Camps-Valls Gustau, Lagrange Adrien, Le Saux Bertrand, Beaupere Anne, Boulch Alexandre, Chan-Hon-Tong Adrien, Herbin Stephane, Randrianarivo Hicham, Ferecatu Marin, Shimoni Michal, Moser Gabriele, Tuia Devis (2016), Processing of Extremely High-Resolution LiDAR and RGB Data: Outcome of the 2015 IEEE GRSS Data Fusion Contest-Part A: 2-D Contest, in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 9(12), 5547-5559.
Nonconvex Regularization in Remote Sensing
Tuia Devis, Flamary Rémi, Barlaud Michel (2016), Nonconvex Regularization in Remote Sensing, in IEEE Transactions on Geoscience and Remote Sensing, 54(11), 6470-6480.
Combining Human Computing and Machine Learning to Make Sense of Big (Aerial) Data for Disaster Response.
Ofli F., Meier P., Imran M., Castillo C., Tuia D., Rey N., Briant J., Millet P., Reinhard F., et al. (2016), Combining Human Computing and Machine Learning to Make Sense of Big (Aerial) Data for Disaster Response., in Big Data, 47.
Discriminative Multiple Kernel Learning for Hyperspectral Image Classification
Wang Qingwang, Gu Yanfeng, Tuia Devis (2016), Discriminative Multiple Kernel Learning for Hyperspectral Image Classification, in IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 54(7), 3912-3927.
Domain Adaptation for the Classification of Remote Sensing Data: An Overview of Recent Advances
Tuia D., Persello C., Bruzzone L. (2016), Domain Adaptation for the Classification of Remote Sensing Data: An Overview of Recent Advances, in IEEE Geoscience and Remote Sensing Magazine, 4(2), 41.
Foreword to the Special Issue on "GeoVision: Computer Vision for Geospatial Applications"
Tuia Devis, Wegner Jan Dirk, Mallet Clement, Yang Michael Ying (2016), Foreword to the Special Issue on "GeoVision: Computer Vision for Geospatial Applications", in IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, 9(7), 2840-2843.
Foreword to the Special Issue on Urban Remote Sensing
Tuia D., Gamba P., Juergens C., Maktav D. (2016), Foreword to the Special Issue on Urban Remote Sensing, in IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., 1763.
Kernel Manifold Alignment for Domain Adaptation.
Tuia D., Camps-Valls G. (2016), Kernel Manifold Alignment for Domain Adaptation., in PLoS One, e0148655.
Multi-temporal and multi-source remote sensing image classification by nonlinear relative normalization
Tuia D., Marcos D., Camps-Valls G. (2016), Multi-temporal and multi-source remote sensing image classification by nonlinear relative normalization, in Journal of the ISPRS, 1.
2015 IEEE GRSS data fusion contest: Extremely high resolution LiDAR and optical data [Technical committees]
Moser G., Tuia D., Shimoni M. (2015), 2015 IEEE GRSS data fusion contest: Extremely high resolution LiDAR and optical data [Technical committees], in IEEE Geoscience and Remote Sensing Magazine, 3(1), 40-41.
Cluster validity measure and merging system for hierarchical clustering considering outliers
De Morsier F., Tuia D., Borgeaud M., Gass V., Thiran J.-P. (2015), Cluster validity measure and merging system for hierarchical clustering considering outliers, in Pattern Recognition, 48(4), 1474-1485.
Foreword to the Special Issue on Hyperspectral Image and Signal Processing
Tuia D., Lopez S., Schaepman M., Chanussot J. (2015), Foreword to the Special Issue on Hyperspectral Image and Signal Processing, in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 8(6), 2337-2340.
Hydrometeor classification from polarimetric radar measurements: A clustering approach
Grazioli J., Tuia D., Berne A. (2015), Hydrometeor classification from polarimetric radar measurements: A clustering approach, in Atmospheric Measurement Techniques, 8(1), 149-170.
Processing of Multiresolution Thermal Hyperspectral and Digital Color Data: Outcome of the 2014 IEEE GRSS Data Fusion Contest
Liao W., Huang X., Van Coillie F., Gautama S., Pizurica A., Philips W., Liu H., Zhu T., Shimoni M., Moser G., Tuia D. (2015), Processing of Multiresolution Thermal Hyperspectral and Digital Color Data: Outcome of the 2014 IEEE GRSS Data Fusion Contest, in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 8(6), 2984-2996.
Semisupervised classification of remote sensing images with hierarchical spatial similarity
Huo L.-Z., Tang P., Zhang Z., Tuia D. (2015), Semisupervised classification of remote sensing images with hierarchical spatial similarity, in IEEE Geoscience and Remote Sensing Letters, 12(1), 150-154.
Semisupervised Transfer Component Analysis for Domain Adaptation in Remote Sensing Image Classification
Matasci G., Volpi M., Kanevski M., Bruzzone L., Tuia D. (2015), Semisupervised Transfer Component Analysis for Domain Adaptation in Remote Sensing Image Classification, in IEEE Transactions on Geoscience and Remote Sensing, 53(7), 3550-3564.
Multiclass feature learning for hyperspectral image classification: Sparse and hierarchical solutions
Tuia D., Flamary R., Courty N. (2014), Multiclass feature learning for hyperspectral image classification: Sparse and hierarchical solutions, in ISPRS Journal of Photogrammetry and Remote Sensing, 105, 272-285.
Understanding angular effects in VHR imagery and their significance for urban land-cover model portability: A study of two multi-angle in-track image sequences
Matasci G., Longbotham N., Pacifici F., Kanevski M., Tuia D. (2014), Understanding angular effects in VHR imagery and their significance for urban land-cover model portability: A study of two multi-angle in-track image sequences, in ISPRS Journal of Photogrammetry and Remote Sensing, 107, 99-111.
Optimal Transport for Domain Adaptation.
Courty Nicolas, Flamary Remi, Tuia Devis, Rakotomamonjy Alain, Optimal Transport for Domain Adaptation., in IEEE Transactions on Pattern Analysis and Machine Intelligence, pp.

Collaboration

Group / person Country
Types of collaboration
Computer Science dept., University of Toronto Canada (North America)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
- Exchange of personnel
Centre de Recherche en Environnement Terrestre, University of Lausanne Switzerland (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
Laboratory LaSIG, Lausanne EPFL Switzerland (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
University of Nice Sophia Antipolis France (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
- Exchange of personnel
Laboratory LTS5, Lausanne EPFL Switzerland (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
- Exchange of personnel
State of Neuchatel Switzerland (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Industry/business/other use-inspired collaboration
University of Rouen France (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
Earth Observation Center, German Space Agency (DLR) Germany (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
- Industry/business/other use-inspired collaboration
RSlab, University of Trento Italy (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
Laboratory TOPO, Lausanne EPFL Switzerland (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
- Research Infrastructure
Université de Bretagne du Sud France (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
- Exchange of personnel
Digital Globe Inc. United States of America (North America)
- Industry/business/other use-inspired collaboration
LARS laboratory, Purdue University United States of America (North America)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
- Exchange of personnel
IPL, Universitat de Valencia Spain (Europe)
- in-depth/constructive exchanges on approaches, methods or results
- Publication
- Exchange of personnel

Scientific events

Active participation

Title Type of contribution Title of article or contribution Date Place Persons involved
ACM SIGSPATIAL workshop ‘AI for Geographic Knowledge Discovery’ Talk given at a conference Multi-label building functions classifica- tion from ground pictures using convolutional neural networks 05.11.2018 Seattle, United States of America Srivastava Shivangi;
European Conference on Machine Learning (ECML), workshop Nectar Talk given at a conference Best practices to train deep models on imbalanced datasets - a case study on animal detection in aerial imagery 17.09.2018 Dublin, Ireland Marcos Gonzalez Diego;
European Conference on Computer Vision (ECCV) Poster Deep joint distribution optimal transport for unsupervised domain adaptation 17.09.2018 Munich, Germany Tuia Devis; Lobry Sylvain;
IEEE International Geoscience and Remote Sensing Symposium (IGARSS) Talk given at a conference A deep network approach to multitemporal cloud detection 23.07.2018 Valencia, Spain Tuia Devis;
IEEE International Geoscience and Remote Sensing Symposium (IGARSS) Talk given at a conference Detecting animals in repeated UAV image acquisitions by matching CNN activations with optimal transport 23.07.2018 Valencia, Spain Tuia Devis;
IEEE International Geoscience and Remote Sensing Symposium (IGARSS) Talk given at a conference Correcting misaligned rural building annotations in Open Street Map using convolutional neural networks evidence 23.07.2018 Valencia, Spain Tuia Devis;
IEEE International Geoscience and Remote Sensing Symposium (IGARSS) Talk given at a conference Improving maps from CNNs trained with sparse, scribbled ground truths using fully connected CRFs 23.07.2018 Valencia, Spain Tuia Devis;
Int. Conf. Mach. Learn (ICML), workshop FAIM Talk given at a conference Scale equivariance in CNNs with vector fields 16.07.2018 Stockholm, Sweden Marcos Gonzalez Diego;
Computer Vision and Pattern Recognition (CVPR) Talk given at a conference Learning deep structure active contours end-to-end 11.06.2018 Salt Lake City, United States of America Marcos Gonzalez Diego; Tuia Devis;
International Conference on Computer Vision (ICCV) Talk given at a conference RotEqNet: rotation equivariant vector field networks 23.10.2017 Venezia, Italy Tuia Devis; Volpi Michele; Marcos Gonzalez Diego;
IEEE Geoscience and Remote Sensing Symposium Talk given at a conference Joint height estimation and semantic labeling of monocular aerial images with CNNs 17.07.2017 Fort Worth, TX, United States of America Srivastava Shivangi;
EARSeL conference 2017 Talk given at a conference Is my method robust to acquisition conditions? An empirical manifold alignment perspective. 19.04.2017 Zurich, Switzerland Marcos Gonzalez Diego; Tuia Devis;
Joint Urban Remote Sensing Event Talk given at a conference Land use modeling in North Rhine-Westphalia with interaction and scaling laws 06.03.2017 Dubai, United Arab Emirates Tuia Devis; Volpi Michele;
International Conference on Pattern Recognition (ICPR) Poster Learning rotation invariant convolutional filters for texture classification 05.12.2016 Cancun, Mexico Tuia Devis; Marcos Gonzalez Diego;
IEEE International Geoscience and Remote Sensing Symposium, (IGARSS) Talk given at a conference Semantic labeling of aerial images by learning class- specific object proposals 25.07.2016 Beijing, China Volpi Michele;
IEEE Geoscience and Remote Sensing Symposium (IGARSS) Talk given at a conference Solving structured segmentation of aerial images as puzzles 25.07.2016 Beijing, China Tuia Devis; Volpi Michele;
IEEE International Geoscience and Remote Sensing Symposium (IGARSS) Talk given at a conference Getting pixels and regions to agree with conditional random fields 25.07.2016 Beijing, China Tuia Devis;
Computer Vision and Pattern Recognition (CVPR) Poster Geospatial correspondence for multimodal registration 13.06.2016 Las Vegas, United States of America Marcos Gonzalez Diego;
UYGU summer school Talk given at a conference Machine learning in remote sensing image classification: making the most out of your data 01.06.2016 Istanbul, Turkey Tuia Devis;
ISPRS Geo Spatial Week Talk given at a conference Aligning data representations of remote sensing multitemporal images 01.10.2015 Montpellier, France Tuia Devis;
IEEE Geoscience and Remote Sensing Symposium (IGARSS) Talk given at a conference Weakly supervised alignment of multisensor images 28.07.2015 Milan, Italy Marcos Gonzalez Diego;
IEEE Geoscience and Remote Sensing Symposium (IGARSS) Talk given at a conference To be or not to be convex? A study on regularization in hyperspectral image classification 23.07.2015 Milan, Italy Tuia Devis;
Multitemp 2015 Talk given at a conference Multitemporal classification without new labels: a solution with optimal transport 22.07.2015 Annecy, France Tuia Devis;
Joint Urban Remote Sensing Event Talk given at a conference Classification of urban structural types with multisource data and contextual priors 31.03.2015 Lausanne, Switzerland Tuia Devis;


Self-organised

Title Date Place
EarthVision 2017 21.07.2017 Honoloulou, United States of America
EarthVision 12.06.2015 Boston, United States of America
Joint Urban Remote Sensing Event 30.03.2015 Lausanne, Switzerland

Knowledge transfer events

Active participation

Title Type of contribution Date Place Persons involved
ESA Phi week Talk 19.11.2018 Rome, Italy Lobry Sylvain;
SGPT annual meeting Talk 15.10.2015 Zurich, Switzerland Tuia Devis;
Earth Observation Open Science 2.0 Poster 12.10.2015 Frascati, Italy Tuia Devis;


Communication with the public

Communication Title Media Place Year
Media relations: radio, television 10vor10 SRF 1 German-speaking Switzerland 2018
Media relations: radio, television la Matinale Radio Suisse Romande Western Switzerland 2018
Media relations: print media, online media The machine vision challenge to better analyze satellite images of Earth MIT technology review International 2018
Media relations: print media, online media Machine-Vision Drones Monitor Animals in the African Savanna MIT Technology Reviews International 2017
Print (books, brochures, leaflets) Column in the leaflet "Space Research in Switzerland 2014-2016" Western Switzerland German-speaking Switzerland Italian-speaking Switzerland 2016

Awards

Title Year
Teacher of the year, department of Geography UZH 2017
Best interactive presentation prize at IGARSS 2015 2016
Best poster award, ASIT-VD day (awarded to Msc student N. Rey) 2016
Best reviewer award IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 2016
Teacher of the year, department of Geography UZH 2016
UV Helava Prize for the best paper published in the Journal of the ISPRS 2012-2015 2016
Best reviewer award IEEE Geoscience and Remote Sensing Letters 2015

Associated projects

Number Title Start Funding scheme
136827 Understanding the underlying structure of remote sensing images: improving adaptation in classification models with artificial intelligence techniques 01.12.2011 Ambizione
144135 “KernelCD phase2”: Change Detection in Remote Sensing Images Using Kernel Based Machine Learning Algorithms 01.10.2012 Project funding (Div. I-III)

Abstract

The research of this project opens new avenues in remote sensing information fusion. Recently, remote sensing has seen important changes, as many new satellite and airborne sensors have been developed and access to optical information has become much easier. Users are now overwhelmed by the sheer amount of information, mainly but not exclusively optical, at their disposal. New strategies for combining and fusing these sources of information are therefore much needed. The research will develop the fundamental knowledge and the tools to integrate sources of information of either similar or widely different natures, in view of producing more comprehensive and highly-informative products for endusers working in domains such as post-catastrophe assessment, urban development or climate studies. It will consider different data modalities, such as images from different sensors, but also vector layers, in-situ measurements, zonal statistics or user knowledge. This highly-informative sources are nearly always discontinuous and uncertain, which has so far prevented their efficient use in remote sensing. I will propose solutions inspired by advances in machine learning and computer vision, yet specific to the problematics of remote sensing data, to deal with this unreliability and exploit the complementarities and specific strengths of the different sources into integrated fusing schemes.
-