Back to overview

Learning Representations of Abstraction in Text

Applicant Henderson James
Number 178862
Funding scheme Project funding (Div. I-III)
Research institution IDIAP Institut de Recherche
Institution of higher education Idiap Research Institute - IDIAP
Main discipline Information Technology
Start/End 01.10.2018 - 30.09.2022
Approved amount 555'884.00
Show all

Keywords (5)

textual entailment; representation learning; deep learning; opinion summmarization; natural language understanding

Lay Summary (French)

Le résumé de l'opinion pour un grand groupe de personnes nécessite d'abstraire les opinions de consensus et les points fréquents de désaccord des divers détails d'opinions individuelles. Dans ce projet, des modèles de résumé d'opinion seront développés, en réalisant des progrès fondamentaux dans les modèles d'apprentissage profonds de l'abstraction sémantique dans le texte.
Lay summary
Les méthodes d'apprentissage profonds (deep learning) ont permis de grands progrès dans la modélisation de la sémantique du langage naturel, mais les représentations apprises sont conçues pour modéliser la similarité, pas l'abstraction.  Ce projet visera à étendre les travaux du chercheur principal sur la modélisation de l'abstraction afin de développer des modèles de deep-learning qui apprennent les représentations de l'abstraction dans le texte.  Ces modèles apprendront à prédire si un texte est une abstraction d'un autre texte (textual entailment), comment grouper des textes en fonction de leur abstractions communes (opinion clustering), et quels textes expriment l'abstraction partagée dans un ensemble de textes (cluster labelling). Ces technologies de base formeront les blocs de construction pour un système de résumé d'opinion à grande échelle.
Ce projet aura comme résultat des contributions fondamentales dans les domaines de l'apprentissage automatique (machine learning) et de la compréhension du langage naturel (NLU), ainsi que des technologies pour le résumé d'opinion à large échelle. Ces technologies permettront des travaux ultérieurs afin de déployer des outils pour la communication en grands groupes de personnes, créant ainsi une forme de média social où l'opinion de chacun compte.
Direct link to Lay Summary Last update: 24.09.2018

Lay Summary (English)

Opinion summarisation for a large group requires abstracting out the consensus opinions and common points of disagreement from the diverse details of individual opinions. This project will develop models of large-scale opinion summarisation through fundamental advances in deep learning models of semantic abstraction in text.
Lay summary
Modelling abstraction in text, also known as textual entailment, is a fundamental problem in natural language semantics.  Deep learning methods have made great progress in modelling natural language semantics, but the representations learned are designed for modelling similarity, not abstraction.  In contrast, the PI's recent work proposes vector-space representations designed for modelling abstraction.  This project will extend this work to develop deep learning models which learn representations of abstraction in text.  These models will learn to predict when one text is an abstraction of another (textual entailment), how to group texts by their shared abstractions (opinion clustering), and what text expresses the shared abstraction in a set of texts (cluster labelling).  These basic technologies will form the building blocks for a system for large-scale opinion summarisation, showing the sizes and labels of clusters of opinions, thus reflecting how many people share common opinions.

This project will result in both fundamental contributions to machine learning and natural language understanding, and technology for large scale opinion summarisation.  These technologies will enable future work developing deployed tools for large group communication, creating a form of social media where everyone's opinion counts.
Direct link to Lay Summary Last update: 24.09.2018

Responsible applicant and co-applicants


Associated projects

Number Title Start Funding scheme
125137 Sampling and Regularization for Latent Structure Models with Feature Induction 01.09.2009 Project funding (Div. I-III)
180320 Automated interpretation of political and economic policy documents: Machine learning using semantic and syntactic information 01.01.2019 Sinergia


Social media has great democratising potential, but the vast diversity of opinions makes it difficult to understand what everyone really thinks. If we could automatically extract the consensus opinions and major current issues from everyone's opinions, and broadcast that summary to everyone, it would be a powerful new channel of social communication. The overall objective of this project is to solve the fundamental technical challenges required for this large-scale opinion summarisation, including how to model semantic abstraction in text.The impressive recent advances in natural language understanding brought by deep learning and related latent vector models put us in a position to solve these challenges. In particular, work by the PI on modelling abstraction in a vector space has already improved models of abstraction between words. The specific aims of this project are to extend this research to models of semantic abstraction between texts and models of summarising large sets of opinions.Detecting when one text is an abstraction of another (also known as textual entailment) is a fundamental problem in natural language semantics. Task~1 of this project will develop textual entailment, including associated machine learning methods for modelling abstraction and unification. Extending the PI's model of abstraction between words (hyponymy, or lexical entailment) will require modelling both semantic composition (unification) and complex non-compositional reasoning. This work will leverage a long-standing theme of the PI's research, non-parametric vector space representations (where the number of vectors grows with the complexity of the semantics). Using a bag of entailment vectors to represent the semantics of a text will allow attention-based deep learning architectures to model both the semantic structure of language and the complex reasoning needed for the general case of textual entailment. Evaluation will be on benchmark textual entailment datasets, such as the Stanford Natural Language Inference corpus, using semi-supervised learning.Such models of abstraction in text will enable advances in summarising sets of opinions. Task~2 will develop models of opinion summarisation, including associated machine learning methods for modelling abstraction and intersection. Abstraction is crucial for controlling the complexity of summaries, given that everyone's opinion is different. The summary should include the consensus opinions and major dimensions of disagreement, which are abstract statements entailed by large proportions of the opinions (intersection). We will develop entailment-based methods for clustering, and for generating statements from bag-of-entailment-vector representations. As well as exploiting the relevant available corpora, evaluation will involve an initial stage of data collection, annotation and analysis, and establishing new benchmark measures and results for opinion summarisation, using unsupervised and semi-supervised learning.A third aim is to develop the above models even for languages where we don't have the data. Task~3 will develop multi-lingual and cross-lingual models of textual entailment and opinion summarisation, through multi-task learning with machine translation. This work will extend attention-based neural machine translation models to induce shared representations between multi-lingual neural machine translation and the above models of textual entailment and opinion summarisation. Evaluation will be done on the same data as above where some statements have been translated.This project will result in both fundamental contributions to machine learning and natural language understanding, and enabling technology for large scale opinion summarisation. We anticipate related projects for leveraging these advances into deployed tools for large group communication, thereby advancing the tradition of direct democracy of which Switzerland is a world leader.