Project

Back to overview

CrowdOS and CrowdLang: Engineering Large-Scale Human Computation Systems

Applicant Bernstein Abraham
Number 143411
Funding scheme Project funding (Div. I-III)
Research institution Institut für Informatik Universität Zürich
Institution of higher education University of Zurich - ZH
Main discipline Information Technology
Start/End 01.08.2013 - 30.11.2017
Approved amount 428'697.00
Show all

Keywords (8)

Social Computing; Human Computation; Crowdsourcing; CSCW (Computer Supported Cooperatie Work); Collective Intelligence; Human-Computer Interaction; Business Process Management; Group Decision Processes

Lay Summary (German)

Lead
Das Internet hat das Ausmass der Zusammenarbeit zwischen Menschen und Maschinen verändert. Heute können wir mit jedermann und jedem Computer irgendwo auf der Welt kooperieren. Unser Verständnis, wie solche weitverteilte und heterogene Systeme zu koordinieren sind, ist jedoch ungenügend.In diesem Projekt wollen wir untersuchen, wie Akteure in gemischten Mensch / Maschine-Systemen systematisch koordiniert werden können, um die heutigen Herausforderungen zu lösen.
Lay summary

Das Internet hat das Ausmass der Zusammenarbeit zwischen Menschen und Maschinen verändert. Als es das Internet noch nicht gab, konnte man nur mit Menschen kooperieren, welche in der Nähe waren. Durch das Internet haben sich die Kosten der Zusammenarbeit mit Mitarbeitern überall auf der Welt auf praktisch null reduziert.

Neue globale Systeme, vergleichbar mit einem globalen Gehirn, kombinieren die kommunikationstechnischen und rechnerischen Fähigkeiten von Computern mit der Kreativität und den kognitiven Fähigkeiten von Menschen, um routinemässig Probleme zu lösen und wichtige Aufgaben zu übernehmen. Auch wenn es bereits hunderte von überzeugenden Beispielen gibt (man denke nur an die Fülle von Informationen in der Wikipedia, die entdeckten Galaxien von Galaxyzoo, oder die Unmengen durch ReCaptcha erkannten Texte), ist unser Verständnis über die Programmierung solcher Systeme sehr beschränkt – insbesondere, da sich Menschen sehr unterschiedlich verhalten.

In diesem Projekt wollen wir untersuchen, wie systematisch Problemlösungen gestaltet werden können, welche entweder zu komplex oder zu teuer sind, um durch rein homogene Maschinen- oder Personengruppen bearbeitet zu werden. Anders ausgedrückt entwickeln wir neuartige Methoden und Ansätze, um das globale Gehirn zu programmieren.

Direct link to Lay Summary Last update: 02.07.2013

Lay Summary (English)

Lead
The Internet has changed the scale at which people and machines can collaborate. Now, we can collaborate with anybody anywhere on the world. Our understanding of how to program such widely heterogeneous systems is still poor because humans are different from computers. In this project we intend to investigate how actors in such mixed human/machine systems can be allocated (or motivated) and they can be systematically programed (or cultivated and coordinated).
Lay summary

The Internet has changed the scale at which people and machines can collaborate. Before the Internet most collaborators had to be close by to work together. Now, the cost of collaborating with anybody anywhere on the world has been reduced to almost zero. New, global systems – almost like a global brain – are now routinely able to solve problems, combining the communication and number-crunching capabilities of computer systems with the creativity and high-level cognitive capabilities of people.

Even though there are already literally hundreds of compelling examples of the global brain at work (just consider the wealth of information gathered by the Wikipedia, the galaxies discovered by Galaxy Zoo, or the reams of OCR tasks solved by ReCaptcha), our understanding of how to “program” the global brain is still poor because humans are different from traditional computers due to the huge motivational, error and cognitive diversity within and between humans. 

In this project we intend to investigate problem-solving processes that are either to difficult or to expensive to solve by either pure machine or pure human crowds. As such, we aim to discover how we can allocate (or motivate) resources and systematically program (or cultivate and coordinate) the global brain. 

Direct link to Lay Summary Last update: 02.07.2013

Responsible applicant and co-applicants

Employees

Publications

Publication
Task Routing and Assignment in Crowdsourcing based on Cognitive Abilities
Goncalves Jorge, Feldman Michael, Hu Subingqian, Kostakos Vassilis, Bernstein Abraham (2017), Task Routing and Assignment in Crowdsourcing based on Cognitive Abilities, in WWW 2017 Companion, Perth, AustraliaACM press, Perth, Australia.
Contemporary Issues of Open Data in Information Systems Research: Considerations and Recommendations
Lienk George, Lumbard Kevin, Conboy Kieran, Feller Joseph, George Jordana, Germonprez Matt, Goggins Sean, Jeske Debora, Kiely Gaye, Schuster Kirsten, Willis Matt (2017), Contemporary Issues of Open Data in Information Systems Research: Considerations and Recommendations, in Communications of the Association for Information Systems, 41(25), 556-577.
Efficiently identifying a well-performing crowd process for a given problem
De Boer Patrick, Bernstein Abraham (2017), Efficiently identifying a well-performing crowd process for a given problem, in 20th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2017), Portland, OR.
Expert estimates for feature relevance are imperfect
De Boer Patrick, Bühler Marcel, Bernstein Abraham (2017), Expert estimates for feature relevance are imperfect, in The 4th IEEE International Conference on Data Science and Advanced Analytics, Tokyo, Japan.
Towards Enabling Crowdsourced Collaborative Data Analysis
Feldmann Michaek, Anastasiu Cristian, Bernstein Abraham (2016), Towards Enabling Crowdsourced Collaborative Data Analysis, in Collective Intelligence (Abstracts), New York, USA.
PPLib: toward the automated generation of crowd computing programs using process recombination and auto-experimentation
De Boer Patrick, Bernstein Abraham (2016), PPLib: toward the automated generation of crowd computing programs using process recombination and auto-experimentation, in ACM Transactions on Intelligent Systems and Technology, 7(4), 49.
Massively Collaborative Complex Work — Exploring the Frontiers of Crowdsourcing
Feldman Michael (2015), Massively Collaborative Complex Work — Exploring the Frontiers of Crowdsourcing, in Doctoral Consortium of the 36th International Conference on Information Systems (ICIS), Fort Worth, USAICIS, Auckland, DE.
PPLib: towards systematic crowd process design using recombination and auto-experimentation
De Boer Patrick, Bernstein Abraham (2015), PPLib: towards systematic crowd process design using recombination and auto-experimentation, in Collective Intelligence 2015 (Abstract), Santa Clara, CAself-publsihed by conference, Santa Clara, CA.
Behavior-Based Quality Assurance in Crowdsourcing Markets Conference
Feldman Michael, Bernstein Abraham (2014), Behavior-Based Quality Assurance in Crowdsourcing Markets Conference, in Human Computation & Crowdsourcing (HCOMP'14) .
Cognition-based Task Routing:Towards Highly-Effective Task-Assignments in Crowdsourcing Settings
Feldman Michael, Bernstein Abraham (2014), Cognition-based Task Routing:Towards Highly-Effective Task-Assignments in Crowdsourcing Settings, in 35th International Conference on Information Systems (ICIS 2014), Auckland, New ZealandInternational Conference on Information Systems, ?.
SHAX: The Semantic Historical Archive eXplorer
Feldman Michael, Gao Shen, Novel Marc, Papaioannou Katerina, Bernstein Abraham (2014), SHAX: The Semantic Historical Archive eXplorer, in ISWC 2014 Posters & Demonstrations Track, Riva del Garda, Italy.

Collaboration

Group / person Country
Types of collaboration
Center for Collective Intelligence / Massachusetts Institute of Technology United States of America (North America)
- in-depth/constructive exchanges on approaches, methods or results
- Exchange of personnel

Scientific events

Active participation

Title Type of contribution Title of article or contribution Date Place Persons involved
The 4th IEEE International Conference on Data Science and Advanced Analytics (DSAA 2017) Individual talk Expert estimates for feature relevance are imperfect 19.10.2017 Tokyo, Japan Feldman Michael;
20th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2017 Individual talk Efficiently identifying a well-performing crowd process for a given problem 25.02.2017 Portland OR, United States of America de Boer Patrick Miguel;
Collective Intelligence 2016 Poster Efficient Exploration of the Crowd Process Design Space 01.06.2016 New York, NY, United States of America de Boer Patrick Miguel; Bernstein Abraham;
Collective Intelligence 2016 Poster Towards Enabling Crowdsourced Collaborative Data Analysis 01.06.2016 New York, NY, United States of America Feldman Michael;
Collective Intelligence 2015 Poster PPLib: towards systematic crowd process design using recombination and auto-experimentation 31.05.2015 Santa Clara, United States of America de Boer Patrick Miguel;
35th International Conference on Information Systems (ICIS 2014) Talk given at a conference Cognition-based Task Routing:Towards Highly-Effective Task-Assignments in Crowdsourcing Settings 14.12.2014 Auckland, NZ, Switzerland Feldman Michael;
Human Computation & Crowdsourcing (HCOMP'14) Poster Behavior-Based Quality Assurance in Crowdsourcing Markets Conference 02.11.2014 Pittsburgh, USA, United States of America Feldman Michael;


Self-organised

Title Date Place
Crowdsourcing and the Semantic Web 07.04.2014 Schloss Dagstuhl, Germany

Communication with the public

Communication Title Media Place Year
Talks/events/exhibitions Crowdsourcing (Talk at Informatikttage) German-speaking Switzerland 2017
Talks/events/exhibitions Wissen aus dem Datensalat Wenn die Crowd die Experten übertrifft (Talks at Scientifica) German-speaking Switzerland 2017
Talks/events/exhibitions Citizen Scientist
From Gofer to First Class Citizen German-speaking Switzerland 2015

Use-inspired outputs


Start-ups

Name Year
Kunendo AG 2014

Abstract

Much of the prosperity gained by the industrialization of the economy in the 18th century arose from the increased productivity by dividing work into smaller tasks performed by more specialized workers. Wikipedia, Google and other stunning success stories show that with the rapid growth of the World Wide Web, this concept of “Division of Labour” can also be applied on knowledge work [6, 40]. Consequently, systems interweaving both the number-crunching capabilities and scalability of computer systems with the creativity and high-level cognitive capabilities of people are now routinely able to solve problems that would have been unthinkably difficult only a few years ago. As the scale, scope and connectivity of these human-computer networks increase, we believe it will become increasingly useful to view these systems as constituting a kind of “global brain” [6].Even though there are already literally hundreds of compelling examples of the global brain at work, our understanding of how to “program” the global brain is still poor because human computers are different from traditional computers due to the huge motivational, error and cognitive diversity within and between humans [6, 45]. In this project we intend to investigate problem-solving processes that are either to difficult or to expensive to solve by either pure machine or pure human crowds. As such, we aim to answer the following research questions:• How can we systematically program, cultivate, and coordinate the global brain whilst automatically adapting to the cognitive variance of human computation resources?• How can we support the seamless reuse of successful interaction patterns resulting in a systematic exploration of the whole design space?• How can we efficiently recruit, incentivize, and allocate human computers in human computation systems taking the requestor’s budget, time, and quality constraints into consideration?To harness the full potential of the global brain, we need new powerful programming metaphors that support the design and implementation of human computation systems, as well as general-purpose infrastructure to execute them. Specifically, to move from a culture of “wizard of oz”-techniques, in which applications are the result of extensive trial-and-error refinements, we propose to build the programming language and framework CrowdLang which will incorporate abstractions such as group decision processes, the CrowdRecombinator, a novel tool to support the engineering process of new human computation systems, and the social operating system CrowdOS which will manage the allocation of human resources to tasks as well as provide robust infrastructure for contracts and payments. We believe using these three components, human computation systems will become truly transformative in a variety of domains.The impact of the project is twofold. On the practical side these tools will help engineers and managers to adopt human computation systems in practice and, thus, will foster the transformation of the Swiss economy in the “age of hyperspecialization” [40]. On the scientific side, our explorations are likely to advance the field by providing new insights about the interplay between human and machine computation, the longterm properties of those systems, and will foster more engineering-oriented approaches in the development process.
-