An dieser Stelle können alle Veröffentlichungen, die im Rahmen des Projekts ARBay entstanden sind, aufgerufen werden.


Automated Retrieval of Graphical User Interface Prototypes from Natural Language Requirements

Kristian Kolthoff, Christian Bartelt, Simone Paolo Ponzetto

High-fidelity Graphical User Interface (GUI) prototyping represents a suitable approach for allowing to clarify and refine requirements elicitated from customers. In particular, GUI prototypes can facilitate to mitigate and reduce misunderstandings between customers and developers, which may occur due to the ambiguity and vagueness of informal Natural Language (NL). However, employing high-fidelity GUI prototypes is more time-consuming and expensive compared to other simpler GUI prototyping methods. In this work, we propose a system that automatically processes Natural Language Requirements (NLR) and retrieves fitting GUI prototypes from a semi-automatically created large-scale GUI repository for mobile applications. We extract several text segments from the GUI hierarchy data to obtain textual representations for the GUIs. To achieve ad-hoc GUI retrieval from NLR, we adopt multiple Information Retrieval (IR) approaches and Automatic Query Expansion (AQE) techniques. We provide an extensive and systematic evaluation of the applied IR and AQE approaches for their effectiveness in terms of GUI retrieval relevance on a manually annotated dataset of NLR in the form of search queries and User Stories (US). We found that our GUI retrieval performs well in the conducted experiments and discuss the results.

DOI: 10.1007/978-3-030-80599-9_33
Link: Automated Retrieval of Graphical User Interface Prototypes from Natural Language Requirements | SpringerLink


GUI2WiRe: Rapid wireframing with a mined and large-scale GUI repository using natural language requirements

Kristian Kolthoff, Christian Bartelt, Simone Paolo Ponzetto

High-fidelity Graphical User Interface (GUI) prototyping is a well-established and suitable method for enabling fruitful discussions, clarification and refinement of requirements formulated by customers. GUI prototypes can help to reduce misunderstandings between customers and developers, which may occur due to the ambiguity comprised in informal Natural Language (NL). However, a disadvantage of employing high-fidelity GUI prototypes is their time-consuming and expensive development. Common GUI prototyping tools are based on combining individual GUI components or manually crafted templates. In this work, we present GUI2WiRe, a tool that enables users to retrieve GUI prototypes from a semi-automatically created large-scale GUI repository for mobile applications matching user requirements specified in Natural Language (NLR). We extract multiple text segments from the GUI hierarchy data and employ various Information Retrieval (IR) models and Automatic Query Expansion (AQE) techniques to achieve ad-hoc GUI retrieval from NLR. Retrieved GUI prototypes mined from applications can be inserted in the graphical editor of GUI2WiRe to rapidly create wireframes. GUI components are extracted automatically from the GUI screenshots and basic editing functionality is provided to the user. Finally, a preview of the application is created from the wireframe to allow interactive exploration of the current design. We evaluated the applied IR and AQE approaches for their effectiveness in terms of GUI retrieval relevance on a manually annotated collection of NLR and discuss our planned user studies.



Show me your Living Room: Investigating the Role of Representing User Environments in AR Remote Consultations

Nicolas Kahrl, Michael Prilla, Oliver Blunk

The study reported here investigates AR based support for remote consultations, in which an on-site user is supported by a remote helper. In such situations, it is important for the remote helper (or, in our case, the consultant) to see the environment of the person asking for support in order to relate to it. Based on literature, we created and tested different mechanisms using a 2D video stream with a captured 2D/3D texturized virtual model of the room. In addition, we compared the often-used way of fixing the remote helper’s view to the view of the on-site user with the possibility to move around freely in the 2D/3D model. The aim of the study was evaluating how to support an on-site user wearing an AR HMD. The study tested four conditions composed from these differences and with nine real furniture consultants. In the study, we compared four mechanisms in which the consultants were able to place furniture in the living room of a customer and advise the customer on their purchase. We found that there were hardly any differences in task load, social presence or perceived support between the four different conditions. However, participants had clear preferences for certain conditions and aspects of them. From our analysis, we provide recommendations for the design of mixed reality support for remote consultations.



The Effects of Consultant Avatar Size and Dynamics on Customer Trust in Online Consultations

Gordon Brown, Michael Prilla

This study investigates the impact of avatars on interactions between customers and consultants in remote, online consultations supported by Augmented Reality (AR). Based on past research, we were interested whether the appearance of an avatar and its dynamics affect important factors for online consultations such as social presence, trust in the consultant and perceived customer satisfaction. In particular, we chose avatar size and dynamics (movement/gaze) to compare different avatars in a 2×2 experiment, in which customers wear AR head mounted devices to consult a remotely located consultant in a mock furniture consultation session. Our results show no significant differences in trust and satisfaction, but significantly different levels of perceived social presence for life-sized, dynamic avatars as well as significantly higher co-presence for all life-sized avatars. Additional data from interviews with the participants revealed a clear preference for dynamic avatars over static ones. Based on an analysis of these findings, we make design recommendations and suggest directions for future research.



Automatic generation of graphical user interface prototypes from unrestricted natural language requirements

Kristian Kolthoff

High-fidelity GUI prototyping provides a meaningful manner for illustrating the developers’ understanding of the requirements formulated by the customer and can be used for productive discussions and clarification of requirements and expectations. However, high-fidelity prototypes are time-consuming and expensive to develop. Furthermore, the interpretation of requirements expressed in informal natural language is often error-prone due to ambiguities and misunderstandings. In this dissertation project, we will develop a methodology based on Natural Language Processing (NLP) for supporting GUI prototyping by automatically translating Natural Language Requirements (NLR) into a formal Domain-Specific Language (DSL) describing the GUI and its navigational schema. The generated DSL can be further translated into corresponding target platform prototypes and directly provided to the user for inspection. Most related systems stop after generating artifacts, however, we introduce an intelligent and automatic interaction mechanism that allows users to provide natural language feedback on generated prototypes in an iterative fashion, which accordingly will be translated into respective prototype changes.



Möglichkeiten der Digitalisierung zur Beratung von hochkonfigurierbaren und hochindividualisierbaren Gütern

Oliver Blunk, Gordon Brown, Michael Prilla

Der Verkauf von hochkonfigurierbaren bzw. hoch-individualisierbaren Gütern ist online aktuell nicht verfügbar, da für den Kunden eine intensive Beratung notwendig ist, um passende Produkte bzw. Konfigurationen zu finden. Zusätzlich stehen Online-Beratungen vor der Herausforderung, dass Kunden unterschiedliche Konfigurationen nicht betrachten oder anfassen können, und der Berater ebenfalls nicht visualisiert wird. Weiterhin verfügen Geschäfte häufig nicht über Ausstellungsstücke, die alle unterschiedlichen Konfigurationsmöglichkeiten abdecken. Basierend auf Literatur und einer eigens in einem Möbelhaus durchgeführten ethnografischen Studie werden Anforderungen an ein System zur Online-Beratung im Kontext der Möbelbranche abgeleitet. Beispielsweise wird untersucht, inwieweit Augmented Reality als Ansatz dabei helfen kann, dass Kunden mit virtuell dargestellten Produkten interagieren können und so unterschiedliche Konfigurationen vergleichen können. Weiterhin werden unterschiedliche Möglichkeiten diskutiert, um den Kunden in diesem Umfeld zu beraten. Basierend auf den Anforderungen und unterschiedlichen Technologien wird ein Szenario erstellt, welches beschreibt, wie eine Online-Beratung durchgeführt werden könnte. Es wird der Einkaufsprozess eines fiktiven Kunden betrachtet. Zusätzlich werden auch Einschränkungen einer online-basierten Beratung diskutiert.

DOI: 10.1365/s40702-020-00630-x


Potentials of AR technology for the digitalization of consultancy intensive sales processes on the example of furniture sales

Oliver Blunk, Gordon Brown, Niklas Osmers, Michael Prilla

Selling highly configurable or complex products in online markets has been difficult as they often require intensive consultation time which currently takes place in a physical store. The consultation process has various facets that could possibly benefit from digital solutions. As a first step in our research process, we conducted an ethnographic observation in a furniture store to analyze current sales and consultation processes. Based on our findings, we derive several requirements which can guide the design of IT solutions to digitalize those processes. Additionally, we propose an augmented reality system offering various benefits to both consultants in stores as well as customers at home.



Evaluating Pointing Modes and Frames of Reference for Remotely Supporting an Augmented Reality User in a Collaborative (Virtual) Environment: Evaluation within the Scope of a Remote Consultation Session

Gordon Brown, Michael Prilla

With the availability of powerful and affordable Augmented Reality (AR) devices, scenarios have become popular in which people wearing AR devices are supported by remote experts. These experts often use 2D peripherals to access the video feed of the 3D head mounted device (HMD) and to augment it with verbal or digital information. This raises the question whether tools that work for these scenarios also work for remote consultations. We conducted a study to (re-)evaluate these tools in a furniture sales consultation context. We focused on the consultant side of these settings and explored how the use of different pointing methods and perspectives affect different situations during a consultation. For this, we developed and evaluated a prototype with ten furniture store workers. Initial results show that while most usability and task load scores were even, the participants reported clear favorites for certain settings. We use these results to derive design recommendations for similar future projects.