Thanks to ACM OpenTOC Service, an Open Access linking service which allows free full-text downloads from the ACM Digital Library.
For much of its history, AI was a scientific discipline defined more by its portrayal in science fiction than by its actual technical achievements. Real AI systems are now catching up to their fictional brethren, and are as likely to be seen in news headlines as on the big screen. Yet as AI systems match or even outperform people on tasks that were once considered yardsticks of human intelligence, one area of human experience still remains largely unchallenged by technology: our sense of humour. This is not for want of trying, as I will show. The true nature of humour has intrigued scholars for millennia, but AI researchers can go one step further than philosophers, linguists and psychologists once could: by building computer systems with a sense of humour, capable of appreciating the jokes of their human users or even of generating jokes of their own, we can turn academic theories into practical realities that amuse, explain, provoke and delight. This talk sets out to challenge the archetype of the rigidly humourless machine in popular culture, and to make a case for the necessity of a truly computational understanding of humour. By giving our machines the ability to understand and generate humour – including sarcasm and irony – we can better understand ourselves too, as we construct machines that are more flexible, more understanding, and more willing to laugh at their own limitations.
In 1968, the Polish born curator Jascia Reichardt opened a landmark exhibition at the ICA in London entitled Cybernetic Serendipty in which all manner of sensor-augmented devices, objects and sculptures stood ready to usher art into a new technological age. Remarkably, while ever more complex sensors, algorithms and devices have steadily increased in the 54 years since Reichardt’s show, essentially the same goal has remained: using artificial sensing as an integral part of an artwork in order for the work to “make sense” about its “world” and respond to it. This phenomenon, what artist and theorist Simon Penny calls the “aesthetics of behavior,” perfectly aligns with the long sought- after imaginaries of artists, designers and technologists to create seamless computational links between our bodies and the larger environment and thus, reorganize the human senses in order for them to act as input for such works. But if the history and practices of “immersion” in the arts has long focused on the senses being transformed through melding them with technologies embedded into the actual physical world, the next wave of immersion seeks the opposite: to capture the senses in order to render a synthetic world that is “realer” than the physical one. In the words of computer graphics pioneer Ivan Sutherland from 1965, the new “ultimate display” (a harbinger of later VR/AR head mounted devices), would need to “serve as many senses as possible.”1 Thus, contrary to the idea that the senses are simply to be replaced by the prosthetics of artificial sensors, a different story seems to be emerging. Our senses are needed to drive and feed ever-new immersive experiences by being increasingly “coupled” or linked to the simulated. This talk will careen through TeamLab’s immersive environments installed in the landfill islands of Tokyo, through the visions of artists in the 1960s to create new kinds of “reactive environments” and our now just emerging “metaverse” age of Extended Reality in order to give a critical historical and socio-technical picture of our present and future visions of art in the age of immersion.
Persistently, data has been part of our lives through the ages. Our use of data is nearly always private, often small, but has continually formed a part of how we live our everyday lives. Very often it is stored, saved, and preserved as part of our arts and crafts through the ages. It is important that we do not let the current ‘big data’ revolution with all its emphasis on bigness and power, alienate us from our deep connection with data. It is important that we share the potentially empowering aspects of the use of small, situated, and embedded data. I will discuss this in relation to my continued research towards promoting data comprehension by creating appropriate interactive technologies that can help people negotiate the everyday transformation of data into understanding. Specifically, I will talk about my research into extending the available visual representations, using interaction to expand the potential of existing visualizations, and into broadening the potential of information visualization by investigating engagement with new audiences. Just as data has in the past, it still has the potential to enrich our daily lives. Let’s not get dis-inherited!
SESSION: Paper and Pictorial Session: AI and Creativity
This paper presents a study of a group of university students using generative machine learning to translate from natural language to computer code. The study explores how the use of the AI tool can be understood in terms of co-creation, focusing on the one hand on how the tool may serve as a resource for understanding and learning, and on the other hand how the tool affects the creative processes. Findings show how the participants search for a ’correct’ syntax in their instructions to the machine learning tool, and how the inconsistent and erroneous behavior can work as a way to generate clues and inspiration for generating creative expressions. The notion of friction is used to describe how systems like this can serve to both lower thresholds for programming, and also interfere with the creative processes, encouraging reflection and exploration of alternative solutions.
Initial Images: Using Image Prompts to Improve Subject Representation in Multimodal AI Generated Art
Advances in text-to-image generative models have made it easier for people to create art by just prompting models with text. However, creating through text leaves users with limited control over the final composition or the way the subject is represented. A potential solution is to use image prompts alongside text prompts to condition the model. To better understand how and when image prompts can improve subject representation in generations, we conduct an annotation experiment to quantify their effect on generations of abstract, concrete plural, and concrete singular subjects. We find that initial images improved subject representation across all subject types, with the most noticeable improvement in concrete singular subjects. In an analysis of different types of initial images, we find that icons and photos produced high quality generations of different aesthetics. We conclude with design guidelines for how initial images can improve subject representation in AI art.
Creative problem solving and innovation powered by Artificial Intelligence (AI) requires detection of user needs that can be reframed into data science problems. We propose a framework of 10 creativity triggers for creative human centered AI opportunity detection, based on research and categorization of information retrieval tasks and cognitive task analysis. The method aims to facilitate a dialog between data scientists and underrepresented groups such as non-technical domain experts.
Impact on problem discovery and idea generation was evaluated in co-creation workshops. Results show that the method significantly increases ideas’ scores on the appropriateness to a specific problem and their AI relevancy. Participants experienced the prompts as a helpful mental framework about AI methods and felt encouraged to decompose user stories into more detailed cognitive tasks that help data scientists relate ideas to high level data science methods.
Understanding User Perceptions, Collaborative Experience and User Engagement in Different Human-AI Interaction Designs for Co-Creative Systems
Human-AI co-creativity involves humans and AI collaborating on a shared creative product as partners. In a creative collaboration, communication is an essential component among collaborators. In many existing co-creative systems, users can communicate with the AI, usually using buttons or sliders. Typically, the AI in co-creative systems cannot communicate back to humans, limiting their potential to be perceived as partners rather than just a tool. This paper presents a study with 38 participants to explore the impact of two interaction designs, with and without AI-to-human communication, on user engagement, collaborative experience and user perception of a co-creative AI. The study involves user interaction with two prototypes of a co-creative system that contributes sketches as design inspirations during a design task. The results show improved collaborative experience and user engagement with the system incorporating AI-to-human communication. Users perceive co-creative AI as more reliable, personal, and intelligent when the AI communicates to users. The findings can be used to design effective co-creative systems, and the insights can be transferred to other fields involving human-AI interaction and collaboration.
In this paper, we investigated the potential of using a tangible interaction interface for creative activities as a preliminary step to achieve human-AI co-creation. We conducted a preliminary study to explore sand as a medium for co-creation and identify potential opportunities of AI mediating the process. We created Sand Playground, a co-creation interface that uses sand as an interaction medium. We describe details for building the interface. We selected two opportunities from the study and developed them as two modes, Artistic Mimicry and Zen Garden. We also conducted a workshop study using Sand Playground to evaluate collaboration and the characteristics of the tangible interface. We identified findings responding to drawing strategies, multisensorial experience, and performative and ephemeral quality of creation. We discuss next steps of achieving our ultimate goal: an AI-mediated interface that collaborates with users physically in real-time on sand.
In this pictorial paper, we present a series of drawing conversations held between two humans, mediated by computational GAN models. We consider how this creative collaboration is affected by the hybrid inclusion of more-than-human participants in the form of watercolour and artificial intelligence. Our drawing experiments were an extension of our search for new ways of seeing and telling, which includes a reflection of the extent to which more-than-human elements took part in our creative process. We discuss our tendencies to form strange interpretations and assign meaning to the unpredictable and ambiguous spaces we created with them. We further speculate on the characteristic material agencies they revealed in our interactions with them. Finally, we contend how such collaborations are already and always embedded and embodied in our ways of seeing and knowing in design and creativity research.
SESSION: Paper and Pictorial Session: Creative Design & Research
I was not born a designer – sometime this identity shift must have happened. I was unaware of it, and if asked, I would still not know how to define a “designer”. Drawing and sketching are activities intrinsic to the design discipline, and are widely understood as tools for communication, documentation, or artefact-driven reasoning. But are they also essential to the understanding of design knowledge? Or a symptom of a designer’s identity rather than a tool for “designerly ways of knowing”? During a week-long design workshop I dealt with difficulties making sense of a panoply of embodied design methods in the absence of a sketchbook. In this pictorial I describe my self-diagnosis as a sketch-bound designer, unable to digest abstract knowledge without holding a pen. I advocate for sketching as focusing, and a primary activity in design epistemology that needs no other than a first-person reason to be performed.
Research ideas are pivotal in research practice. While research domains, topics, and methods are often outlined by specific research fields, the process of capturing and developing research ideas is less categorical. Conceiving and developing research ideas requires continuous creative thinking, usually supported by various different tools, each more or less carefully selected by a researcher to fulfill a specific purpose. In this paper, we investigate the creative work practices of academic researchers, with a focus on the workflows and tools they employ to manage ideas. Through a qualitative survey (n=51) and in-depth interviews (n=18) with researchers from a wide range of fields, we identify and describe typical processes of managing research ideas, different types of research ideas (a research question or problem, a method, a hypothesis or antithesis, and a theory), properties of good research ideas, as well as potentials for tools and technology to support idea management for researchers.
Identifying Cognitive and Creative Support Needs for Remote Scientific Collaboration using VR: Practices, Affordances, and Design Implications
Remote scientific collaborations have been pivotal in generating scientific discoveries and breakthroughs that accelerate research in many fields. Emerging VR applications for remote work, which utilize commercially available head-mounted displays (HMDs), offer the promise to enhance collaboration, through spatial and embodied experiences. However, there is little evidence on how professionals in general, and scientists in particular, could use existing commercial VR applications to support their cognitive and creative collaborative processes while exploring real-world data as part of day-to-day collaborative work. In this paper, we present findings from an empirical study with 14 coral reef scientists, examining how they chose to utilize available resources in existing virtual environments for their ongoing data-driven collaborative research. We shed light on scientists’ data organization practices, identify affordances unique to VR for supporting cognition in a collaborative setting, and highlight design requirements for supporting cognitive and creative collaboration processes in future tools.
Explorations of Interactive Research Artifacts in Use: Applying Research through Design to Understand Ways Scholars Leverage Interactivity in their Research Practices
New technologies and digitization have the potential to vastly alter our knowledge infrastructures. Specifically, this work focuses on the effects of interactive technologies on research practices, referred to as “interactive research artifacts.” Current research investigates the communicative affordances of such technologies, but minimal work critically examines the creative ways scholars are engaging with these artifacts. Through in-depth interviews with 14 scholars, and Design Studies literature, this work arrives at an understanding of interactive research artifacts as knowledge creation tools, rather than simply communicative tools. As such, to design for a future where interactive research artifacts become widely used scholarly tools, a comprehensive understanding of the ways in which these artifacts are used as means of knowledge creation is critical.
Collaborative creativity is an essential part of modern teamwork and is often supported by formal techniques, such as design thinking. Current support tools are often limited in scope as understanding the time-varying nature and structure of team communication is insufficient. We investigate how collaborative creative activities in new product development teams can be digitally supported while maintaining face-to-face communication. This work analyzes to what extent paralinguistic and proxemic features of team interaction relate to performance in new product development teams and if and how this relationship differs for different stages in the design process. This is investigated by applying multilevel modeling on data collected during a four-week new product development cycle. The cycle was completed by four teams, during which data were collected automatically using sociometric badges that capture social signals of team interactions. In addition, the data are combined with survey-based measurements on the team’s daily design process and periodic performance evaluations. The current paper provides evidence that social signals are related to team performance and that this relationship varies across the stages in the product design process. Certain social signals contribute positively in one stage but less in other stages, showing the importance of using multimodal signals when modeling high-level collaborative patterns. This research contributes to the literature by providing a better understanding of relevant factors when designing supporting tools or methods for collaborative creative problem solving.
Exploring affordances through design-after-design: the re-purposing of an exhibition artefact by museum visitors
Museums have increasingly focused on digital technologies and play as means to provide personalized, engaging experiences for their audience. Balancing educational and playful values is often conflicting. To address that conflict, museums often employ participatory design strategies. However, those strategies usually end after the deployment of those experiences, thus they do not accommodate for what occurs during actual use. In this article, we follow Light House, a research-through-design experiment of an installation that was developed using an iterative design approach which expands on actual use by deploying undetermined artefacts to support the discovery of novel interactions by visitors. Through our findings, we explore the “failure” of Light House to support the discovery of such interactions in relation to its educational character, but rather it inspired people to incorporate it in the activities supported by the surrounding space. Finally, we discuss the implications those discovered interactions in terms of potential re-design directions.
SESSION: Paper and Pictorial Session: Narratives and Sensemaking
The body plays an important role in building and maintaining emotional relationships. When a loved one dies, it is inevitable that their body will decease, meaning that physical closeness can no longer be experienced. In this paper, we engage with material ways to extend experiences of physical and emotional nearness beyond death. We explore the design space of ‘bodily heirlooms’: crafted, precious, interactive objects designed to pass on bodily impressions beyond lifespan. We review cultural practices that intend to bequeath the body, and related work that addresses death, bereavement and digital heirlooms. After that, we describe the design and making process of three bodily heirlooms, reflections in and on action, and reactions of visitors who inspected the heirlooms during an exhibition. We conclude by discussing notions of heirlooms’ preciousness, the emotional bond between heirlooms and their maker and how the process can be understood as a feminist practice.
The written word is an asynchronous form of communication through which static texts are exchanged. However, the act of writing is a dynamic cognitive process in which evolving ideas are organized into a coherent narrative. This process is not shared with readers because they only see the final product. With the emerging culture of live streaming and sharing creative practice with online audiences, writing replays can be a new expressive medium and content for writers and another way to engage with reading for the audience. In this paper, we explore the benefits of watching a real-time writing replay. We recorded the writing processes of professional writers (n = 13) and interviewed them about their and others’ replays. These writers found replays engaging. Replays facilitated self-reflection and helped writers empathize with each other. In addition, we conducted an online survey (n = 78) to compare readers’ comprehension and perception of writing replays with those of traditional text. We found that writing replays enhanced user engagement and the perceived quality of the text for some writing styles. Participants from both studies considered using writing replays regularly.
Educators in professional higher education experience difficulties addressing students’ self-awareness in their courses. The topic is often dismissed by students as ‘vague’ or ‘irrelevant’. This is detrimental for learning since knowing who you are and who you want to be is crucial: it guides behaviour and helps to feel balanced and in control. Support is needed to trigger students’ self-awareness and to make this process less demanding. In this paper we present guidelines for interactive triggers supporting students in developing their self-awareness. We asked students to discuss self-made photos in small groups and offered them three paper prototypes of triggers to work with. Questionnaire results and analysis of students’ discussions resulted in insights on how these triggers provoke distinct interaction and support self-awareness. Insights in advantages and disadvantages of the triggers can be used to design and implement innovative interactive systems that engage students in the process of developing self-awareness.
Do humans dream of electronic and digital devices? Or are our dreams free of technology, its influences and disruptions. When technology does appear, what part does it play in our subconscious narratives? Dreams are a reflection and distillation of our daily cognitive processes and waking experiences, and ubiquitous items such as mobile phones have become part of our everyday existence. This pictorial explores subjective dream imagery and narrative concerning the appearance and user experience of technology within our subconscious.
The history of western philosophy and art reveals a complex relation drawn between sight and knowledge, light and truth, the human gaze and subjectivity. In the 21st century this relationship takes a new turn, with the introduction of new technological means, especially in computational neuroscience and machine vision, that aim at recording, quantifying, analyzing, reproducing and automating the gaze. This pictorial essay is a multilayered investigation applying new modes of technological objectification to creative representations of eery, uncanny images of blindness and displaced eyeballs, in the history of art and cinema. It highlights the problematic nature of this form of objectification, and calls into question the means, assumptions and conclusions that are brought about in the process of objectifying the gaze, by pointing out the culturally saturated substrate already embedded in the act of viewing.
SESSION: Paper and Pictorial Session: Motion and Making
We present a new abstract representation of choreographic motion that conveys the movement quality of fluidity that is central to the style of modern dance pioneer Isadora Duncan. We designed our model through a collaboration with an expert Duncanian dancer, using five flexible ribbons joining at the solar plexus and animated it from motion capture data using a tailored optimization-based algorithm. We display our model in a Hololens headset and provide features that allow to visualize and manipulate it in order to understand and learn Duncan’s choreographic style. Through a series of workshops, we explored our system with professional dancers and were able to observe how it provides them with an immersive experience of a novel visualization of Duncan movement qualities in a way that was not possible with traditional human-like or skeleton-based representations.
Contrary to stereotypes, the capacity for creativity and innovation does not diminish post-retirement. The domain of older adult-led maker-based communities offers HCI researchers the opportunity to explore not only how older adults innovatively and collaboratively create in later life, but also how we might design with these communities rather than for them. We report findings from a design workshop study spanning over two weeks involving 13 members of an urban Australian men’s shed. Our findings show that making in the men’s shed was influenced by (1) socio-materiality, (2) interpersonal dynamics and (3) shared expertise. We provide HCI researchers with an understanding of how older adults worked collaboratively within their own creative community as found through the deployment of a novel card-based probe, a workshop and followup investigations. We further unpack recommendations for organising design workshops with older adult maker communities that work towards meaningful community-oriented participation and greater agency in co-design studies.
As digital fabrication machines have become more accessible and widely available, practitioners in maker communities have become increasingly responsive to the opportunities to achieve bespoke modifications, known colloquially as ‘modding’. Drawing on interviews with five experienced makers who engage in modding a laser cutter, along with ethnographic observations of maker-machine interactions, we analyse makers’ experiences and ‘war stories’ to frame modding as a prevalent but less explored maker activity. We highlight how makers care for machines, how they cope with risks when engaging in modding, and how mods are essentially creative projects. Based on our findings, we present the conceptualisation of the ‘pliable machine’ – a socio-technical system constituted by, (1) an accessible machine that can be altered, (2) maker skills that go beyond intended use, and (3) a surrounding ‘maker culture’ of caring, sharing and experimentation. Treating the machine as a material offers an alternative perspective on our interactions with technology; we show how the laser cutter becomes pliable in the hands of those who mod.
Craft and motion are strongly connected, and reflection about this connection enhances our understanding of digital creative experiences. We summarize seven key steps for reflection in craft and create machine-assisted motion-centric tools to support reflection for each key step. These tools reveal different properties of the recorded movement sequence and the relationship between movement and created models. We tested the tools with a sample of novice users who were asked to create a wire-based jewelry model with a set of creation intentions. The resultant qualitative data and visual analysis charts show that introducing the motion information helps the user to meaningfully reflect on their creative experiences and ultimately enhance their decision-making in terms of selecting the models and creating subsequent digital craft-making movements. We therefore hope that our study will stimulate future discussion to bring motion-centric reflection to digital craft processes.
Creating sound by manipulating the location of sound-producing objects using gestures is an interesting interaction paradigm. To understand it better, we analyzed videos of users interacting with ‘Random Access Lattice’. In this sound installation, users move a loudspeaker to explore sound laid out in space using a time to space mapping. We performed an inductive analysis of user movement in relation to the visitor intention and the sonic outcome. We identified several body, hand, and grip gestures which were performed with different movement qualities to manipulate the loudspeaker at variable speeds, orientations, and body areas. These were used to search and trace the sonic material in a goal-oriented fashion but also to interact creatively with sound by looping and modulating sounds. We provide a visual index of our findings which can be used when designing gesture interactions with sound.
Tapis Magique: Machine-knitted Electronic Textile Carpet for Interactive Choreomusical Performance and Immersive Environments
Tapis Magique is a pressure-sensitive, knitted large-scale electronic textile carpet that generates three-dimensional sensor data based on body location, postures, and gestures and drives an immersive sonic environment in real-time. Inspired by traditional textile arts and cultures, we applied an artistic approach into technological textile design and merged new materials, sensing technologies, and digital fabrication with contemporary dance and music into one united and harmonious piece of object and performance. In this pictorial, we present and discuss our textile design rationale, fabrication strategies, hardware systems, musical mappings, as well as our collaborative effort with a professional dancer to demonstrate the interactive electronic textile carpet. Our work challenges the conventional relationship between choreography and music and unveils dancer and/or sound artist’s creative possibilities of agency, intimacy, and improvisation over the choreomusical performance through a novel textile interface.
SESSION: Paper and Pictorial Session: Robots and Games
In this work, we explore the effect of a social robot’s embodiment and creativity scaffolding on children’s creative problem solving skills in the context of a digital creative problem-solving game called Escape!Bot. Children aged 5-11 years played the video game, which involved assembling contraptions to escape a digital world, and the robot Jibo acted as a collaborative peer that offered questions, reflective prompts, challenges, and ideas. In order to evaluate the role of the robot’s co-presence and creativity scaffolding, we ran a 2×2 experiment to determine the factorial efficacy of the robot’s embodiment and creativity scaffolding behaviors. We observed mixed results, with the robot’s creativity scaffolding having a positive influence on the time taken to complete the game, but not on the overall use of novel objects or reuse of objects. We present the system design, user study and findings from Escape!Bot to investigate the feasibility of designing social robots to support creative problem solving.
This study introduces Co-gnito, a participatory physicalization game that supports collaborative urban mental mapping through storytelling. Through Co-gnito we investigate gaming as a means to elicit subjective spatial experiences and to steer the synchronous construction of a physicalization that aligns and represents them. Co-gnito was evaluated during seven deployments by analyzing how 28 players mapped their spatial experiences of two university campuses. Our results indicate that storytelling as a gaming mechanic, guided and motivated the gradual addition of personal contributions towards a collective outcome, but its reward system did not nudge the mapping direction as expected. We also demonstrate how the shared construction process of a physicalization is influenced by how the data encoding scheme was negotiated, by the token physical affordances and by the game mechanics. We therefore believe that our core contributions, comprising of: 1) a working research prototype; 2) an augmentation of the physicalization pipeline towards collaborative settings; and 3) a set of reflective considerations, provide actionable knowledge on how to design participatory physicalizations in the future.
Game masters (GMs) are creative practitioners who plan and orchestrate tabletop role-playing games. Through an interview study, we investigate how eight expert game masters adapt everyday technologies and materials as creativity support tools (CSTs) for improvisational and collaborative play. We integrate theories of improvisational and distributed creativity with the human-artifact model, which provides an activity-theoretical vocabulary for analyzing the mediating relationships between specialist practitioners and their tools. We show how GMs prepare and deploy readymade artifacts: analog and digital CSTs that flexibly mediate recurring creative tasks in their practice, such as improvising narrative elements, facilitating smooth play, and creating aesthetic effects. We find that GMs demonstrate designerly thinking as they create, share, and refine repertoires of readymade artifacts. We argue that our theoretical approach can inform future studies of IT-mediated creativity, and that readymade artifacts can be an analytical and generative concept for the design of novel creativity support tools.
Tinycade is a platform designed to help game designers build their own mini arcade games by hand. With this platform, one can craft functioning game controllers out of everyday materials such as cardboard and toothpicks. By utilizing computer vision markers, we can create a variety of inputs without a single wire. In this pictorial, we discuss the functionality of Tinycade and showcase three games that demonstrate the variety of controls possible with this platform.
SESSION: Paper and Pictorial Session: Arts and Crafts
Creative practice often requires persevering through moments of ambiguity, where the outcome of a process is unclear. Creative practitioners intentionally manage this process, for example by developing strategies to break out of creative ruts, or stay motivated through uncertainty. Understanding the way experts engage with and manage these creativity-relevant processes represents a rich source of foundational knowledge for designers of Creativity Support Tools. These strategies represent an opportunity for CST research: to create CSTs that embody emotional and process-focused strategies and techniques. Through interviews with expert practitioners in diverse domains including performance, craft, engineering, and design, we identify four strategies for managing process: Strategic Forgetting, Mode Switching, Embodying Process, and Aestheticizing. Understanding tool- and domain-agnostic creative strategies used by experts to manage their own creative process can inform the design of future CSTs that amplify the benefits of successful strategies and scaffold new techniques.
In this pictorial, we present Kolam, a visual artform originating in Tamilnadu, South India, as an ecofeminist computational art practice. We provide a visual documentation of Kolam’s Intangible Cultural Heritage (ICH) through eight characteristics based on existing research and authors’ personal experiences as Kolam practitioners. We begin by framing Kolam as an ecofeminist practice, highlighting cultural and ecological characteristics of Kolam as a Tamil tradition. We then illustrate evolving hybrid multimedia and contemporary technological practices that characterize Kolam as computational art. Our aim is to present a cohesive and compelling visual narrative using the artwork of authors and four contemporary Kolam practitioners to inspire creativity and highlight challenges for relational knowledge production in design and Human-Computer Interaction (HCI) research..
Portraying What is in Front of You: Virtual Tours and Online Whiteboards to Facilitate Art Practice during the COVID-19 Pandemic
The arts and education fields recognize the importance of museums and art galleries as not only buildings that house hundreds, often thousands, of specialized objects, artworks, research, and conservation but rather as institutions sharing the history of people and their environment, thus they play a substantive role in modern society. For art students, practitioners, and hobbyists, these institutions are often visited to provide inspiration and practice, although the COVID-19 pandemic has had a profound impact. National lockdowns and restrictions, and social distancing measures meant museums and art galleries underwent mandated temporary closures, and socially distanced specified visitor routes often meant onsite art practice was no longer permitted. This exploratory pictorial takes a first-person research method to present a collection of reflections, experiences, and example artworks by facilitators of an art practice group that moved from on-site to online practice during the pandemic.
Goldwork Embroidery: Interviews with Practitioners on Working with Metal Threads and Opportunities for E-textile Hybrid Crafts
Within the emerging field of e-textiles, goldwork embroidery (also known as metalwork) which uses metal threads and materials is an underexplored area, despite being a centuries-old practice in traditional crafts of different cultures. In this paper, we explore the material culture of textile goldwork to better understand how e-textile researchers can leverage their material properties, palette, and practices. First, we provide a historical background of English goldwork to give HCI researchers context on this craft field including technological and cultural influences. Then, we interview 13 contemporary goldwork practitioners on their creative practice to better understand the tools, techniques, and skills they employ. Our study findings show how goldwork practitioners deal with the unique constraints of metal threads and materials, and how these materials need to be handled differently than regular non-metal threads and fibers. This paper contributes an analysis of goldwork practices for HCI audiences with suggestions on how we can leverage these practices for the future of e-textile hybrid crafts.
Curating Interactive Art for Online Conferences: Artist, Curator and Technologist Experiences in Gather.Town
We present the results of a reflective, practice-based study with creative practitioners who contributed to the Art Track at Creativity & Cognition 2021. We investigate curating an interactive online gallery in the context of the COVID-19 pandemic, focusing on the opportunities afforded and design constraints imposed by the Gather.Town platform and the experiences of the participating practitioners. We present auto-ethnographic reflections from Author 1, who created the online gallery space. We draw on the experience of Author 2 in curatorial research to analyze the experience of emerging practice through interviews with participating artists and curators. Results show that many of the artists were positively surprised by the platform engagement opportunities and conference attendees’ engagement with the artworks at the gallery opening, and appreciated the equitable, global reach. Further analysis reveals a desire for future iterations, and an exploration of the platform in a hybrid context, alongside an in-person exhibition.
Seagrass meadows are twice as effective as forests at capturing and storing carbon, but human activities have caused them to gradually disappear over the last few decades. We take a nature-centered design approach on contextual inquiry and collaborative designs methods to consolidate knowledge from marine and material sciences to industrial design. This pictorial documents a dialogue between designers and scientists to co-create an ecological intervention using digital fabrication to manufacture morphing ceramics for seagrass meadow restoration.
Crafts are human-material interaction. Documentation of handicrafts is challenging and needs creativity because individual craftwork heavily depends on the understanding of the materials at hand and the hand movement to apply the creator’s unique kinesthetic knowledge. In this study, we investigate the layers of information for crocheting intelligence visualization. To frame the structure of the learning process, we conduct introspective ethnographic research and visual analysis. From the insights from the crocheter’s sketches, we develop idea sketches and prototypes of overlay designs for instructional videos and augmented reality. Discussion follows on design implications for visual learning of kinesthetic cognition over augmented reality and visual medium.
SESSION: Paper and Pictorial Session: Sound and Music
While recent work has shown that it is possible to find disentangled directions in the latent space of image generative networks, finding directions in the latent space of sequential models for music generation remains a largely unexplored topic. In this work, we propose a method for discovering linear directions in the latent space of a musicgenerating Variational Auto-Encoder (VAE). We use PCA, a statistical method, to transform the input data such that the variation along the new axes is maximized. We apply PCA to the latent space activations of our model and find largely disentangled directions that change the style and characteristics of the input music. Our experiments show that the found directions are often monotonic, global and encode fundamental musical characteristics such as colorfulness, speed, and repetitiveness. Moreover, we propose a set of quantitative metrics to describe different musical styles and characteristics to evaluate our results. We show that the found directions decouple content and can be utilized for style transfer and conditional music generation tasks. Our project page can be found at http://catlab-team.github.io/midispace.
Flow with the Beat! Human-Centered Design of Virtual Environments for Musical Creativity Support in VR
As previous studies have shown, the environment of creative people can have a significant impact on their creative process and thus on their creations. However, with the advent of digital tools such as virtual instruments and digital audio workstations, more and more creative work is digital and decoupled from the creator’s environment. Virtual Reality technologies open up new possibilities here, as creative tools can seamlessly merge with any virtual environment the user finds himself in. This paper reports on the human-centered design process of a VR application that aims at supporting the user’s individual needs to support their creativity while composing percussive beats in virtual environments. For this purpose, we derived factors that influence creativity from literature and conducted focus group interviews in order to learn how virtual environments and 3DUI can be designed for creativity support. In a subsequent laboratory study, we let users interact with a virtual step sequencer UI in virtual environments that were either customizable or fixed/unchangeable. By analyzing post-test ratings from music experts, self-report questionnaires, and user behavior data, we examined the effects of such customizable virtual environments on user creativity, user experience, flow, and subjective creativity support scales. While we did not observe a significant impact of this independent variable on user creativity, user experience or flow, we found that users had specific individual needs regarding their virtual surroundings and strongly preferred customizable virtual environments, even though the fixed virtual environment was designed to be creatively stimulating. We also observed consistently high flow and user experience ratings, which promote human-centered design of VR-based creativity support tools in a musical context.
Identifying points of engagement from a person’s interaction with computers could be used to assess their experience and to adapt user interfaces in real-time. However, it is difficult to identify points of engagement unobtrusively; HCI studies typically use retrospective protocols or rely on cumbersome sensors for real-time analysis. We present a case study on how children compose digital music at home in which we remotely identify points of engagement from patterns of interaction with a musical interface. A mixed-methods approach is contributed in which video recordings of children’s interactions whilst composing are labelled for engagement and linked to i) interaction logs from the interface to identify indicators of engagement in interaction, and ii) interview data gathered using a remote video-cued recall technique to understand the experiential qualities of engaging interactions directly from users. We conclude by speculating on how the suggested indicators of engagement inform the design of adaptive music systems.
This one-day workshop invites designers, researchers and practitioners whose work might involve design, to collectively speculate about designed artefacts and technologies through the creation of drawing conversations: visual dialogues resulting from the merging of drawings created by different people. The workshop aims to use drawing as an activity for collaborative engagement with ambiguity, interpretation and mutual learning. Through drawing activities, we aim to join in Venice’s rich creative traditions, and develop speculative visualisations in order to find common grounds between the diverse research interests of our organisers and participants.
Augmenting Personal Creativity with Artificial Intelligence: Workshop proposal for Creativity and Cognition 2022
This workshop focuses on emerging approaches for using Artificial Intelligence (AI) systems to support and augment personal creativity. Recent developments in generative Machine Learning demonstrate the ability of AI systems to perform tasks which are often associated with creativity – generating imagery, composing music, writing prose, etc. This workshop will examine opportunities for incorporating this kind of functionality into the creative practice of designers, artists and craftspeople, in practical and experimental ways. It will focus on how AI might enhance, rather than supplant, individual human creativity, through collaboration, serendipity, and creative reflection. We seek to engage a broad range of creative practitioners and researchers, bringing together those already using AI in their practice with those who are new to the technology, to understand emerging approaches and define future opportunities.
Co-design methods and toolkits are commonly used to involve people from diverse backgrounds and disciplines in design processes, promoting collaboration, design thinking, shared decision making, and creativity. These methods and toolkits are generally tailored to in-person workshops supported by different physical artifacts (e.g. card-sets) in a shared physical location. Physical co-location and artifacts allow participants to interact in seamless ways, relying on everyday modalities of interaction. The CoViD-19 pandemic has forced many of such workshops online. This required transforming location, methods, toolkits and to rethink interaction among participants. With this workshop we aim to look back at these experiences of transformation and to reflect on the affordances of the physical and the virtual in co-design workshops. What are the challenges of transforming location, methods, and toolkits that are designed for in-person workshops into the digital? In which ways can in-person and virtual workshops co-exist and complement each other? We invite participants to share their experiences and reflect on how to bring together virtual and in-person co-design workshops.
LEADING TEAMS TO VICTORY: Where Creative Theory Meets Practice: Where Creative Theory Meets Practice
This research-based, interactive workshop invites attendees to explore the sociocultural theory of creativity and to engage in interactive activities that spark team creativity and innovation. We will introduce the research of the VICTORY model (Vision, Ideation, Combination, Team, Openness, Risk-taking, and Yes-I-can Mindset) as a method to promote team creativity and innovation. Workshop participants – researchers, educators and creative practitioners – will have the opportunity to experience putting theory to practice. We will then bridge research with creativity training and introduce techniques to build powerful and effective multicultural, interdisciplinary teams.
Hybrid Environments – spaces where physical and digital elements are blended in-situ – have been growing and expanding over the past few decades. As these technologies become woven into our everyday lives, new challenges and opportunities emerge as they transform our experiences in our homes, workplaces, museums, and cities. Designing Hybrid Environments requires training in both the design of physical environments and interactive technologies. Cross-trained students in higher education need to be able to creatively respond and adapt to these emerging landscapes. Students must bridge fundamental knowledge in design, architecture, art, interactivity, and computation. This is a complex pedagogical practice. How are educators meeting this emergent need? The workshop’s primary goals are to (i) form a collective understanding of the current technological and pedagogical challenges to teaching the design and understanding of Hybrid Environments; (ii) bring together educators and practitioners interested in the future of Hybrid Environments, (iii) discuss creativity support tools as well as frameworks, methods, and approaches for teaching emerging technologies, (iv) form a multidisciplinary community to shape the future of pedagogical inquiry and implementation of Hybrid Environments.
Making AI: Advancing creative approaches to the design of AI systems through the craft of making them
The objective of this workshop is to rethink ‘making’ AI through a focus on the physical materials involved in designing, producing, and running artificially intelligent systems. The recent manufacturing chip shortage illuminates digital technologies’ physicality and the fragility of the commodity and production networks that underpin the AI systems our cities, governments, and workplaces have come to rely on. Those that are conventionally considered to ‘make’ AI through the design of AI systems are largely divorced from AI’s materiality and the craft of making AI. Corresponding research on AI and creativity focuses primarily on the digital artefacts, potentials and imaginaries AI creates, and less so on the social and material artefacts embedded in its ability to create. We hope to push participants beyond the theoretical knowing of AI materiality to tactile knowing through a practice-based approach to ‘making’ AI. Reorienting the focus of AI to materials and the supply chain as sites of creative intervention could leverage the potential of sensory, tactile experiences to spur reimaginations of AI technologies and infrastructures. Ultimately, the aim is to advance creative approaches to the design of AI systems through the craft of making them.
A Workshop on Brainstorming Workshops Successful brainstorming with diverse creators in a hybrid work environment
Creating new products, services and research topics requires harnessing the best ideas from a broad set of cross-functional stakeholders who are creatively working together. Brainstorming workshops are powerful tools for developing design solutions or interventions, creating empathy, driving alignment, decision making and prioritization, and also for creating a deep understanding across a diverse group of creators. However, holding useful and energizing brainstorming workshops can be challenging given the variety of stakeholders and their ways of problem solving – made even more so with remote or hybrid work teams. This “Workshop on Brainstorming Workshops” is designed to highlight and model effective tools and best practices for practitioners to drive their own useful design thinking based brainstorming workshops in a fully remote or hybrid environment. We expect to share real world brainstorming techniques, trainings and templates from embedded researchers at Meta, sharing our insights and best practices, as well as facilitating a space for sharing of all participants’ insights around these challenges. Additionally, we expect to create a space to learn together and improve practitioners’ techniques.
Non-profit approaches in the sharing economy are proliferating in everyday life (e.g., tool libraries, food sharing collectives). They focus on promoting social, economic, and ecological sustainability within local communities. However, prior research uncovered emergent challenges of these communities that prevent their endurance and growth. Despite growing academic interest in studying these communities, designers found it difficult to effectively adopt newly generated knowledge in practice. Addressing the research-practice gap, we developed a translational resource for design practitioners to conceptualize new value-added features for supporting technologies in the sharing economy context. Specifically, we synthesized emergent challenges in non-profit sharing economy communities and key social and environmental sustainability efforts in design research into a familiar format for designers – the card-based toolkit to support generative design activities. Beyond introducing the toolkit, we offer a “how-to” guide for its adoption in collaborative settings.
We present a real-time multi-user interactive system in which children can use their drawings to create and perform cartoon animation spontaneously and collaboratively. With our system, children can control the movements and actions of the characters and tell stories like puppeteers. The interaction is simple and intuitive for children to learn and use. The preliminary results of this field study suggest that the system can effectively stimulate children’s creative desire and imagination, encourage personal expression and exploration, and provide a way to communicate and cooperate with peers. Being low-cost, flexible, and robust, the system can be easily and quickly deployed in schools and other places while maintaining strong practicability and affordability that even extend into remote rural areas.
Many researchers focus on the relationship between culture and creativity, often analyzing the influence of culture on design creativity. However, most studies only draw conclusions about the positive or negative effects of culture on creativity, rather than providing solutions to mitigate the adverse impact or magnify the positive impact. Therefore, in the current study, we first developed a dedicated framework to explain the influence of culture on design creativity based on the 4Ps model of creativity (person, process, product, and press). Then, we analyzed the influence of culture on design creativity based on Hofstede’s dimensions of culture. Following this, we identified the methods and solutions that can be adopted to mitigate negative cultural influences and prime positive cultural influences on design creativity. Finally, we proposed possible solutions to mitigate the negative cultural influences and prime the positive cultural influences by applying innovative technology to design creativity. Our study will inspire researchers to mitigate the negative influences of culture, magnify the positive effects, and apply new technology to enhance creativity.
In this paper, we argue for the potential and relevance of modelling novel AI co-creative systems after key aspects which characterises the specific kind of design processes unfolding in hackathons. There has recently been an increased interest into the potentials and challenges of creative collaborations between humans and AI, however, there have not been many practical implementations of co-creative systems. We argue that by designing co-creative systems to support the specific kind of creative design work in hackathons, valuable contributions can be made in the context of an overlooked aspect in interaction design; how to support fast design thinking. The specificity of hackathon participation involves high demands for fast idea generation, decision-making, and prototyping, which ideally ends in a functioning and novel prototype. Specifically, we identify three key characteristics of hackathon participation, and for each characteristic we discuss how they may model Human-AI interaction in the context of co-creative systems for fast design thinking. The contribution of this paper is to provide future research on co-creative systems with inspiration grounded in the parallels between aspects of AI and key characteristics of hackathon participation.
The Non-myth of the Noble Red: Exploring Brechtian Principles of Storytelling and Performance in the Authoring of a Tangible Narrative
World-building is among the most relevant and challenging aspects of authoring an interactive digital narrative. These challenges can even be more significant when designing tangible interactive narratives. The author must define how and when to express the material aspects of the narrative in the real world and how artefacts can support this storytelling process. This poster introduces the creative process of world-building a tangible interactive narrative: The Non-myth of the Noble Red. In this tangible narrative, world-building is based on a Brechtian approach to explore ways in which closing the gap between performance and space can involve the user into a more active role while becoming conscious of the social implications presented in the narrative. Reaching out to principles and concepts brought from theatre and other forms of performative arts places the performative component and discourse of the narrative in the foreground of the experience. This approach can be used to identify considerations that are usually ignored in mainstream interactive experiences by prioritizing the narrative’s critical discourse and exploring more artistic and experimental approaches to storytelling.
Creating Stories of Learning, for Learning: Exploring the Potential of Self-Narrative in Education with ‘Our Journey’
Participatory research led us to identify that students lacked means to represent their individual journeys through study and that their diverse goals, challenges and personal contexts were not easily understood. Prompted by this, we created ‘Our Journey’ to support them and harness the value of these narratives as a means for reflection and communication. This paper explores self-narrative processes and how they can be beneficial, describes the design of Our Journey, and introduces examples of collaboratively-developed pilots, where different activities using the platform are being devised and trialled. We reflect here on how new opportunities for self-narrative creativity can be supported in simple and engaging ways, and how flexibility in the design means the same underlying structure can be used online and in a physical form, for one-off retrospective narratives and ongoing journal activities, and with prompts for individual and shared reflection as appropriate.
Virtual communication has turned into a necessity in current culture. Although major technological breakthroughs were made in the past decade, virtual communication platforms do not prioritize the emotional connection between people. Shmoodle is an online platform in which two participants communicate orally while doodling with each other virtually on a shared canvas. Designed to favor emotional exchange over efficiency and clarity, Shmoodle integrates the users’ facial expressions into the conversations via control over the brush color and responds to synchronous facial expressions via the canvas background color. Initial testing with three pairs of users revealed that drawing and exploring the effects of the platform together gave them a heightened sense of connection and a playful disposition.
Improve User Experience by Adjusting Draft Design through Retouching System for Paper-cutting Production
This study proposed a system for quantifying artistic production difficulty and adjusting difficulty via retouching, with an aim to improve production. A paper-cutting artwork was created controlling a knife to cut the designed paper. Paper-cutting draft designs have two parts: a white and black areas for cutting-out and painting, respectively. The difficulty level of the picture was measured based on the thickness of the cutting borderlines. Thus, in this study, a retouching system that reduces the difficulty of draft design by adjusting the line thickness was developed. Further, the effects of this system on a novice production was experimentally investigated. Consequently, the system was confirmed to improve accuracy and psychological state (flow state) via retouching the draft design.
Morphing Matter for Girls: Designing interdisciplinary learning experiences to broaden teenage girls’ participation in STEM
Morphing Matter (MM) is an emerging multidisciplinary field that combines material science and digital fabrication. It empowers people to engineer materials that respond to a specific energy or stimuli and transform its shape or other characteristics such as stiffness, opacity, and phase. This poster introduces creative pathways with MM to engage girls in science, technology, engineering, and mathematics (STEM) learning, sparking interest in the field. We share our experience designing a set of learning tools that introduce novices to MM by fabricating morphing artifacts triggered by water. Further, we present a pilot MM workshop highlighting our reflections on designing learning experiences to foster high-school-aged girls’ curiosity and interest in STEM.
Cognitive and affective processes are interconnected and should be considered in educational contexts in order to support learners’ awareness and regulation of their cognitive-affective states. In Science, Technology, Engineering, the Arts and Mathematics (STEAM) educational activities engage the learners in creative problem solving (CPS). STEAM CPS activities are challenging and could raise maths and science anxiety among the participants of these activities. In this study, we engage three teams in a STEAM activity with IoT STM32 educational boards in order to develop a prototype addressing a real-world problem. In this context, we support the emergence of the awareness of the participants’ cognitive-affective states through a CPS board aiming to report the cognitive-affective states and their subsequent regulation through CPS regulation strategies to permit improving the climate within the team but also the advancement of the STEAM CPS.
Exploring the creative process in a brainstorming session to develop a web-based system for idea selection
Betting on new and promising ideas is fundamental for innovation in various sectors, helping the world that faces so many adversities. To increase these ideas, the stages of evaluation and selection of ideas in the creative processes need to be improved and not neglected. To minimize this problem and contribute to the scientific advance in the creativity-oriented HCI, we present a tool for selecting ideas used after brainstorming sessions. Our prototype is inspired by the conclusions of an observational study that we conducted with six students in a classroom, in which the students’ objective was to plan the making of a video for a social institution that helps the neediest people.
Nowadays, a vision-driven culture dominates the immersive design. Since the digital immersion mostly depends on the high-performance and expensive head-mounted device (HMD) to operate a complex 3D virtual world for stimulating the visual illusion, financial, technological, and accessibility barriers exist to consuming these devices. There is a significant amount of prevalent research on provoking participants’ visual senses in an immersive experience. However, the audio aspect, often auxiliary to the visual illusions in immersive experience design, remains underexplored. This poster describes the creative process, design rationale, and preliminary findings of Taoisonic Zone, an interactive interface rooted in ancient Taoism and traditional Chinese five-elements musicality aesthetic. My design provides an affordable, minimalistic, and interactive web-based audio immersion experience, contributing to the small but exciting corpus of works aiming at emphasizing audio instead of visual experience in meditative immersion.
In this paper, we present our exploration of using and interpreting a generative tool, fiction collages, which integrates different perspectives in a materialized form and constructs narrative design ideas through a co-speculation process. We conducted a workshop with a research group of multidisciplinary experts who were ideating techno-social interventions for Yunlin, an aging county in Taiwan. The purpose of collaging is to provide a malleable but robust metaphor that stays open to interpretation. Four types of visual materials, e.g., app icons, everyday objects, scenes photos, and local publications, are served as content for narrative construction. The results illustrate how the generated collages facilitate situated co-speculation through both solo- and co-narration among experts. The juxtaposed collages and elicited stories as a whole construct the design fiction for near-future Yulin. To conclude, the study contributes to the community with a narrative way of using collages for co-speculation that aims for mutual informing.
Khong Khro is a reflected light artwork that visualizes the participant’s focus and attention. It uses a commercial EEG headset to monitor the participant’s beta and theta brainwaves. When the viewer is focused the displayed color field unifies into a single hue, and when distracted the field separates into multiple different hues with a mottled pattern. The movement of the color patterns is regulated by a three-dimensional flocking algorithm, and the system runs on a microprocessor. Using biofeedback adds interactive dynamics to the aesthetics of the artwork. This paper describes the techniques and processes used in the artwork’s design and places it in context with other brain art pieces.
Quantum Mechanics could have fundamental impact on design models and measurement. Quantum mechanics allows us to fill in the blanks of classical models of design, through its ability to explain ambiguous states of design. An ambiguous state is where design exists in between two binary states, as a superposition. Designers are most likely to be unfamiliar with quantum mechanics, as well as the subject of quantum mechanics being complex and sometimes contradictory to human scale mechanics. By discussing the opportunities of quantum mechanics for design, we are proposing a framework to model and measure ambiguous dimensions of design through quantum superpositions. The proposed framework includes the dimensions for the directionality of design (convergence or divergence), the degree of design embodiment (from low to high) and the decision-making of the designer (yes to no). Once the designer attempts the measurement of a superposition, a binary state can be distilled. For the act of designing, filling in the blanks is equal to sculpting away superposed states. In this philosophy, to design is to measure. This early stage research raises areas of opportunities and suggests further research directions for quantum mechanics and design.
Stories of Herbs is a visual, auditory, tactile and olfactory VR experience around the theme of herbs. Our goal is to advance the understanding of how smell can be used actively in our interactions within VR. Specifically, we use physical objects – a mortar and pestle – with which participants actively create smell as part of their action in a corresponding world within VR. In the current experiment participants crush mint leaves, and corresponding virtual leaves guide their attention to an animated story about mint. We examined how enjoyment, recollection of the story, the experienced connection between physical and virtual, and the perceived role of smell were influenced by crushing real mint leaves or odorless paper ‘leaves’. Initial findings from an exploratory study with 15 participants show that the smell was a positive element in the experience, that participants in both the odor and odorless groups experienced a strong connection between the physical and digital elements, and that while some participants felt that smell helped them remember the story, the smell may have contributed to a sensory load that took attention away from the story.
How Different Artifacts Elicit Different Caregiver-Child Interactions: An Examination of Book Sharing and Puzzle Play
Interactions between children and their caregivers represent an important factor of child development. Book sharing and other play interactions are common ways in which caregivers and their preschool-age children interact. Shared book reading has many benefits in early childhood, but some researchers have suggested that children may become passive in such interactions. Additionally, with caregivers having sole access to the information in the text, they may be less open to contributions the child puts forth if they conflict with the text. In contrast, a more symmetrical and cooperative activity, such as putting together a puzzle, may elicit more participation from the child and less categorical input from the caregiver. In a study with 59 2- and 3-year-olds and their caregivers engaging in one of these two activities, we find that interactions centered around the puzzle artifact are characterized by more meaningful participation on the part of the child and less definitive corrections on the part of the caregiver compared to book-based interactions. These findings suggest that alternatives to shared book reading with 2- and 3-year-olds may nudge children to express themselves more creatively when interacting with caregivers. Implications for the design of learning experiences for preschoolers are discussed.
Demonstrating DIY Methods for Actuating Morphing Matter: Hands-on methods with off-the-shelf materials to actuate flexible substrates.
The field of Morphing Matter (MM) is often associated with high-tech equipment or inaccessible lab environments. In this work, we explore morphing matter applications within the context of DIY craft, showcasing different actuation methods for morphing artifacts created using compact, inexpensive, and easily replicable toolkits. This demo presents three morphing matter toolsets that enable novice learners to design shape-changing food, artistic figurines that transform underwater and inflatable wearables. We aim to thoroughly demonstrate each step in the design process for morphing arts and crafts projects triggered by various stimuli such as heat, moisture, and air pressure. Through open-source simulation tools, readily available materials, and easy-to-follow tutorials, morphing matter toolkits can empower users to express their creativity in new ways while learning science and engineering concepts, skills, and techniques.
First Author’s Name, Initials, and Last Name, Second Author’s Name, Initials, and Last Name, and Third Author’s Name, Initials, and Last Name. 2018. The Title of the Paper: ACM Conference Proceedings Manuscript Submission Template: This is the subtitle of the paper, this document both explains and embodies the submission format for authors using Word. In Woodstock ’18: ACM Symposium on Neural Gaze Detection, June 03–05, 2018, Woodstock, NY. ACM, New York, NY, USA, 10 pages. NOTE: This block will be automatically generated when manuscripts are processed after acceptance. This technical demonstration introduces Tekniverse (Figure 1), an end-to-end Internet-of-Things (IoT) toolkit designed for education, targeting middle and high school students as a tool for environmental data literacy. Tekniverse is a learning platform for data driven projects that teaches you how to create a network using physical objects — “things”— that can sense the environment, analyze data and exchange information. Tekniverse blends physical hands-on and “virtual fieldwork” to increase and equalize student resources. These features further cultivate critical thinking and anticipatory skills, acting as a vehicle for students to collaborate.
Sport Sparks™ is a new digital tool that has been researched and developed to augment the creative thinking of sports coaches when resolving challenges experienced by athletes. Most sports coaches work across multiple training sites and have limited opportunities and resources for creative thinking. The Sport Sparks™ tool was designed to provide ready-to-use guidance for creative thinking, via mobile devices, that is generated using automated natural language processing, rule-based reasoning and creative search. This short technical demonstration paper summarises and demonstrates with examples the different strategies and features implemented in Sport Sparks™ to augment coach creative thinking.
Calliope is a web application for co-creative multi-track music composition (MMM) in the symbolic domain. It is built to facilitate the use of multi-track music machine (MMM). The user can upload Musical Instrument Digital Interface (MIDI) files, visualize and edit MIDI tracks, and generate partial (via bar in-filling) or complete multi-track content using the Multi-Track Music Machine (MMM). Generation of new MIDI excerpts can be done in batch and can be combined with active playback listening for an enhanced Computer-assisted Composition (CAC) workflow. The user can export generated materials as MIDI files or directly stream MIDI playback from the system to their favorite Digital Audio Workstation (DAW). Calliope can be used for creative ideation and for exploring alternatives of musical phrases in composition.
JECT.AI is a research-based digital tool that was developed to augment journalist creative thinking. It integrates natural language processing, creative search and interactive creative thinking guidance to support journalists to discover novel ideas, angles and voices when writing new articles. This technical demonstration paper summarises JECT.AI’s architecture, algorithms and key interactive features.
Voice assistants have become a mainstay of our increasingly smart domestic spaces. Within this design-through-research inquiry, how new forms of intelligence might creatively respond to everyday interactions. In this, we are particularly interested in non-verbal communication and expression as a design material for interaction design. In this demonstration, we present a prototype illustrating this opportunity: a tinyML-enabled helper that augments the Google Home and enables it to recognize and respond to everyday phenomenon such as a sneeze, a door opening, or rapping on a table. The helper offers a creative toolkit for designers and end-users to quickly explore, understand, and imagine the affordances of everyday sounds.
We introduce the Idea Machine, a creativity support tool that leverages large language models (LLMs) to empower people engaged in idea generation tasks. The tool includes a number of affordances that can be used to enable various levels of automation and intelligent support. Each idea entered into the system can be expanded, rewritten, or combined with other ideas or concepts. An idea suggestion mode can also be enabled to make the system proactively suggest ideas.
The current pandemic has forced many galleries and museums to shut their doors for several months. These closures have created a void in social gatherings, cultural events and outgoings and have urged artists and cultural workers to rethink ways of making, exhibiting and bringing their work to the general public. In this project, we propose to free the gallery from the constraints posed by its traditional spatial and cultural configuration, as well as its new relocation online, by redesigning it as a multipurpose mobile object emerging in dialogue with the city and its population. The result is a complete rethinking of what the gallery space means and represents: in a disrupted world where the need for safety limits the need for physical interaction, the following probing questions emerge: How can we imagine alternative formats, new forms of interactions, new safer and more inclusive spaces? Can we take the gallery to the streets and turn it into an emergent space? What are the possible experiences it would generate? What new dialogues can a redesign of the gallery as a living, breathing entity foster?
POSTcard Landscapes from Lanzarote I and II are two AI-generated video works accompanied by two sound pieces. The StyleGAN 2-created visuals were composed of location-based datasets, more precisely, from Flickr photos of Lanzarote – a tourists’ Mecca of Canary Islands. One video intends to represent a touristic, and the other, a local view of the place. The project draws attention to the influence of the tourist gaze on the landscape and identity formation of the Lanzarote island in Spain. Heavily dominated by the imaginative geographies that have been constantly reproduced by the visitors, create a conflict between desired touristic rituals, which one is preprogrammed to reproduce when arrived at the destination, and the reality. Better said in the words of an expert on tourist photography Jonas Larsen “[…] circulating images overpower reality: ‘reality’ becomes touristic, an item for visual consumption.” 
What do news landscapes look and sound like? And how can we transform them into tangible objects to see clusters of over-reporting, or to hear the missing voices in under-reported topics? A new artwork called Fragile Perspectives was developed to explore how news coverage clusters for different themes. Using a very large news information set generated to explore creative angles and voices in news, new tangible objects of news landscapes were produced by clustering 10000s of individual news articles into 300 dimensions, then reducing these dimensions to 3-dimensional maps. These 3-dimensional maps were then manufactured as glass objects produced using the glassblowing techniques similar to those perfected on the Venetian island of Murano, then digitally scanned to excite virtual strings to produce the sound using a data sonification method based on physical modelling sound synthesis. These glass objects and the generated sound were combined to produce the Fragile Perspectives artwork.
Coffee cup reading is a popular social activity in some cultures and seen as a playful way of speculating on a person’s past, present and future life. If we think of coffee cup reading as a way to envision futures, can we use this practice as a method for creating speculative design fictions? This paper proposes a workshop in which 10 people living in Venice will enact future inhabitants of the city and read each other’s coffee cup in pairs to create speculative design fictions that envision Venice in 2030. The workshop will result in a catalogue of design fictions comprising text and drawings produced by each pair that will present different future visions for Venice. After the exhibition, this catalogue will be exhibited in a dedicated online space to present the design fictions and photos and/or videos created during the workshop as well as visitors’ feedback. The workshop aims to provide insights about how coffee cup reading as a traditional way of looking into possible futures could be used as a low-threshold design activity to reveal urban inhabitants’ visions and concerns towards uncertain futures as well as today’s challenges.
Water Mirror is an aesthetic experiment in perception and interactive art. It is a new development from a series of prior works concerned with light, wind data and place. Comprising synthetic and video imagery viewed through two-way mirrored glass, Water Mirror focuses on effecting a distended perception for the viewer to occupy the in-between space of the physically present self, various virtual selves, and a virtual representation of the city. The experience is one of leaning into an organic unfolding, with your image and self being revealed from different angles and times, while ‘submerged’ in ‘virtual waters of Venice’. This draws on the shimmer of light that sparkles on water surfaces as wind disturbs them. Here, this visual language informs a data visualization of weather conditions in the lagoon outside. It is an organic, fluid interaction that moves the viewer into hybrid spaces, where the self is mirrored back to you within an abstract rendering of weather data.
What could the Venice of the future look like? A project for an ideal city can take off from an allegory: arts and sciences inside a seed of Lodoicea maldivica, whose bipartition reminds us of a human brain. From the seeds left by the past, we derive the vision of the future. The ideal city certainly needs brains able to conceptualize images and develop ideas, and bridges to strengthen connections and interactions. A well-working brain needs “bridges” as connections between ideas and techniques. My vision for a future city contains a livable and stimulating space enhancing at one time creativity, enthusiasm, and scientific development. To this aim, I use the image and the metaphor of cerebral hemispheres, specialized in activities of different typologies yet interconnected. The image of the brain is one of the symbols of cognition. The right hemisphere deals with artistic expression, while the left hemisphere refers to logic thinking, mathematics, scientific attitude toward the world. Different cerebral areas, which contribute to the complex and rich life of an individual, are a metaphor for different places in a city, contributing to the completeness of a community life. Separation between hemispheres, also as a homage to Venice, is seen as a sort of Canal Grande, and connections are represented by bridges. The whole brain is seen as a giant seed of Lodoicea maldivica: the city of the future needs to develop from seeds, that is, knowledge inherited from the past and new ideas from our minds and thoughts. Thoughts that, in turn, are fed upon the Science of Complexity.
A virtual reality artwork is presented featuring procedurally generated content created from individual viewers’ Google user profiles. Drawing on folklore of the selkie, a shape-shifting seal whose enchanted skin is a source of both power and vulnerability, the work explores the complexities of our relationships to the digital footprints we leave behind as we move through contemporary urban and virtual environments.
Digital-Tropical: Venice of the East highlights the significance of traditional cultural wisdom in the contemporary context of global warming. Although the Malaysian climate is different from Venice, however, like Venice, houses in Malaysia are traditionally designed to function around water, and in the rise and fall of river levels due to the monsoon and tides. Traditional Malay houses are made with timber using modular design that can be reused for generations. Although the houses are built high on stilts, there are doors which do not have stairs that are only used during flooding to access boats. In this artwork, a LiDAR video of a 300-year-old traditional Malay house, is juxtaposed with visualization of historical ocean wave height data using light projection. The data were collected near Tioman Island for two years for a renewable energy project. Traditional Malay houses ‘work around’ the natural environment to function in the uncertainties of nature unlike many technologies which arose from scientific advancement that ‘subdue’ the natural world by making irreversible and detrimental changes to it. Renewable energy technology also needs to function in the uncertainties of nature by ‘working around’ it. We need the knowledge of wave patterns to generate electricity. Thus, this artwork juxtaposes the tropical architecture of the old and new technology paradigm, both that ‘work around’ nature, to preserve nature. Malaysia came under Western colonization with the fall of the entrepôt city of Malacca in 1511. The title of this artwork is inspired by Vasco da Gama, a Portuguese explorer, who once described Malacca as “Venice of the East.” With this, we would like to highlight that even though Malaysia and Venice are separated by climate, cultural and geographical distance, we are living together in a warming world which will affect Malaysian cities as much as it will affect Venice.
By allowing participants to co-construct geolocated Venetian soundscapes through interactive, sonified objects, this artwork re-imagines how humans can coexist in post-pandemic public spaces.
Pelican Stairs is a multimedia art project started during the depths of Covid-19 lockdown in 2020. The project’s reflections on how one’s umwelt (one’s unique experience of living in one’s environment) changes in response to an all-encompassing crisis can serve as a guide, or a provocation, about how future cities need to be reimagined to encompass flexible life-pattern options for their denizens not only to survive but to thrive through the challenges before us. I took photos of my local neighbourhood, Wapping in London, on my daily walks between March and September. I often visited the Thames shore area by climbing a set of steps known as Pelican Stairs. Using those photos as a training set, I generated new images with a Generative Adversarial Network (GAN), which I paired with excerpts from my diary during the same period. The unsettling, almost-real images contrasted with the mundane reality of the diary entries allow the viewer to viscerally (re)experience the inherent tension between an increasingly uncertain external reality and internal attempts at control or sense-making through normal, everyday habits. The project is hosted at https://pelicanstairs.art, and the project includes an interactive element through a Twitter bot which responds to a specific prompt by sharing a random image of generated art from the project.
Lure of Slowness, hydrological Rhythms is an interactive, audiovisual augmented reality installation and multi-channel sound piece co-imagining watery interspecies relationships through the gelatinous lens of a snail. The Virtual installation is housed in the application Lure of Slowness available for download on Apple and Android store.
SESSION: Graduate Student Symposium
Sketching is a widely used activity in creative processes. One can argue that we now have Artificial Intelligence (AI) capable of making better drawings and sketches than humans. Some computational drawing applications build upon human input, be it text, an input image, or even a hand-drawn path using the cursor. However, drawing along a present-day computer yields a very different experience from drawing collaboratively with another person. For one, the shared canvas is likely not a physical sheet of paper but rather a screen of some sort. Even with paper-like computer interfaces, the computer mainly draws back virtually, needing no material embodiment to make a drawing. My research investigates human collaboration with computers that draw, specifically focused on the act of sketching. I build exploratory physical systems that allow people to draw and sketch together with a computer. Through my studies, I investigate interaction design to allow for human-computer collaborative sketching. My initial studies with a first experimental platform show that the computers are social actors (CASA) paradigm also applies to non-anthropomorphic pen plotters when performing a Tic-Tac-Toe playing task.
Data physicalization enables people to represent and interact with data physically rather than digitally. Physical representations afford visual analysis in comparable ways to traditional, desktop-based visualization by introducing new capabilities, such as facilitating tactile manipulation, accessible interactions, and immersion, that are beyond traditional 2D visualizations. However, physicalization has historically been a niche aspect of visualization research due to its unique challenges. In this paper, I discuss the current challenges of data physicalization and address three areas where data physicalization can aid other research thrusts: broadening participation, supporting analytics, and promoting creative expression. This paper exemplifies each approach through the lens of my work.
Redefining Access to Large Audiovisual Archives through Embodied Experiences in Immersive Environments: Creativity & Cognition 2022 – Graduate Student Symposium
Audiovisual archives are the mnemonic archives of the 21st century, with important cultural institutions increasingly digitizing their video collections. However, these remain mostly inaccessible, due to the sheer amount of content combined with the lack of innovative forms of engagement through compelling frameworks for their exploration. The present research therefore aims at redefining access to large video collections through embodied experiences in immersive environments. The author claims that, once users are empowered to be actors of the experience rather than mere spectators, their creativity is stimulated and narrative can emerge.
When scientists and artists engage in exploratory research, they can reach new insight through unexpected information encounters. As both glitch and serendipity share the connection of discovery through the unexpected, I encourage pushing the limitations of interfaces, structures, and systems to bring about novelty. This experimental practice-based research engages with a new framework of alternative information seeking that opens up potential to discovery, known as glitch serendipity. I share the current status of my research along with a concrete example of how digital experiments led to a serendipitous demonstration of nonlinearity in image expression. Future work considers a holistic cognitive framework of surprise mediated by the values of unpredictability, unexpectedness, and unfamiliarity.
Creativity is an important driver of innovation and corporate success. Due to the digital transformation, creative collaboration increasingly occurs in virtual teams. This raises the research question how to design digital work environments to foster creative virtual collaboration. Therefore, this PhD project aims to develop design knowledge for virtual collaboration based on the Design Science Research (DSR) approach. The identification of Creativity Drivers (CDs) anchors the descriptive knowledge in a rigorous theoretical foundation. Prescriptive knowledge about design requirements will be derived from expert interviews with creative professionals. To combine both, prescriptive and descriptive knowledge, Design Principles (DPs) will be developed that address creativity drivers in five areas: functionality, process, mood, meaning, and collaboration. The appropriateness of the DPs will be validated by interviews with experts. The results will be used to review the CDs and refine the DPs. Further, the DPs will be used to define an instantiation of a virtual working environment for creative groups, which will be evaluated in an experimental setting. The results contribute to the scientific literature by combining and expanding relevant theories. The practical contribution lies in the applicability of design knowledge in different business-related and educational contexts.
This abstract outlines the research trajectory during my doctoral study, including completed work and future plans. My primary research focus is fashion and sustainable practices in the context of companion robots. After exploring people’s spontaneous behaviors of dressing up their robots, I want to further extend the connotation of robot fashion and use fashion and sustainability as frameworks to explore how people treat their companion robots.
In this paper, we present an overview of the problem by discussing the landscape of extended reality (XR) and attention guidance techniques considering the effects of current trends, cybersickness, presence and user performance. Then, we discuss designing and rapid prototyping for immersive storytelling as a case study on the applications of these techniques and their related challenges. In conclusion, we propose a research plan that addresses two interrelated themes, and potential research directions to ensure better ways to design and rapid prototype for attention guidance techniques in XR and, as a result, better XR experiences.
The thesis project presented aims to use social information in conjunction with modern AI/ML techniques to develop artificial creative societies. The main objective is to explore social creativity as it could be. The scope consists of three core aspects: adaption, intention, and evaluation. Current work exploring mechanising conceptual spaces is discussed, and future work directions are provided. The research trajectory consists of three phases. Each phase explores the core aspect concerning the individual, the field, and the domain. This work contributes new approaches toward adaptive CC systems, evaluation methods, and subsequently, the potential to inform other disciplines, such as art & design.
Alternative Controllers (Alt Controls) enable game designers to creatively explore how humans interact with games and challenge the status-quo of game interfaces. Alt Controls, however, require technical skills and fabrication infrastructure that often make them inaccessible to the average designer. Tangible User Interface researchers stand to benefit from the unique approach that Alt Controls promote. My research aims to bridge the gap between game developers and Alt Controls through the use of everyday materials and crafting techniques. In this paper, I discuss a framework for physical computing that uses computer vision (Beholder) and an example introductory platform for Alt Controller design (TinyCade). Further research will refine this framework and incorporate the perspective of other game designers.