The Visit

RESEARCH STATEMENT

The Visit is an interactive real-time video installation and Virtual Reality experience, developed from a ground-breaking interdisciplinary research project conducted by artists and psychologists working with women living with dementia. Visitors are invited to sit with Viv, a life-sized, realistic animated character, drawing us into a world of perceptual uncertainty, while at the same time confounding stereotypes and confronting fears about dementia. The characterisation has scientific validity but also the qualities of a rich, emotion-driven film narrative. The point of the work is to draw the viewer into the emotional/perceptual world of Viv.

​

The script is created largely from verbatim interviews, utilising grounded practical theory for unpacking the interview data to connect feelings of loss, attachment and meaning to ideas, events and objects. Like the women who co-created her, Viv experiences various dementia-related symptoms, including hallucinations and confabulation, but she is also insightful and reflective. Viv is living a life and coming to terms with a neurological change.

​

The character is sensitive to the motions of the viewer, she is ‘aware’ of the presence in her home by making eye-contact and directly addressing the visitor. The narrative develops in two parallel streams. Virtual production techniques such as digital human, motion and performance capture and virtual set have been employed to create this non-linear experience. The film is rendered in real-time in a computer game engine.

​

This project investigates the affective potential of immersive technologies and content from in-depth qualitative research with people with lived experience of living with dementia. The ageing population and dementia are key issues facing society in the 21st Century. Biomedical, social science, and neurological perspectives are increasingly well-documented. What is not understood well is the lived experience. The project develops new methodologies for engaging with people living with dementia to understand their experiences and uses immersive technologies to embed the audience as a visitor into Viv’s world.

​

The project contributes to research in three ways. It takes on the issues related to ageism and stigmatisation in relation to dementia. It engages in innovative methodology development for understanding lived experience. It investigates how the audience through the use of technology become immersed in the visit with Viv. In doing so they become aware of their own responses to the character and the condition and, as evaluation has shown, increase empathy, and decrease emotional distance.

​

The Visit VR has been invited to be shown to Dementias Australia CEO and staff in Melbourne and Sydney. Audience members commented on the experience with Viv suggesting “It has definitely opened my eyes to the creative ways we can express or translate mental health research to make it more applicable to the general population.”

​

CreditsDirector and Producer: Jill BennettArt Director and Production Design: Volker KuchelmeisterCast: Voice actress: Heather MitchellMotion Capture actress: Emma KewScript: Jill Bennett, Natasha GinnivanScript consultants: Melissa Neidorf, Gail Kenning, David PledgerInterviewees/contributors: Prue Uther, Joan Eva, Jennifer Bute, Wendy MitchellResearchers: Natasha Ginnivan, Chris Papadopoulos, Melissa Neidorf, Gail Kenning.Motion Capture: Joe HollowayVoice recording: Tobias Gilbert3D Artist: Chris Norris, Jason Dobra3D scanning: Matt Cabanag (UNSW LITEroom)Music: Peter Sculthorpe

​

Special thanks: Stephen Sewell, Luis Dominguez, Karen Kriss, Steve Weymouth, Eva Nolan, the A&D resource centre team, Ant Banister and the participants in the study.

​

Collaborators with lived experience are at the core of our work. Collaborators may choose to remain anonymous or to be credited using their real name or a pseudonym. In each case, the decision rests with them (where appropriate in consultation with carers/family). We thank them all.

​

FundingThe author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this project: This research was supported by the Australian Research Council under its Laureate Fellowship program (grant number FL170100131), awarded to Professor Jill Bennett.

​

​

PEER REVIEW 1

​

The use of new tech to provide insight/empathy/understanding into dementia as an increasingly significant social issue is the key significance and contribution of this project. Its interdisciplinary nature, with narrative/cinema clearly informed by science, is a key strength, and central to its potential impact/application. Character is well-realized (design and script), the set provides authenticity. The distinction between this and other similar projects (which evidence a first-person POV) of a disembodied second person pov is particularly effective in generating a conscious empathy for the protagonist; further, the viewer’s positioning in the subjective experience (hallucination) of the protagonist creates presence and engagement.

​

Alongside the strengths indicated above, the few reservations I have regards the project’s realisation include the floaty robotic nature of the acting/ movement, and the fairly limited range of facial expressions, both of which tend to alienate rather than bring us closer to the subject. I’m aware of the technical limitations, however, these effects might have been attended to at the mo cap stage, or in clean up — traditional character animation might have been employed for facial expressions, timing and ticks, towards enhanced naturalism; the (lack of) movement of the eyes can be disconcerting.  I also question the use of some of the musical interludes (during times of distress) and sound fx (bees) — for me, they distance me from the subject rather than pull me in with a curiosity as to what she is experiencing; my second-person POV is disrupted.

​

That said, scripting, voice-over, production design (set, props, character modelling) all contribute very well to a holistic and compelling portrait.

​

The following related questions are clearly addressed in The Visit:

• How can new tech best be utilized to address health/social issues, specifically as an empathy tool?

• How might new tech leverage cross-disciplinary collaboration to interrogate a social/health issue?

• How can an audience be positioned for the greatest insight into, and empathy for, subjective psychological experience?

Due to its tremendous technical challenges, any project in VR, and that utilises realistic avatars, has endured many problems solving, evidenced creative nimbleness, and therefore innovation in its realization.

The positioning of the viewer as a second person POV, and implicating them in the narrative —for greater empathy and presence— is innovative.

​

Is the work contextualised within specific social/artistic theoretical fields? 

Yes: immersive VR as empathy machine; new tech in service of/interrogation of health issues.

​

Is there evidence of new knowledge, interpretation, insights or experiences? 

This compassionate portrait of a dementia sufferer, firmly rooted in a familiar cultural and social context, evidences significant attention to detail based on cross-disciplinary research. It is therefore a unique, compelling and clear contribution to the arts in service of health.

​

The positioning of the viewer as immersed second-person visitor, embedded into the narrative and protagonist’s experience, is impactful.

​

​

PEER REVIEW 2

​

This project has significance in relation to three principle sites: 1) interdisciplinary collaboration between creative screen media and health professionals/ psychologist; 2) the approach to a narrativised script taken from interview material; and 3) the use of advanced virtual production technologies to represent and embody character.

​

The execution of the work is at a high level with regards to the conceptual understanding of the potential of the VR medium to induce empathy and the execution of the work for an audience. The core elements of the single-space kitchen, the use of sound/music and dialogue drawn from real-world interview sources is effective. The authors may consider duration in the context of research on duration of works for an audience – theorised to be optimized at the under 10m mark. With respect to this particular character-based scenario – dementia is the end of a person’s life and one that has the capacity to not value that the prior connected experience of the character. Some sense of the degree to which dementia represents a fundamental change in character (often a sad and complicated one) may benefit from celebrating/recognising aspects of the lived-life of happier times and moments and connectivity. I may have not comprehended this fully – but the ways in which ‘the evaluation has shown’ to increase empathy and decrease emotional distance – may need consolidation.

​

This submission presents a rigorous and effective model of practice as research in relation to the creative development (scripting) and hybrid production/post-production execution. It sets out a series of interwoven research questions which are engaged with (can a virtual mode of lived-experience VR produce a higher degree of empathy and emotional engagement). It is innovative in relation to virtual production technology and in relation to the organization of virtual reality media. The forthcoming article cited in the statement would be a welcome contribution to accompany the work and unpack the question of an evaluation of the ways this work evidences an improvement to empathy/engagement in relation to dementia.

​

​

RESEARCHER RESPONSE

​

I would like to thank the reviewers for their considered and insightful feedback. In our response, we would like to address some of the observations and expand on the research statement, with a focus on viewer representation, interactivity, production design and evaluation.

​

Viewer Representation

The audience does not visually embody a specific character. However, they are also not an objective viewer. The perspective is subjective as from the viewpoint of a visitor, a guest sitting with the main character (Viv) at her kitchen table. The viewer, as it emerges from the script, becomes a disembodied hallucination. The project uses an unexpected approach to first-person perspective in Virtual Reality by invoking a strong sense of presence while keeping them in a disembodied state. The fEEL (UNSW felt Experience and Empathy Lab) focus is on a new approach to the study of subjective and lived experience by simulating seeing through the eyes of another. Previous VR projects (waumananyi: the song on the wind and Being Debra) are conceptualised around the idea of embodiment by consequently applying a first-person perspective. The Visit takes a deliberate departure from this paradigm to draw the audience into the ambiguous perceptual world of Viv.

​

Interactivity

Interactivity operates in The Visit across disciplines. The audience in The Visit is an active participant in the narrative but with limited agency. In terms of the psychology aspect of the cultivation of empathy, the audience experience is interactive, in their coming to understand the perception of the character (with dementia and hallucinations) and in recognising their own response to this realisation. In terms of immersive media, not unlike a computer game, the audience’s actions have consequences in the virtual world, not only in the spatial configuration (6 degree of movement and real-time rendering), but also in terms of the character response and the narrative flow. The character is ‘aware’ of the audience’s presence in her home. She makes eye-contact, smiles and addresses them directly. Interactivity is also a mechanism for the screen-based gallery version. Once the audience departs the character turns around and walks slowly to her default spot, looking out of a window, only to come back if someone approaches the screen. She then encourages the viewer to take a seat and acknowledges the response.

​

The non-linear narrative structure is comprised of an opening scene, two sets of two parallel storylines and a closing scene (diagram below). The individual branches are selected randomly. For a viewer, the experience varies across multiple viewings with a total duration between 13:10 and 14:24 sec.

​

Production Design

The Visit is designed as a real-time computer-generated interactive film, utilising virtual production (VP) methods. VP is a broad term referring to a spectrum of computer-aided production and visualisation filmmaking methods. It is where filmmaking and computer game technology or the physical and the digital world meet. VP merges traditional practices with current and ongoing advances in real-time technology to enable filmmakers to make better creative choices much earlier in the production process. For a filmmaker, this is probably the most exciting and enjoyable part of the process. The system encourages creative experimentation with instant feedback and quick iteration.

​

The character and set in The Visit are represented as digital 3D models, created by employing body and spatial 3D scanning. To reduce resources required for animation, the character’s movement is driven by motion capture, lip-sync is based on a machine learning model and the facial expressions recorded in real-time with a custom tactile animation system. Her emotional states are limited to a basic repertoire and the voice actress was directed to keep the characterisation contained. A procedural animation layer allows for gaze control and dynamic emotive expressions.

​

The environment the character inhabits is based on a real location, a former home of one of our project collaborators. The set is comprised of a kitchen, a living room, furniture and various props, digitised utilising photogrammetry. While relatively complex in their creation, 3D scans offer a range of advantages over traditional hand-crafted models. It captures not only the geometry of a scene, but the quality of the light and shadows, natural imperfections and variations of materials, all time-consuming to achieve with traditional modelling and lighting techniques (Kuchelmeister 2020).

​

Evaluation

Awareness of dementia is growing. However, dementia continues to be stigmatised and misunderstood. Often people with limited experience of dementia relate the condition to the symptoms exhibited in the later stages of the condition. They fail to recognise the agency, abilities and coping strategies employed by those in mid and earlier stages. The Visit engaged directly with these misunderstandings in creating a character with dementia and agency. Building on existing research showing arts-based approaches can promote empathetic interactions using immersive technologies, we designed The Visit as an experimental form of knowledge translation that would create conditions for cultivating empathy. This was tested by evaluating changes in state empathy and emotional distance/aversion. The study used a measure of state empathy (CSES) which assumes that empathy can be trained and improved with appropriate interventions. It measures changes before and after exposure to the interventions designed to increase empathy (Levett-Jones, Lapkin, Govind, Pich, Hoffman 2017). CSES consists of 30 questions across six subscales (empathic concern, distress, shared affect, empathic imagination, helping motivation and cognitive empathy). The study also used a measure of ‘emotional distance’ specifically designed for this work. It provided an index of changes in how emotionally uncomfortable/aversive people found their engagement with an older adult with dementia before and after exposure. Audiences to The Visit were invited to complete pre- and post-engagement questionnaires. The findings showed state empathy was significantly greater after engaging with The Visit, and correspondingly, there was a significant decrease in emotional distance (aversion) (Papadopoulos, Kenning, Bennett, Kuchelmeister, Ginnivan & Neidorf 2021).

​

References

Kuchelmeister, Volker. 2020. “Virtual Production and Realtime Filmmaking technologies for the independent filmmakers: An Overview” Volker Kuchelmeister. FKT Fachzeitschrift für Fernsehen, Film und Elektronische Medien. Vol 74, Berlin.

​

Levett-Jones, T., Lapkin, S., Govind, N., Pich, J., Hoffman, K., et al. 2017. “Measuring the impact of a ‘point of view’ disability simulation on nursing students’ empathy using the comprehensive state empathy scale,” Nurse Education Today, 59: 75-81.

​

Papadopoulos, C., Kenning, G., Bennett, J., Kuchelmeister, V., Ginnivan, N., & Neidorf, M. 2021. “A visit with Viv: Empathising with a digital human character embodying the lived experiences of dementia.” Dementia: The International Journal of Social Research and Practice. https://doi.org/10.1177/1471301221998888