Context: Master's Degree Capstone/Thesis - I worked with an academic supervisor to conduct design research into a topic of choice. The outcome was a design project along with a research paper detailing a framework for effectively designing playful interactions with hybrid resolution media facades
My Role: Solo research project - Researcher, Designer and Programmer
Academic supervisor: Luke Hespanol
Background
The Footbridge Gallery
The Footbridge Gallery, located at the University of Sydney, is a digital placemaking area with embedded light and sound technologies opened in 2019. It is the brainchild of Dr Luke Hespanhol and Associate Professor Martin Tomitsch from the University’s School of Architecture, Design and Planning.
For this project I was to create a design and conduct research into a topic of interest using the technologies at The Footbridge Gallery which consisted of a large projection on the building (High resolution) and programmable LED strips (Low resolution)
Literature Review
Background Research
Understanding what has been done before
The first step was conducting a literature review through reading numerous academic journal papers, articles and websites. The aim was to gather information and analyse it to gain an idea of what has been done in this subject area and the type of knowledge that has been created. Since “Research is defined as the creation of new knowledge and/or the use of existing knowledge in a new and creative way so as to generate new concepts, methodologies, inventions and understandings” (Australian Research Council), it was an important step to find a gap in which my research could be directed and scoped to fit into this field in a way that would generate new knowledge.
I then synthesised the information organised it into these topics:
Smart cities - Data about the city and its citizens are collected through pervasive technologies such as wearables and social media to build a city that is responsive and adaptive to the needs of citizens, with the main focus being 'efficiency’.
Playful cities - emerged from this opposition to this focus on "efficiency" and control to create an ideal "city as a machine" . Popular concerns include control of citizens, data privacy and concerns that smart cities tend to be driven by rules that focus on productivity and codifying behaviour enforcing a social norm for citizens to be.
Designing for high resolution - many studies have been conducted into effective techniques and strategies to designing high resolution screens such as mobile phones, projections and large interactive screens
Designing for Low resolution - although less studied that high resolution, content and interaction design for low resolution has been studied.
However, with the placement of the high and low resolution mediums being on top of each other at the Footbridge Gallery I decided to look into these technologies being combined together. The current designs that have been at the Footbridge Gallery were also only either only LEDS or only Projection. Thus this became the focus of my project and termed it "Hybrid Resolution".
The Design Knowledge Gap
I also looked at the design precedents and knowledge which could be summarised into this table below which shows a clear knowledge gap that I could aim to address in my research.
I tried to find studies and existing designs that involved hybrid resolution, but there were very limited amount of studies. With that in mind I decided to base my study on designing playful interactions with Hybrid resolution Media facades - filling in that design knowledge gap
What type of Playful Interaction?
Through further research I came across the concept of personifying or animating things in the world. Associating spirits with objects or personifying them has been recognised by Sociologists as a concept widespread among the peoples of the world (Bell, 1997). A modern-day example of this is a wedding ring given by the spouse that would feel different to a replacement that looks the exact same. Bell argues that in the same way as giving souls to objects, we give space social meaning.
Along with the fact that Halloween day was near the project deadline, I decided to create an interactive ghost character with the technologies in the public space to conduct the research.
Define Scope
Aim of the Research Project
There were two components to this Research Project:
A) Design Project
Design and implement a playful interactive character in the public urban space
B) Academic Research
Investigate design challenges and strategies for creating a playful interaction with hybrid-resolution media façades as well as people's perceptions towards such an interface. In particular, the research was to address these questions:
How can high- and low- resolution media façades be combined effectively to design a playful interaction with hybrid-resolution media façades?
What are the user perceptions towards an interactive character media installation?
How does audience behaviour differ towards the installation when coming to the area expecting to interact with it (e.g. an event) vs unintentionally interacting with it (part of urban life)?
The two components were intertwined in that I would be creating the design while conducting research into designing it.
Methodology
Research In and Through Design
I adopted a methodology based on an approach called "Research In and Through Design" where the interaction design researchers are engaged in "the development of an interactive façade in order to generate insights into how to plan and carry out design for this type of interactive systems” (Dalsgaard, 2010, p.1).
This was chosen because there is little established knowledge on the design process for this type of interface (Hybrid-Resolution media facades). The characteristic of this methodology is that it has two parts (seen as 2 loops in diagram):
Part 1 - First series of design experiments to gain insight into how to conduct the research and design aspects to consider for the novel interface
Part 2 - Insights from part 1 can then be used to produce a final design through design iterations while continuing to conduct research
The Study Method
Translating the above process into my project, I decided the two parts could be as follows:
Part 1 : Non-Hybrid-Resolution Design Experiments
3 Studies and a Baseline study looking at the merits of Low resolution and High resolution designs separately
Part 2: Hybrid-Resolution Design Iterations
5 studies conducted that involved iterations of design concepts that combined the High and Low resolutions
What happens in each Study?
Each Study was 1 hour in duration consisting of two 30 minutes sessions:
Session A – running Design 1
Session B – running Design 2
Each study either differed in experiment variables or designs. For each session we kept “interaction counts" by using a tool I developed to tally the number of people who would:
Pass - walked across the installation without looking up at it
Glance - looked up when passing by for a short time, without stopping.
Interact - stopped to look at the installation or perform actions beyond glancing e.g. taking photos or pointing
Observations from a distance and semi-structured interviews were also conducted during the sessions.
Technologies Used
The hardware infrastructure - which includes the LEDs attached to the building facade and Projector attached to building opposite the wall were setup by my supervisor the year before
The interface to interact with the hardware - was set up by my supervisor including the use of screen sharing and ARTNET coding libraries
LED light pattern - I used the Processing Java library to code the LEDs with the base code from interface described above
Projector graphic - I used the Processing Java library to code the animations, control and the OpenKinect Library to interface with the xbox Kinect for detecting passers-by
The Interaction counter tool - coded with Processing Java Library
Part 1
Non-Hybrid Design Experiments
For each study in Part 1, we used the first 30 minutes to show a Low-res design (using only the LEDs) and then a high-resolution design (using only the projection). Both had similar designs so that I could compare the merits and disadvantages of each.
These initial studies also allowed me to test concepts and learn about designing for this particular set of technologies. It gave me the knowledge and practice to implement further concepts and ideas.
These were also all "Wizard of Oz" I.e. the eyes "following" people were manually controlled by my supervisor while I conducted the interviews.
Curiosity Objects: Additional decorations and items aimed to attract curiosity and attention - I stuck paper ghost cutouts near the installation and a put cloth over a statue with two paper eyes
Summary of Findings:
The fact that the LED lights and projection are a permanent installation that has been there for a year and turned on every night meant that people didn't pay attention to a different design - particularly, the low-resolution ghost designs were not recognised to be different from the usual abstract colours
The large majority did not suspect the eyes to be "following" them
While "eyes" were thought to be creepy, adding a face/body to it turned it into a character that was mostly described as "Cute"
Adding the curiosity objects and sound attracted much more attention and passers-by took interest because they thought "It's only today, it's not usually there"
The baseline study saw people to pay attention for longer, leading to a hypothesis that the changing nature of it (different designs throughout the 30mins) compared to the ghost (one design throughout 30 mins) retained more attention
For the first trial of the concept, I decided to put two big eyes on the building facade which would follow people as they walked across
First was the Low resolution LED Design, then the High resolution Projection Design
I found that the location of the installation being on the side of a passageway and up high could have made it hard to grab attention
People could not tell what the LED design was showing, but the High Resolution Projection eyes were easily recognised and attracted a bit more attention
Most said they thought it was a bit creepy to have eyes following them
Since the lights were on every night (with a different design), people didn't realise that these eyes were not normally there
Baseline Study
We spent a night watching the behaviours when the usual content was displayed (since this facade normally has abstract low-res patterns then video projections everynight)
The aim was to obtain interaction counts to have a baseline to compare with
There was a slightly higher interaction count compared to study one
From the observations I had two hypotheses: that the more colourful designs attracted attention and that the changing nature of the designs retained more attention - these two aspects differed from the design in study 1
Study 2: Adding Curiosity Objects + Face
Since we found that people didn't suspect anything different about the design compared to the usual lights, I decided to add "Curiosity Objects"
Curiosity objects as described in literature, are aimed to attract curiosity and attention - I made paper ghost cutouts and put a cloth over a statue with two paper eyes
I also added a Face/Body to the Ghost eyes to make it less creepy as the intention was to make it fun
The curiosity objects, particularly the Ghost cloth statue attracted a lot of attention
People noted that it caught their attention because "It's not usual"
Perceptions of the ghost were "Cute" and "Cartoon like" rather than "Creepy
Study 3: Adding Sound with the Curiosity Objects
For the next experiment I wanted to see if adding sounds and music would affect interpretations or impact the interaction counts
So along with the curiosity objects, speakers played music from a "fantasy magical spooky music" spotify playlist and "ping" sounds when the eyes blinked
Sounds greatly increased the ‘glance’ count
The sounds and music seemed to have influenced interpretation e.g. mentions of “haunted”, “spooky”, “party"
Low-res design was still difficult to interpret
Part 2
Hybrid-Resolution Design Concepts
Part 2 involved creating and iterating on a Hybrid Resolution Design and really started to look what the hybrid-resolution ghost character would be like. From part one findings, two design challenges were chosen as the basis for two Hybrid-resolution design concepts:
Concept 1: Light Shadow
Challenge: No one perceived that it could be interactive
Concept: Using Low-Res as visual feedback for interaction to support the High-Res image.
As people walk across, LEDs will light up to form a “shadow” of the user as they walk across. With multiple people, the person/shadow being “tracked" will be coloured while the rest are white
Concept 2: Hybrid Ghost
Challenge: Low-Resolution image was difficult to recognise
Concept: Creating a continuous image using the High-Res image to help recognition of the Low-Res image.
The High-Res displays the “head” of the ghost whereas the Low-Res will be connected to show the “body” of the ghost. The eye pupils will follow people
Hybrid Resolution Design Iterations
I conducted 3 iterations through 5 Studies. For each study, I split the hour into 30 minutes each for the 2 concepts. Only Concept 1 (Light Shadow) used an X-box Kinect to detect motion, the second concept was kept as manually controlled but the Kinect was left in front of the installation.
Defining the Goal Experience
Although the main aim is to conduct research, I also focused on making it into a great experience. Thus from part one, I defined 2 types of passers-by to cater for:
Regular
Regularly walks past the area
Unamused by current installation as it is always there but does like it as "colourful lights"
First time
Walks past the area for the first time
Unfamiliar with the university and notices the installation, usually wondering if it is a permanent installation or temporary
I then also defined keywords to describe the experience I hope to create for them:
Fun - having no particular purpose other than bringing some joy
Engaging - people would interact with it, more than a glance
Surprise - to be recognised as unusual and out of people's expectations of the installation
Curiosity - create a sense of wonder and prompt people to question it
Creating the Experience
There were also study "pairs" where I used the same designs across two studies but the difference was whether I promoted the study as an event or not (through event posters around campus and social media) - to answer the third research question - Will behaviours and perception differ when people come to the area knowing about the Installation?
Similar to Part 1, I conducted onsite semi-structured interviews and interaction counts were recorded. Findings were analysed and changes implemented for each iteration as seen below.
Summary of Design considerations and findings
I decided to keep the curiosity objects (ghost Stickers, ghost statue and the audio) from part 1 for most of the sessions as people noted that it added to the vibe and particularly the music was a large influence to attract attention - I decided on the "spooky magical" playlist over a more "haunted" playlist as the ghost was not a scary design
For concept 1 (Light shadow), having the light shadow position more accurately align to the position of the user (through technical iterations) greatly impacted understanding of it
For concept 2 (Hybrid ghost) the initial change from only the ghost eye pupils following passersby to having the whole ghost follow the passersby made a big difference to the understanding of the Low-resolution "body" being connected to the High resolution "head"
I then added colour to the design; red, blue and Green themes that changed at random intervals which retained attention. As one interviewee who stopped for a while explained “I was looking at the colours and I was like how many, is it just looping like the one thing?” and after being explained the concept they said “I want to keep watching it, because it’s not on a loop"
I also made the pupils change to a heart shape if people interacted with it - to further enhance the idea that some people said it could be like a companion - the change saw more people ask questions about it as if it was alive like “does it come out every night?” rather than previously (no heart pupil) “is this here every night?”. One stated that it showed “friendliness towards the people…if I come here to chill, I will feel like as if I have a companion nearby so I won't be lonely”.
The "ping" sounds that triggered when the eyes blinked were changed to a cute "pop" sound as people expressed previously that the"game" sound didn't fit with the friendly ghost theme.
As the studies drew closer to Halloween day, people started to guess that it was for halloween and recognise it as a ghost
For both concepts, the Ghost stayed at the middle of the wall with small "floating" movements while the pupils would follow the passerby as they walked across.
There were technical difficulties on the night such as the music failing to play and the kinect tracking was not working as intended so it was switched to manual.
Although the music failed to play, the sound "ping" still played when the ghost blinked. This saw the "ping" sound to be more prominent without the music where the interact rate doubled as people thought "there was a party or game somewhere"
The main problem I discovered was that people did not perceive the high and low-resolution parts to be working together, but rather as separate parts where LED was permanent whereas the projection was the newer part that followed people (eyes) - so for the next iteration I wanted to make the whole ghost move to emphasise the continuous flow as the lower part would have movement too
Study 5 (Unpromoted) & 6 (Promoted event): Hybrid Resolution Version 2
For Light Shadows, the high-res head moved left or right left or right as detected by the xbox kinect while the low-res showed a "shadow" of passer-bys as they walked across.
For the hybrid ghost the whole ghost (high-res "head" with low-res "body") moved as controlled manually to follow passer-bys.
For the hybrid ghost, people were able to perceived the two resolutions as related. Particularly when the ghost moved down, going out of view of the projection where the "eyes" moved into the-low area, people were able to see it.
Surprisingly, for the light shadows, more people realised that they were being followed, but when asked about the light shadow, the majority said they did not notice the LEDs suggesting an unconscious effect that it may have had
Study 7 (Unpromoted) & 8 (Halloween Event): Hybrid Resolution Version 3
For both concepts, I added an animated stars background that changed colours at random intervals and the pupils would turn heart shaped as controlled by us whenever someone interacts with it. The original "ping" sounds for blinking was changed to a cure "pop" sound that match the heart pupil change.
The colour and animation were noted by interviews as key-points of what made it appealing to them
Particularly those who had seen the ghost from my previous studies, they found it more "interesting" and "nice"
The changing colours also meant many wondered "how many colours" or "what other shapes" there are as they stopped to look at it, confirming my hypothesis that the changing nature can retain attention
The heart pupils also led to more interviewees describing the ghost to have "personality" and emotion saying "it loves me" - the pop sound was also said to add to the "cuteness" of the ghost
The concept of the "Emerging champion" as described was also observed, where previous interviewees who frequented the are would point out the ghost and its story to their friends
These last two studies were also close to Halloween, most were able to guess that it was for Halloween which added to the vibe and recognition of it as a"Ghost" for "Halloween"
Final Design - Halloween Day
Ghasper The Campus Ghost
Video of Final Iterations
Outcomes
The RES Framework
Research Question 1: How can high- and low- resolution media façades be combined effectively to design a playful interaction with hybrid-resolution media façades?
Through synthesising the results after each study in terms of which design changes had the most impact, I derived three key considerations and corresponding strategies for designing hybrid-resolution media façades into the RES framework along with the 3Cs Strategy, which could inform future designs of hybrid-resolution installations. This framework and its sample application to this project can be seen below:
Research Question 2: What are the user perceptions towards an interactive character media installation?
Interviewee responses were very positive and showed that I was able to meet my experience goals:
Fun - was a keyword often used by interviewees when asked to describe it
Engaging - some tested out how "fast" they could be detected and engaged with it, many took photos
Surprise - it caught some people by surprise particularly when the colours and heart pupils were added. The curiosity objects added to this
Curiosity - Many questions were asked by interviewees and during interviews, few would stop by to listen in to my explanations of the installation. People also stopped to wonder what else there was.
A bonus effect was that people reported that it:
Gave a sense of Safety -"From a female point of view...so you are not walking in the dark, it would be unsafe if no one was here...It makes it seem like something is happening... adds liveliness"
Was like a companion - "I would be friends with it, its a lonely ghost...I would just stand in front of the sensor and it would just stare at me forever"
Research Question 3: How does audience behaviour differ towards the installation when coming to the area expecting to interact with it (e.g. an event) vs unintentionally interacting with it (part of urban life)?
Unfortunately, my attempt to promote the event was not very effective, due to challenges in affixing posters around the campus, I only have preliminary results for the difference in audience behaviours between the two scenarios.
Unintentional:
Often stopped late and went back to interact or keep walking while turning their heads
See interaction triggered by others in front of them which pigued curiosity
Questions they asked focused on whys and what the ghost is doing
Expecting:
Tended to linger longer and watched people
Came to take in the vibe
Questions they asked focused on what else they can do to trigger effects
Learnings
Conducting a Solo Research Project
There was a lot more freedom than I had expected- no particular methods or topics were set which meant really I had to plan everything. The guidance received was helpful, but nothing really dictated what I should do or even what to try to find out.
Although having a research plan is common sense, I was surprised at the amount of effort it involved and the importance of really scoping it through formulating research questions - it provides the direction
I learnt about "creating new knowledge" - the lack of guidelines and practices on designing with Hybrid-resolution facades meant I had to find out and think about it by myself on how to obtain that knowledge
Having a one on one supervisor was new to me where through this project I learnt to be comfortable asking about all the doubts and confusions I had as well as making sure I plan each meeting to not waste his time
I also learnt that documenting the research or design process is very helpful, both for others and for the future you. This includes taking pictures and videos of the work to be showcased to others or for you to remember certain decisions.
Conducting interviews in the wild
Conducting the interviews by walking up to random strangers got me out of my comfort zone - it provided me a chance to practice striking up conversations with strangers
At the start of the semester I had a lot of fears around being rejected from the interview requests, but it helped to remember that the worse that can happen is they say no. No big deal. Near the end, I could easily just walk up and say, “Hi can we have a chat for two minutes about that?” and not fear rejection.
Asking for voice recordings was also something I was afraid everyone would decline. But later on, due to the need to remember what people said, I asked and found that most people will say ok and it is ok if they decline.
Managing time commitments
With full time work, hobbies and other commitments, I learnt that managing time for this research project came down to two things: Priorities and Balance
Priorities - knowing that this is just one semester, I could figure out commitments that I am willing to pause for a while to focus on the priorities.
Balance - meant that of those commitments I made sure to keep the important ones as well as some leisure ones to keep me motivated.