Investigating The Effects of Narrative Storytelling As Gamification Element Through LARPing
Questionnaire
Literature Review
Quantitative research on investigating how gamification affects user experience.
January 2023 - 2 month sprint
3 person team
​
Participant Sampling and Recruitment
Designing Experiments
Facilitating workshops
SPSS Data Analysis
Validating Experimental Results
Analytical Thinking
Written Communication Skills
Verbal Communication & Presentation Skills
My Role
As the lead user researcher, my role encompassed:
-
Ideating the research problem
-
Conducting an in-depth literature review
-
Carefully selecting the sample size while recruiting participants through Doodle Poll
-
Meticulously planning the research and experimental design
-
Skillfully conducting the experiments
-
Validating the experimental results through a comprehensive literature review, ensuring robust and reliable conclusions
-
Crafting a concise and clear report that effectively guided readers through the content demonstrating my strong written communication skills
-
Presenting my findings to both stakeholders and peers, showcasing my verbal communication abilities
The Motivation
Gamification is commonly used in mobile apps, websites, and e-learning platforms to encourage users to complete tasks, explore features, and achieve specific goals. By incorporating elements like progress bars, achievements, rewards, and even narratives, users are more likely to stay engaged and motivated to continue using the platform or app.
The motivation for this research stems from the crucial link between gamification's focus on user experience and meticulous planning, especially in complex research scenarios. As a UX researcher, I wanted to understand the impact of narrative storytelling gamification versus non-narrative storytelling gamification on participants' self-perception of ability. I ran a controlled experiment involving 2 groups tasked with rebranding a beverage, one with Live Action Role Play (LARP) activity and the other without, in order to explore whether narrative storytelling gamification influences participants' self-efficacy in completing the given tasks compared to non-narrative storytelling gamification. The ultimate goal is to shed light on the role of storytelling in gamification and its potential implications for user engagement and performance. This investigation contributed valuable insights into the evolving field of gamification and user-centred design.
The Challenge
My research involved studying various gamification models and game design elements used in non-game settings, and an extensive literature review highlighted the importance of contextually relevant gamified system design for its users. Research shows that gamified experiments do not have the same effect on their users if the gamified system (the setting and activity) is not relevant to the users. Hence, LARPers were the ideal participants for this particular study since I investigated the effects of narrative storytelling gamification through LARPing activities.
​
Furthermore, to investigate the effects of narrative storytelling gamification, I had to isolate the narrative element from other gamification elements, as previous studies often yielded inconclusive results due to mixed elements.
​
Time constraints prevented a mixed methods study, cross-sectional or longitudinal study. Remarkably, no previous research has compared the effects of narrative storytelling gamification VS non-narrative storytelling gamification, which added complexity to the study design.
Based on the research findings from previous empirical studies, too much immersion would subconsciously influence the participants of the narrative storytelling group, which would enhance their self-efficacy scores. After all, the goal of this study was to investigate participants' self-efficacy scores by investigating the effects of narrative storytelling gamification through LARPing. To get rid of this confound, I had to make sure the prompt for the narrative storytelling group (experimental group) was 'just immersive enough' but not 'too immersive', so I could make fair comparisons to the vanilla prompt for the non-narrative storytelling group (control group). I also had to include the same gamified tasks in both prompts for a fair comparison, with slight narrative changes for the experimental group. This opened up the risk of introducing/mixing other elements in my study, which directly conflicted with the rule of isolating gamified elements to further study its effect without any potential confounds interfering.
​
How can you effectively design a gamified system that incorporates the narrative element without overpowering its effect and creating an unfair advantage? The results from data analysis proved the findings to be inconclusive. But that does not mean the experiment failed! On the contrary, this gave me enough insights to understand what improvements were needed if I want to further investigate the effects of narrative storytelling gamification. No research is perfect.
​​
Research & Planning
Research Question
​
"How does incorporating narrative storytelling elements, such as a narrative or characters, through LARPing, impact participant self-efficacy?"
​
Literature Review
​
Due to the uniqueness of the research statement, I had to carefully study and adapt a series of gamification models in my gamification design to understand the effects of narrative elements on participant self-efficacy through LARPing. Instead of going through the literature review in detail, I am adding a brief overview of the research background.
The Process
Quantitative Methodology Justification
​
The goal of this experiment was to understand if isolating narrative storytelling as a gamification element would yield a better self-efficacy score from the participants, in comparison to the self-efficacy scores from the participants of the non-narrative storytelling gamification group. For this reason, the participants were split into 2 groups with 2 different conditions. They had their self-efficacy scores measured before and after they completed the same set of challenged-based tasks (experimental group) with narrative scenarios and characters and the challenged-based tasks (control group) without any narrative setting.
​
Due to the comparative nature of this research, I employed a quantitative between-subjects design to compare the effects of narrative gamification with non-narrative gamification, utilising LARP as the medium for self-efficacy in challenging yet achievable tasks, fostering collaboration and cooperation.
Recruitment & Sample Selection
Previous literature proved how important the target group (who is affected by this gamification design) is for the goal of the gamification. The gamification goal was to study the effect of narrative storytelling gamification, for which LARP and therefore people who had LARPing experience was ideal.
​
But our recruitment process faced challenges due to departmental restrictions on recruiting participants outside of the University as well as alumni. Initially aiming for LARPers, restrictions led to reopening recruitment to the general student population, resulting in a final sample size of 10 participants split into 2 groups of 5. I used Doodle Poll for recruiting participants.
​
Measures & Materials
​
The data collection for this experiment relied on pre-and post-activity questionnaires completed by participants, using Gilad Chen's The New General Self-Efficacy Scale (NGSES) to measure self-efficacy. The questionnaire was recreated in Qualtrics to ensure an accurate measurement of self-efficacy as the targeted construct for the experiment. The NGSES demonstrated good psychometric qualities and internal consistency, proving to be a valid testing tool. Participants were provided with materials such as white paper, colourful pens, and markers to complete the tasks, and the artefacts produced during the sessions were not included in the research analysis.
Gilad Chen’s New General Self-Efficacy Scale (NGSES) [2001]
Gamification Design Justification
​
The experiment comprised 2 groups: the non-narrative storytelling group and the narrative storytelling group, with each group's prompt designed based on the integrated gamification models explored in the literature review. The non-narrative group received a set of tasks focusing on goal-oriented, challenging but achievable tasks promoting collaboration and self-expression. Similarly, the narrative group received a short scenario and created characters to role-play, completing tasks based on the narrative. We designed the tasks for both groups ensuring contextually relevant and personalized gamification without relying on any single framework. The experiment aimed to maintain consistency between groups while avoiding potential experimental confounds as much as we could, adhering to the cautionary empirical studies on mixing gamification elements.
Non-Narrative Storytelling Prompt
Narrative Storytelling Prompt
Experimental Observations
​
During the experiment, several observations had been made between the narrative storytelling and non-narrative storytelling groups. The narrative storytelling group was notably quieter, with less vocal communication between participants. In contrast, the non-narrative storytelling group was much more active in terms of communication, with everyone participating in discussions and sharing their ideas. I did not conduct any interviews due to time constraints, but from my observation, it looked like the participants of the non-narrative storytelling group had existing rapport with each other. Perhaps a few of them signed up for the same time slot together and so they were assigned to the same group together. The participants were randomly assigned to either the experimental or control group so they had no way of knowing which group they would end up in beforehand.
The narrative storytelling group completed the task much earlier than the non-narrative storytelling group, taking only 30 minutes to finish. It is worth noting that both groups showed engagement and motivation in completing the given tasks, albeit in different ways. I observed that the non-narrative storytelling group's participants were strangers to each other. Perhaps that is why they were focused on the tasks and the time limit, rather than engaging in friendly banter. Hence, they finished earlier.
​
Since neither group had prior experience with LARPing, the non-narrative storytelling group devoted all their time coming up with exciting new designs for their rebranding tasks. While the narrative storytelling group struggled to role-play and incorporate their created fictional character into the narrative of the story.
Overall, these observations suggest that the narrative storytelling and non-narrative storytelling groups had different dynamics and approaches to completing the tasks. These differences could potentially impact the participants' self-efficacy, as the narrative storytelling group may have felt more confident and efficient in their task completion, while the non-narrative storytelling group may have felt a greater sense of ownership and collaboration in their output. One way of mitigating the lack of immersion in the experimental group could be incorporating a narrated voice for the NPC.
The Artefacts
​
These artefacts were collected after each group's session. Due to time constraints, we didn't conduct any interviews or collect any additional data other than the 8-item NGSES questionnaire. No conclusions were drawn about the research from these artefacts, this is not a source of data for the study. There were only bi-products which were created from the two sessions. The non-narrative storytelling group used several pages to create and refine their rebranded design. Whereas, the narrative storytelling group used a single sheet of paper and completed the tasks in 30 minutes.
Non-narrative storytelling group's artefacts
Narrative storytelling group's artefact
Data Analysis & Results
Experimental Null Hypothesis
​
The population means from the narrative storytelling group and the non-narrative storytelling group are equal.
​
Data Analysis
​
Each of the 8 questions in the Likert scale questionnaire had scores attached to them (e.g. 1 = strongly disagree; 2 = disagree; 3 = neither agree nor disagree; 4 = agree; 5 = strongly agree). The total self-efficacy of each participant is calculated and then the average self-efficacy of each participant was calculated by dividing each participant’s score by 8, as instructed by the NGSES. The table consists of all participants’ data (n = 10) both before and after their one-hour activity.
​
The data was analysed using SPSS version 28. The data from the questionnaire was cleaned up and analysis was run on three variables - Initial Self-Efficacy (before the activity), Final Self-Efficacy (after the activity) and Self-Efficacy Difference for each participant.
​
An Independent Samples T-test was conducted to compare the mean scores of both groups. Our findings proved the null hypothesis to be correct as the result of the T-test was inconclusive (p>0.05). P is the value of statistical significance. Differences happen as a result of chance. When we believe something is statistically significant, we believe the difference is larger than can reasonably be explained as a chance occurrence. The value of p for this experiment supports the null hypothesis, meaning the population means from the narrative storytelling group and the non-narrative storytelling group are equal.
​
Summary statistics of the experiment
Boxplot showing initial total self-efficacy score of each group.
Boxplot showing final total self-efficacy score of each group.
Boxplot showing total self-efficacy score difference of each group.
There were several outliers within the data. Upon carefully checking the data again, it can be confirmed that these values are not errors. Outliers make data more variable, which reduces the statistical power of data. So, removing the outliers can make data statistically significant. The outliers have been kept in order to not force the findings to be statistically significant. The inconclusive results of this experiment were expected and unexpected for several different reasons. I will justify why the inconclusive findings and more importantly, the research itself is still very useful.
Justification of the findings
The Conflicts In Literature Review
​
This study investigated the effects of isolating Narration as a gamification element while keeping the designs of both groups closely relatable to avoid potential confounds. At the same time, prior literature highlights the limitations of relying on a one-size-fits-all gamification approach, resulting in inconclusive findings. In my controlled experiment, I tried to eliminate internal validity discrepancies for more accurate analysis by adopting this approach to gamification and also, mixing gamification elements by creating both prompts as comparable as possible, the design is personalised to fit the goal of the study. The literature was conflicting and I tried my best to balance both.
​
The inconclusive findings might have been due to the narrative prompt not being immersive enough to effectively increase participants' self-efficacy through the gamified tasks. I avoided making the narrative gamification more immersive to prevent potential confounds and force statistically significant results. Additionally, distractions caused by narrative, as mentioned in past literature, might have impacted the study outcomes.
​
Participant Sampling Limitation
​
The study was not contextually relevant, as I needed to recruit LARPers as the target audience. The general student body and their inexperience with creating characters based on narrative and LARPing could have affected the results, leading to inconclusiveness.
The aforementioned reasons collectively contributed to the inconclusive findings, and based on the experimental design, I found no grounds to dispute them. I believe the research was conducted meticulously, with well-designed experiments supported by robust literature. While prioritizing internal validity, external validity was not maintained, which is a common trade-off in research. It is important to acknowledge that no study can be flawless, and this research also critiqued the generalisability of much existing research and the statistical significance of those findings.
Conclusion
The Impact
​
Gamification is a UX strategy that leverages game-like elements to influence user behaviour positively and create a more satisfying experience. The impact of gamification on user experience in the business domain is about creating engaging, enjoyable, and rewarding interactions with their users. By understanding their target audience and using gamification strategically, businesses can leverage this approach to drive user engagement, build brand loyalty, and achieve their business objectives.
My study is impactful because it presents how essential it is to design gamification elements thoughtfully, considering the context, user needs, and overall objectives. Poorly implemented gamification can lead to disengagement, frustration, or even user alienation. Therefore, a user-centred approach is crucial when incorporating gamification to ensure it aligns with users' preferences, motivations, and goals, ultimately contributing to a positive and meaningful user experience.
What I learned
​
Similar to my user research study on Community Engaged Volunteering Platform, I came to realize the unpredictability of research. While I had not anticipated inconclusive findings, upon deeper analysis and considering empirical studies, the results made sense. I gained insights into the significant impact of group dynamics on gamification designs, which I initially underestimated. The small sample size also contributed to the inconclusive results of the experiment. Nevertheless, this study offers valuable insights into the effects of narrative in gamification through Live-Action-Role-Playing.
​
I also learned the importance of carefully considering both internal and external validity when setting up an experiment. Taking the time to design a study with strong internal and external validity ensures more meaningful and far-reaching implications, leading to stronger conclusions based on the results obtained. Therefore, prioritizing a well-structured and thoughtful study design is essential for achieving reliable and impactful outcomes.
What I would do differently
​
I would enhance this study by allowing more time to conduct in-depth interviews with the participants, gaining valuable insights into the effects of narrative storytelling gamification. To achieve this, I would opt for a mixed methods study design, providing a comprehensive exploration of the topic. Additionally, I will ensure to specifically recruit LARPers as my target audience, aligning the chosen medium of investigation with the gamification context.
Having said that, being a user researcher in this study was a meaningful experience that allowed me to expand my skill set. I am excited about the prospect of designing more gamified systems in the future!