Methodology

To evaluate the usability of the Korean Language Game, we conducted a mixed-methods usability study involving a total of 6 participants. The participants primarily consisted of beginner Korean learners, as well as individuals with informal exposure to the language through media such as K-dramas. This demographic was selected to reflect the platform’s intended user base of novice learners who require intuitive navigation and supportive learning tools.

The usability study consisted of three main phases: pre-test, task-based testing, and post-test evaluation. During the pre-test phase, participants were asked a set of background questions to assess their prior experience with Korean, their learning goals, and their familiarity with other language-learning applications. These questions provided important context for interpreting user behavior and expectations during the testing process.

Participants were then asked to complete a series of seven representative tasks designed to simulate real user interactions with the platform. These tasks included creating an account, locating developer information, studying vocabulary through flashcards, playing vocabulary games, checking progress metrics, submitting feedback, and logging out. The sessions were conducted both remotely and in person and were recorded for analysis, allowing us to observe user interactions and identify usability issues.

Following task completion, participants responded to a set of post-test questions to evaluate their experience with navigation, task difficulty, and overall usability. These responses provided insight into both the most problematic and most enjoyable aspects of the platform. Additionally, task completion times were recorded for each participant, enabling comparison across tasks and identification of areas requiring improvement. By combining both quantitative performance data and qualitative feedback, this methodology allowed us to develop a comprehensive understanding of the platform’s usability strengths and weaknesses.

Findings

Participant Background (Pre-Test)

None of the six participants were currently enrolled in a Korean class, though one (P3) expressed interest in joining. Prior study experience ranged from a few months of informal self-study (P1, P2, P3) to one to two years of practice (P4, P5), to roughly three years studied at a younger age (P6). Motivations included understanding K-dramas (P1, P2), conversational travel readiness (P3), self-assessment (P4), spelling practice (P5), and casual language review (P6). Only P5 had previously used a comparable language-learning app (Duolingo), noting that its beginner levels felt too easy. P2 also mentioned Duolingo but disliked its pushiness. The remaining participants had no prior app experience, giving them a neutral baseline for evaluation.

Usability Issues Identified

Issue 1: Vocabulary Game Input Problems (Task 4): Task 4, which asked participants to play the vocabulary game, was the hardest task for all six participants. As shown in Figure 1, it had the highest average completion time at 80.5 seconds. For comparison, Task 1 averaged 48.3 seconds, meaning Task 4 took more than twice as long on average. Figure 2 shows an even bigger problem: individual completion times ranged from just 20 seconds for P3 all the way up to 136 seconds for P6. That is nearly a seven times difference, and it was not because some participants knew more Korean than others. The real problem was the game's design. The game required players to type answers using Korean characters, but the website did not provide a Korean keyboard. The game was also case sensitive, which meant a correct answer could still be marked wrong if it was typed with the wrong capitalization. Instead of actually learning vocabulary, participants spent most of their time trying to figure out how to type in Korean. P5 said: "The game's answers had some open-endedness and the game is case sensitive." P6 added: "The game experience was fun but I need an option to turn on the Korean keyboard."

Issue 2: Study Pages Were Hard to Navigate (Task 3): Task 3 asked participants to study vocabulary using the flashcard feature. This task had the second highest average completion time at 41.8 seconds, and P5 took 97 seconds on this task alone. The problem was not the flashcard content itself. The issue was that the flashcards and the vocabulary games were located on two completely separate pages. If a user wanted to switch between the two study modes, they had to click away from the current page and go searching for the other one. Participant 4 put it plainly: "It wasn't the easiest. The flashcards and games are on a different page." There were also no icons or labels to help users figure out where each study mode was before clicking through, which made it even more frustrating to navigate.

Issue 3: Most of the Website Was Written Only in Korean: Two out of six participants, P3 and P6, both brought up the same issue on their own without being asked: large sections of the platform were written entirely in Korean with no English translation provided. This is a serious problem because the app is designed for beginner Korean learners, meaning many users may not be able to read Korean at all yet. P6 said: "Earlier sections need an English translation for those who don't know any Korean." P3 agreed, saying: "Everything was in Korean so it might be better if it was in English [for early sections]." The fact that both participants brought this up on their own, without any prompting, shows it was a very noticeable problem. It does not make sense for users to need to already know Korean just to get around an app that is supposed to teach them Korean.

Issue 4: The Feedback Form Crashed or Did Not Work (Task 6): Task 6 asked participants to submit feedback using the website's feedback form. This was the only task in the entire study where some participants could not finish at all. As shown in Figure 3, only 4 out of 6 participants (67%) were able to complete this task. P4's submission caused the entire website to crash. P6 received a "Not Found" error after clicking submit. Even the participants who did submit feedback successfully only had a single blank text box to fill out, with no categories or guidance on what to include. P4 said: "[Feedback] did crash the whole website. I want specific categories for feedback." This shows that the feedback page had both a technical problem and a design problem happening at the same time.

Issue 5: The Sign-Up Button Was Hard to Find (Task 1): Task 1 asked participants to create an account on the platform. On average, this task took 48.3 seconds, the third highest of all seven tasks. The main reason it was difficult was that the landing page only showed a "Login" button. There was no visible "Register" or "Sign Up" button for new users. P4 pointed this out directly: "There isn't a register button on the page, only login." P4 had to guess that the login page would also lead to account creation, which is not something most new users would expect. As shown in Figure 4, total session times ranged from 96 seconds for P3 to 341 seconds for P5. That is a 3.5 times difference between the fastest and slowest participants, and small problems like this missing button added up to make the experience much harder for some users than others.

Participant Quotes Highlighting Frustration and Confusion

Analysis

Lookiing at Figure 2, the line graph of Task 4 completion times highlights the problem. P5 took 123 seconds and P6 took 136 seconds, while P3 finished the same task in just 20 seconds. That is more than a six times difference between the fastest and slowest participants. This gap was not caused by differences in Korean knowledge. Both P5 and P6 completed other tasks at a normal speed. The issue was the game itself since it required Korean character input but did not include a keyboard, and it marked answers wrong if the capitalization was off. Four out of six participants took over 75 seconds on this single task, which shows how big the problem was. Because of this, our team recommends adding a virtual Korean keyboard directly inside the game, as well as updating the game to accept answers regardless of capitalization. These two changes would remove the biggest source of difficulty found in the entire study.

As shown in Figure 1, Task 3 (studying flashcards) had an averagThat is more than a six times difference between the fastest and slowest pe completion time of 41.8 seconds. That is 2.4 times longer than Task 2, which averaged just 17.5 seconds and only required users to look up developer information. The gap is surprising because studying should be one of the easiest things to do on a learning app. The reason it was slow was that flashcards and vocabulary games were stored on different pages. Users expected to find them together, but instead had to navigate away to switch between study modes. P4 said there was "an unnecessary click for each unit," and P5 spent 97 seconds just trying to access the flashcards. This is why our team redesigned the Study Options page to put all study modes, including flashcards, vocabulary games, grammar games, and listening games, onto a single screen with a clear icon for each option.

The participant feedback points to a problem that the timing data alone does not fully show. Two out of six participants, which is 33%, brought up the same issue on their own without any prompting: too much of the website was written only in Korean. Since the app is meant for people who are just starting to learn the language, many users cannot read Korean yet. This puts them in a tough spot where they cannot use the app until they already know some of what the app is supposed to teach them. Participant 3 said, "Everything was in Korean so it might be better if it was in English," and Participant 6 agreed, saying, "Earlier sections need an English translation for those who don't know any Korean." To fix this, our redesigned Units page includes English translations on all unit titles, navigation labels, and instructions so that beginners can use the app from the very start.

Figure 3 shows the most serious problem we found in the study. Task 6, which involved submitting feedback, was the only task where some participants could not finish at all. Two out of six people (33%) ran into major errors which was that P4's form submission crashed the entire website, and P6 got a "Not Found" error message after clicking submit. The four participants who did manage to submit feedback were only given a blank text box with no instructions or categories to guide them. P4 said, "[Feedback] did crash the whole website. I want specific categories for feedback." This shows that the feedback page failed on both the technical side and the design side at the same time. Our redesigned feedback form fixes the broken submission so the page no longer crashes, and it replaces the empty text box with a dropdown menu that lets users choose a feedback category such as Bug Report, Feature Suggestion, or General Inquiry. This way, users can quickly pick an option instead of having to figure out what to write on their own.

As seen in Figure 4, total session times ranged from 96 seconds for P3 to 341 seconds for P5. That is a 3.5 times difference between the two, even though both participants came in with similar backgrounds and Korean experience. This large gap shows that the problems found in this study did not affect everyone equally. Some users hit multiple issues in a row, and those problems piled up. The missing "Register" button on the landing page is one example since it was a small design oversight that added real confusion to Task 1 for certain participants. When you look at all five problems together, including the game input issues, the confusing page navigation, the Korean-only text, the broken feedback form, and the missing sign-up button, a clear pattern comes through. These are all fixable problems on top of what is otherwise a creative and fun platform. Participants genuinely enjoyed the gamified learning format, with P1 calling it "really fun" and "more fun than just memorizing words." That means the core idea is strong. Our design changes focus on clearing away the specific problems that were getting in the way of that experience.