Brilla AI: AI Contestant for the National Science and Maths Quiz

According to UNESCO, only 65% of primary school teachers in Sub-Saharan Africa possessed the necessary minimum qualifications [19]. Moreover, the average student-teacher ratio at the primary education level in Sub-Saharan Africa stood at 38:1 in 2019, a figure significantly higher than the ratio of 13.5:1 observed in Europe [6]. Consequently, the region requires an additional 15 million teachers by 2030 to meet education objectives, a formidable and costly challenge [20]. The scarcity of adequately qualified educators in Africa undermines the provision of effective learning support for students. Introducing an Artificial Intelligence (AI) teaching assistant for educators holds promise in augmenting the efforts of the limited teaching workforce, facilitating tasks such as one-on-one tutoring and answering student queries. However, the absence of a robust benchmark tailored to real-world scenarios and the African context poses a significant obstacle to evaluating the effectiveness of such AI solutions.

Motivated by this need, Boateng et al. proposed a grand challenge in education — NSMQ AI Grand Challenge — which is a robust, real-world challenge in education for such an AI: “Build an AI to compete in Ghana’s National Science and Maths Quiz Ghana (NSMQ) competition and win — performing better than the best contestants in all rounds and stages of the competition” [2]. In that work, they detailed the motivation for the challenge and key technical challenges that must be addressed for an AI to win the NSMQ. We created an open-source project ¹¹1Brilla AI Open-Source Project: https://github.com/nsmq-ai/nsmqai to conquer this grand challenge and built Brilla AI as the first key output which extends our prior work [3]. Brilla AI is an AI contestant that we deployed to unofficially compete remotely and live in the Riddles round of the 2023 NSMQ Grand Finale, the first of its kind in the 30-year history of the competition.

The NSMQ is an exciting, annual live science and mathematics quiz competition for senior secondary school students in Ghana in which 3 teams of 2 students compete by answering questions across biology, chemistry, physics, and math in 5 rounds over 5 progressive stages until a winning team is crowned for that year [13]. The competition has been run for 30 years and poses interesting technical challenges across speech-to-text, text-to-speech, question-answering, and human-computer interaction. Given the complexity of the challenge, we decided to start with one round, round 5 — Riddles. This round, the final one, is arguably the most exciting as the competition’s winner is generally determined by the performance in the round. In the Riddles round, students answer riddles across Biology, Chemistry, Physics, and Mathematics. Three (3) or more clues are read to the teams that compete against each other to be first to provide an answer (usually a word or a phrase) by ringing their bell. The clues start vague and get more specific. To make it more exciting and encourage educated risk-taking, answering on the 1st clue fetches 5 points, on the 2nd clue — 4 points, and on the 3rd or any clue thereafter, 3 points. There are 4 riddles for each contest with each riddle focusing on one of the 4 subjects. Speed and accuracy are key to winning the Riddles round. An example riddle with clues and the answer is as follows (see a live example here: ²²2Video Example of Riddle: https://www.youtube.com/watch?v=kdaxoFjiYJg. Question: (1) I am a property of a periodic propagating disturbance. (2) Therefore, I am a property of a wave. (3) I describe a relationship that can exist between particle displacement and wave propagation direction in a mechanical wave. (4) I am only applicable to waves for which displacement is perpendicular to the direction of wave propagation. (5) I am that property of an electromagnetic wave which is demonstrated using a polaroid film. Who am I? Answer: Polarization.

Here are some of the key technical challenges an AI system will need to address. How can it accurately provide real-time transcripts of Ghanaian-accented riddle questions read in English? Speech-to-text systems tend to be trained using data from the West and generally do not work well for African-accented speech [14]. How can it infer the start and end of each riddle and extract only the clues while discarding all statements (e.g., the quiz mistress pausing the reading of the clues after the ring of a bell to listen to an answer from a school, instructions that are read at the start of the round, etc.)? How does the AI know the optimal time to attempt to provide an answer optimizing for speed and accuracy? Providing an answer too early might result in a wrong answer given that earlier clues tend to be vague. Waiting too long could result in an answer being provided by one of the contesting teams. How can the AI say its answer with a Ghanaian accent? Similar to STT systems, several text-to-speech systems tend to be trained using data from the West and generally do not speak with African accents. How do we ensure all these work seamlessly in real time without any noticeable latency?

Similar work on grand challenges that entail question answering include the DeepQA project in which Watson won Jeopardy! in 2011 [21], and the International Math Olympiad (IMO) Grand Challenge [7] which has an education focus. The Brilla AI project has some similarities with Jeopardy! given they are both live quiz shows, that entail question answering. It differs, given its science education focus making it arguably more challenging, and provides unique sets of technical challenges discussed previously and also by Boateng et al. [2]. Despite the similarity with the IMO Grand Challenge as both focus on STEM education, it differs by having a scope of both science and math education, and consists of a live competition (IMO is only written). The most important way Brilla AI differs from these 2 challenges though is the African context focus, a rarity in the literature for grand challenges which makes this work the first of its kind.

Our contribution is the first end-to-end, real-time AI system deployed as an unofficial, AI contestant together with 3 human competing teams for the 2023 NSMQ Grand Finale (the first of its kind in the 30-year history of the competition) that (1) transcribes Ghanaian-accented speech of scientific riddles, (2) extracts relevant portions of the riddles (clues) by inferring the start and end of each riddle and segmenting the clues, (3) provides an answer to the riddle, and (4) then says it with a Ghanaian accent.

The task of question answering has two primary paradigms: Extractive QA and Generative QA. Extractive QA involves machine reading, wherein a segment of text within a larger body (referred to as the context) is selected to directly answer a question. The context may be provided directly, as exemplified by the task outlined in the SQuAD dataset. Alternatively, relevant passages can be retrieved from diverse documents through a process known as retrieval. Chen et al. employed this approach in their DrQA system, utilizing TF-IDF for retrieval and LSTM for answer extraction [4]. Presently, BERT-based models are the current state-of-the-art for reader implementation [5]. In contrast, Generative QA employs generative models such as T5 [18] to produce answers given a question as input. These models can generate responses based on various contexts provided alongside the question, a technique termed retrieval augmented generation (RAG) [11]. We use both extractive QA and generative QA in this work and leave RAG for future work.

Brilla AI is currently available as a web app (built with Streamlit) (Fig 1) that livestreams the Riddles round of the contest, and runs 4 machine-learning (ML) systems in real-time via FastAPI together running as an AI server on Google Colab: (1) speech-to-text (using Whisper [16]) which transcribes Ghanaian-accented speech of scientific riddles (2) question extraction (using BERT [5]) which extracts relevant portions of the riddles (clues) by inferring the start and end of each riddle and segmenting the clues (3) question answering (using Mistral [8]) which provides an answer to the riddle and (4) text-to-speech (using VITS [10]) which says the answer with a Ghanaian accent (Figure 2). We used Streamlit due to its applicability to ML and Data science use cases, and quick development cycles.

Here is an overview of the sequential processing and communication between components in the Brilla AI system. The web app extracts audio from the video and chunks it in 5 seconds using FFmpeg to allow for low-latency processing, and then sends and receives relevant data from the AI server sequentially. First, the audio chunk is sent to the STT API for processing which extracts the transcript, and passes it to the QE service which detects the riddle start and extracts the clues. The clues along with information about whether it is a new riddle are sent to the web app. It displays the transcript and clues and then sends the clues to the QA API which determines whether to provide an answer (details described later). If an answer is generated, it is returned to the web app which then displays it and sends it to the TTS API. It generates an audio of the answer and sends it to the web app which then plays it. This current sequential operation is a bottleneck within the system which caused some latency issues (discussed later). We are exploring parallelizing operations to remove such bottlenecks from the system.

The web app consists of 2 modes: demo and live ³³3Brilla AI Demos: https://youtube.com/playlist?list=PLl-Yq3F_E8VJWFLpsnizorhIz8N-OqUqw&si=ddrl7u0DBZcbNt7H. The demo mode showcases the AI functionality outside of the quiz season. Functionality has been added for users to experience the AI using curated text, audio, and video content but also allows them to provide real-time content to test the AI. For example, in the demo mode, the user can provide real-time audio input on the Speech-to-Text page. The web app processes this audio and sends an API call to the Speech-to-Text API server. The server returns the real-time transcript displayed in the web app to the user. Also, users can provide a YouTube link of any NSMQ quiz video to the web app and see the AI process the content in real time and compete for that video. To demonstrate the AI during active quiz season, the web app utilizes a live mode. In this mode, the user can supply a live stream link from YouTube to the web app and see the AI compete in real-time as well.

We curated a dataset of NSMQ contests from 2012-2022 containing videos of the contest and corresponding metadata, text form of riddles questions, and open-source science textbooks. First, we created a Google sheet with information about each contest such as the contest date, names of the competing schools, and the marks they acquired at the end of each contest. We additionally provided information for the riddles round such as which school answered each riddle and at which clue, which was used to calculate the points obtained by the best human contestants for each riddle. We then used that information to automatically extract video clips of the contests. Next, we manually generated annotations for the riddle round containing information about the start and end timestamps of each riddle and clue. These were then used to automatically extract the video and audio clip segments of riddles.

For the text of riddles, we purchased the digital version of the questions, parsed them, and manually reformatted them into CSVs for usage. Each CSV file had columns “Clue 1” to “Clue 9” for all the clues in per riddle, “Answer” for the ground truth answer, and “Answer 1” to “Answer 4” for alternative ground truth answers, if any, (e.g., hydrogen and h2 as alternative answers). We included additional columns to track information like subject, contest number, and year of the contest. We applied the following preprocessing steps and created riddle-answer mappings: converted all clue texts and answers to lowercase, removed punctuation, fixed whitespace, and removed articles (e.g., “the”, “a” “an”).

We extracted HTML files from the following open-source Science textbooks on OpenStax — High School Physics, College Physics, College Biology, College Chemistry, and College Algebra — segmenting content into chapters, sections, paragraphs, and passages. We have completed the sheet with information about contests (with some contest information missing for some years as they could not be found online). For information about the performance of contestants in the riddle round, we only have complete information for the years 2019 and 2020. Though we have complete data for the text version of the riddle questions, we only used riddles for these 2 years for evaluation so we could compare the performance of our system with human performances. Consequently, we used a total of 316 riddles (156 in 2019 and 160 in 2020). Work is ongoing to annotate the timestamps of each of the clues for all riddles and also to completely parse the textbooks. Hence, we did not have a complete set of video and audio clips of riddles for this work.

In its debut in the NSMQ Grand Finale in October 2023, Brilla AI answered one of the 4 riddles ahead of the 3 human contesting teams, unofficially placing second (tied) and achieved a 25% EM accuracy ⁸⁸8Brilla AI Debut: https://youtu.be/2AUpiVB6zA4 [9]. For the first riddle, the QE component could not detect the start of the riddle as STT wrongly transcribed “first riddle” as “test riddle”. Consequently, no clues were extracted, QA received no data, and thus could not answer the riddle (the answer shown was from a previous test attempt). For the second and third riddles, the start and ends were detected and the clues were extracted but QA attempted too early (perhaps due to the confidence threshold being quite low) and got the answer wrong. For the fourth and final riddle, QE detected the start of the riddle, could not extract clue one, but extracted the second clue and third clue, and then QA generated an answer, resulting in Brilla AI answering correctly before one of the students answered correctly also (which he did after all the clues). Overall, we recorded a success rate of 25% unofficially placing second (tied) as the best-performing team answered 2 riddles correctly (excluding the one the AI answered), one team answered only one and the third answered none. This result is an important milestone, especially given it was our first live deployment and there were a lot of real-world challenges like noise, inaccurate transcripts, undetected clues, and irrelevant data sent along to QA. Our automatic transcripts were not always accurate and some clues were not detected which all compounded and posed challenges for QA. Our real-time transcription began to lag the live stream of the contest because of the sequential processing of our pipeline, which resulted in our AI appearing to have provided an answer after all clues, even though it did so only after receiving the 3rd clue. This issue could have resulted in a late answer.

One key challenge we have is curating the NSQM data. At the end of each contest, the competing schools and their respective scores are shared via images or PDFs on social media and blogs. Sometimes finding these materials is difficult as they may be scattered on the web or may not have been posted by the media channel in charge. Video recording errors due to power surges and Internet connection problems also make getting accurate data difficult.

In the web app, the sequential approach of calling APIs was a bottleneck which caused some latency issues for the live deployment. We plan to explore parallelizing operations to remove such bottlenecks from the system. Also, we plan to deploy the web app and ML models on Google Cloud Platform to ensure that both the front and backends of Brilla AI are running on cloud systems to increase the security and stability of the web app.

We plan to fine-tune our STT model since we did not do so in this work due to a lack of complete NSMQ annotated data, which could improve performance. To improve the QA performance, we will explore using RAG after completing the curation and processing of the science textbooks. We will also explore using reinforcement learning to improve our confidence modeling approach. For TTS, we will curate a dataset that includes recordings of scientific and mathematical expressions, as well as single-word answers, to address its shortcomings.

In this work, we built Brilla AI, an AI contestant that we deployed to unofficially compete remotely and live in the Riddles round of the 2023 NSMQ Grand Finale, the first of its kind in the 30-year history of the competition. Our AI answered one of the 4 riddles ahead of the 3 human contesting teams, unofficially placing second (tied). Our next step is to improve the performance of our system ahead of 2024 NSMQ during which we plan to unofficially compete in both Round 5 (Riddles) and Round 4 (True or False) for all 5 stages of the competition. Aside from being an interesting intellectual challenge, we are building an AI that addresses the unique context of Africans — transcribes speech with a Ghanaian accent, provides answers to scientific questions drawn from a Ghanaian quiz, and says answers with a Ghanaian accent — that could be integrated into an education tool. Imagine a student in a rural part of Ghana calling a toll-free number with a feature phone, asking this AI numerous science questions, and the AI understands her even with her Ghanaian accent and then provides explanations using local context in a Ghanaian accent. That vision is our long-term goal and would radically transform learning support for students across Africa, enabling millions of young people to have one-on-one interactions and support even with limited access to teachers, computers, or even smartphones leading to the democratizing of science education across Africa!

We are grateful to all the volunteers that contributed to this project. We are also grateful to Isaac Sesi for donating his voice which we used to train the TTS model for the voice of our AI, and our project advisors for their feedback and support throughout this work: Professor Elsie Effah Kaufmann, the NSMQ Quiz Mistress, and Timothy Kotin, a member of the 2006 NSMQ winning team.