Learning Analytics from Spoken Discussion Dialogs in Flipped Classroom

With the popularity of flipped classrooms in practical teaching and learning, much research in the field of education has been devoted to this topic in recent years.

Much previous work aimed to investigate whether the flipped classroom enhances student learning and motivation, often through comparisons with traditional teaching approaches. [Hew and Lo 2018] claimed that the flipped classroom approach in health professions education yields a significant improvement in student learning compared with traditional teaching methods [8]. By measuring students’ performance on the final examination and students’ self-reported satisfaction, [Joseph et al. 2021] found that flipped classroom improves nursing students’ performance and satisfaction in anatomy and physiology compared to a traditional classroom [9]. [Zhao et al. 2021] used questionnaires to demonstrate that students in the flipped class have a better learning performance than students in a traditional classroom [10]. By conducting a self-efficacy survey, [Namaziandost and Çakmak 2020] found that students in a flipped classroom have higher self-efficacy score compared to traditional classroom [11].

The strategy and design of the flipped classroom approach are important for its effectiveness [12, 13]. [Chen and Yen 2021] investigated how to make a better pre-class instruction when using animated demonstrations, in order to enhance student engagement and understanding in the flipped classroom [14]. By using self-efficacy questionnaire and interview questions, [Hsia and Sung 2020] found that the flipped classroom with peer review could significantly enhance students’ intrinsic motivation and strengthen their focus and reflection during activities [15]. [Song and Kapur 2017] compared the “productive failure-based flipped classroom”(PFFC) pedagogical design with traditional flipped classroom pedagogical design and they found that students in PFFC achieved higher scores in solving conceptual questions [16]. [Lin et al. 2021] proposed the “Scaffolding, Questioning, Interflow, Reflection and Comparison” mobile flipped learning approach (SQIRC-based mobile flipped learning approach) and found that the proposed approach improved the students’ learning performance, self-efficacy and learning motivation compared to traditional flipped classroom [17].

Previous work have also studied the impact of different indicators on learning performance in flipped classrooms. The indicators of students’ learning performance are investigated based on students’ learning strategies measured from students’ behaviors in pre-class activities such as online course and online assessment [18, 19]. [Wang 2021] explored indicators of learning outcome in the flipped classroom by analyzing learning management systems log data and questionnaires [20]. [Murillo-Zamorano et al. 2019] used questionnaires to investigate the causal relationships of knowledge, skills, and engagement with students’ satisfaction [21]. [Zheng and Zhang 2020] did analyses on students’ response of survey and found that students’ behaviors such as peer learning and help seeking were positively associated with students’ learning outcomes [22]. [Lin and Hwang 2018] studied how the students’ online feedback on video learning materials and video recordings of classmates’ work was relevant to their performance [23]. Quantitative measures such as learning hours and attendance have also been studied to identify their relevance to learning performance [24].

Although many quantitative researches on flipped classrooms have been conducted, few research paid attention to the in-class discussion dialogs. Spoken discussion dialogs commonly occur in flipped classrooms, which embed rich information related to learning that cannot be provided objectively by conventional data. Therefore, the spoken discussion dialogs in flipped classrooms should not be ignored in learning analytics.

Due to increasing availability of sensors as well as computational resources and algorithms, some studies on analysis of face-to-face discussion were performed. [Kubasova et al. 2019] used verbal and nonverbal features of group discussions for predicting group performance in a cooperative game [25]. [Avci and Oya 2016] used aural and visual cues with their novel classifier to predict the group performance in decision-making tasks [26]. [Murray and Oertel 2018] predicted group performance in task-based interactions by utilizing speech and linguistic features [27]. However, these studies were conducted in the context of gaming or decision-making rather than in an educational context. [Ochoa et al. 2013] used a data set of video, audio and pen strokes to extract features that can discriminate between experts and non-experts in math problem discussions [28]. [Scherer et al. 2012] investigated predictors from audio and writing modalities for the separation and identification of socially dominant leaders and experts within a study group [29]. [Martinez-Maldonado et al. 2013] determined the level of cooperation at an enriched interactive tabletop from verbal and physical information [30]. [Reilly and Schneider 2019] conducted an educational experiment which showed that Coh-metrix could extract learning gain indicators, and they also predicted the quality of collaboration in discussions [31]. [Spikol et al. 2017] estimated the success of small collaborative learning groups in an experimental setting that extracted features from vision, user generated content and learning objects [32]. To the best of our knowledge, there is no prior work that investigates how the learning dynamics captured from student group discussion dialogs correlates with the learning outcomes in the flipped classroom.

This research is conducted in an engineering mathematics flipped classroom [33], which has recently been transformed from the conventional instructional setup. In this course, students watch lecture videos before class and solve in-class exercise problems in groups, aided by the professor and teaching assistants during the class time. The quality of group discussions in such flipped classroom is highly related to student group grade since problem solving discussions take up most of the class time. The remaining, minor portion of class time is for announcements and sharing of good solutions to given problems.

Students enrolled in the class form their own groups of 3-4 members for the semester. Students in the same group sat together in the classroom, which was a laboratory with computers. Fig. 2 shows the setup, where the red dots showed the placement of a speech recorder. We had selected the TASCAM DR-05 recorder because of its relative small size, and hence they could be placed in a non-intrusive position for each student group. Adjacent recorders were placed at least 1.5 meters apart and were positioned to point towards the members in the same student group for sound capture. The sampling rate was set at 44.1 kHZ. We obtained consent from the students through a consent form in order to record their speech during in-class discussions. We also offered the option that students were free to press the button to stop recording at any time during class, but no student did that throughout the whole semester. After the semester’s add-drop deadline for courses, we recorded discussions from the ten groups in class until the end of the semester.

Each recording contains discussion speech as well as broadcasted information such as general class announcements and solutions to problems. Since we are more interested in the portions of the audio that only contain group discussions, an unsupervised classification of lecture and discussion is achieved by using a customized audio processing technique [34]. In designing the classification algorithm, we aim to fully leverage the simultaneous recordings from the devices placed around the classroom. Specifically, lectures have high similarity across various simultaneous recordings. Therefore, after recordings are coarsely synchronized to a common start time by using Fast Fourier Transform (FFT) convolution, we compute the normalized similarity between a given window and temporally proximate window segments in other recordings. Histogram plot automatically categorizes higher similarity windows as lecture and lower ones as discussion. With the proposed method, broadcasted information is eliminated in each of our audio, and only discussion speech is further analyzed in this work.

Presently, we do not have a sufficiently accurate, Chinese-English code-switched automatic speech recognition system that can transcribe the discussion audio recordings. Instead, we use manual transcriptions, aiming for a diarization task of “who spoke what at when”. Fig. 3 shows an example of the transcription. Speech time information, speaker information and discussion content are recorded.

Discussion dialogs are segmented based on two criteria: 1) Speaker changes from one person to another. 2) The occurrence of a long pause of duration greater than a second. The start and end times of each segment are recorded. Transcribed speakers are classified as the professor, teaching assistants, in-group students and out-of-group students. The transcription focuses mainly on the speech of the in-group students as well as the professor, teaching assistants and out-of-group students who talk to in-group students. All other sounds are ignored. Each speaker is identified by a generic speaker label to protect privacy, such as Student Male 1(SM1), Student Female 1(SF1), Teaching assistant Male 1(TM1), Professor(P), Out-group Male 1(OM1), etc. Except for out-of-group speakers, the voice recordings of the in-group speakers are given to the transcriber so that they can label them with the most similar identifier. If a voice does not match any of the in-group speakers, it will be labeled as out-of-group. Overlapping speech from multiple speakers may occur, and the transcribers will try their best to offer separate transcriptions for the two speakers. Since this course is taught in English but students are mainly Chinese, the group discussions are often conducted in Chinese-English code-switch modes. The transcriptions will strictly follow the languages spoken. All of the transcriptions are double checked to ensure the quality of the transcription. All transcribers signed a confidentiality agreement to protect the confidentiality of the recordings.

In order to capture the activity and amount of information exchange in discussion, the following speech features are considered.

• •

  Speech Time (ST)

  The original audio data is segmented and annotated with start and end timestamps. The duration between the start time and end time is calculated as the speech time of that segment.

• •

  Number of Words (NoW)

  The number of words for each segment is a measure that differs in English and Chinese. English is counted in word-unit and Chinese is counted in character-unit. The punctuation is ignored.

• •

  Number of Turns (NoT)

  As mentioned in Sect. III-D, the discussion dialog are segmented if speaker changes or a long pause of duration greater than one second occurs. The number of dialog turns is counted as number of dialog segments in transcription.

• •

  Speaking Rate Score (SRS)

  Speaking rate for one segment is initially calculated as the number of words divided by the speech time. Then each segment is rated as Slow, Normal or Fast, which are subsequently scored as 0,1,2 respectively. The classification thresholds for Slow, Normal and Fast are based on statistical findings of Yuan et al. [35], which claimed that the normal speed of short spoken English is 100-120 words per minute and the normal speed of short spoken Chinese is 225-255 characters per minute. Thus the thresholds are calculated as an equation (1)(2)

  $$Thres_{S/N}=100*Eng_{prop}+225*Cn_{prop}$$

  (1)

  $$Thres_{N/F}=120*Eng_{prop}+255*Cn_{prop}$$

  (2)

  where $Thres_{S/N}$ is the classification threshold between Slow and Normal, $Thres_{N/F}$ is the classification threshold between Normal and Fast, $Eng_{prop}$ and $Cn_{prop}$ are the proportion of English and Chinese used in one segment correspondingly.

The audio signal also contains a variety of information related to the dialog, such as emotions during discussions. Therefore, we extract certain features directly from the audio data. All of the audio features are first calculated for every frame (0.01 second in duration). Then the mean, maximal and minimal values within a segment are calculated as feature values of each segment. Gender broadly affects the values of audio features, especially related to F0. Presently we are not investigating the gender of the speakers, hence, all the audio features are gender-normalized. The following features are considered in this work:

• •

  Fundamental Frequency (F0)

  The fundamental frequency (F0) of a speech signal refers to the approximate frequency of the (quasi-)periodic structure of voiced speech signals. F0 can be used to measure the level of arousal in discussions and it was used to disambiguate discussion roles between a leader and an expert in the work of Scherer et al [29]. In this work, F0 detection is performed by COVAREP [45].

• •

  Energy

  Energy can be used to measure the level of arousal in discussions, and it was also used in the work of Scherer et al [29]. Energy of each frame is measured by Equation (3),

  $$Energy=\sum_{n}^{N}{x^{2}(n)}$$

  (3)

  where $N$ is the total number of samples in one frame, $x$ represents the audio signal, $n$ is the current sample in audio signal.

• •

  Formants

  In speech signal processing community, first formant (F1), second formant (F2) and third formant (F3) are the first, second and third acoustic harmonics of the fundamental frequency, respectively. Formants are important for emotion detection in Mandarin [46], and they were also used in the work of Worsley and Blikstein [47]. In our work, the value of F1, F2 and F3 are calculated based on the Praat software [48].

• •

  Glottal Features

  Glottal features describe the movements of vocalizing muscles. Glottal features have been shown to have a relationship with emotions and demeanor [49, 50, 51, 52], and they have been used in the field of learning analytics [29]. In this work, creak [53], rd [54, 55], Peak Slope (PS) [56], the difference between the first and second harmonics (H1-H2), Quasi Open Quotient (QOQ) [57], Maxima Dispersion Quotient (MDQ) [58] and Normalized Amplitude Quotient (NAQ) [51] are extracted by using COVAREP [45].

Audio feature abbreviations are defined and used as follows: {Max, Min, Avg}_{F0, Energy, NAQ, etc.}. For instance, Max_F0 represents that the feature value of one segment is obtained by taking the maximum value of F0 within that segment.

The features mentioned in Sections IV-A,  IV-B and  IV-C are created for each dialog segment. In order to derive some characteristics in group discussion dialog, all features need to be transformed to group-level. In this work, 7 types of group-level features in 2 aspects are considered.

The segments spoken by an in-group student can be aggregated together to be the feature of this student. Characteristics of a group discussion can be measured by the mean/variance of all in-group students’ characteristics. Therefore, we first take the sum/mean of feature value across the segments for each in-group student, and then the mean/variance among all in-group students. For example, we first calculate the sum/mean number of math terms per segment for each in-group student, and then take the mean/variance across the students within the group. This provides us with 4 types of group-level features as shown below (take math term as example):

• •

  Mean of the total number of math terms spoken by each in-group student

• •

  Variance in the total number of math terms spoken by each in-group student

• •

  Mean of the average number of math terms spoken by each in-group student

• •

  Variance in the average number of math terms spoken by each in-group student

In addition, we also look at the entire discussion dialog of a group for the topic of a given week, which contains dialog segments including those from the professor and teaching assistants to give guidance, as well as from students in a neighboring group who join in temporarily. We can compute the sum/mean/variance for features across all segments in the entire discussion. This provides us with 3 types of group-level features as presented below (take math term as example):

• •

  Sum of the number of math terms in an entire discussion dialog

• •

  Mean of the number of math terms in an entire discussion dialog

• •

  Variance in the number of math terms in an entire discussion dialog

We chose to use the midterm and final examination scores as measurement of group learning outcomes. The curriculum was carefully designed in such that each week contains a set of unique topics, and exact questions from the midterm and final examinations can be identified for each topic. For example, topics in Week 9 are covered by questions 10 and 15 in the final examination. Such a mapping enables us to extract the grades of the members of a group on particular questions, in order to assess the group’s learning outcome for the topic of a given week. Considering that the grades of the midterm and final examinations respectively accounts for 20% and 30% of the final grade for each student, we decided to use proportional weighting to compute the learning outcome for a given student group on the topic of a specific week, as shown in Equation (4).

where $S_{m}$ represents the grades earned by the group members from the midterm exam questions for the topic of a specific week, $S_{f}$ represents the grades earned by the group members from the final exam questions for the topic of a specific week, the full marks of $S_{m}$ is represented by $F_{m}$, and full marks of $S_{f}$ is represented by $F_{f}$.

Since we are able to identify the group learning outcome for the topics covered in each week, we are also able to use the scores of the group to rank their relative learning outcome in each week. Since there are 10 groups in total, we can also give them a ranking score as 10(top) to 1(bottom). The ranking score comes in handy for comparison especially when the groups have very close scores for specific topics in a week.

In order to explore the indicators of group learning outcomes from group discussions, correlation analysis and ANalysis Of VAriance (ANOVA) are performed between group-level features and group learning outcomes.

In this section, we will highlight the key results from statistical analyses. Results from correlation analysis will be presented by scatter plot, where the y-axis is the measurement of learning outcome and the x-axis is the normalized group-level features. Pearson correlation coefficient (r-value) and p-value are also provided below each sub-figure. ANOVA results will be presented by box plots, where the y-axis is group learning outcome and the x-axis is the normalized group-level features. The p-value is also provided below each sub-figure. In this work, we conducted statistical analysis at the significance level $\alpha=0.05$.