Population-Based Hierarchical Non-negative Matrix Factorization for Survey Data
^††thanks: This work was partially supported by NSF grants DMS-2011140, DMS-2108479, and DMS-2103093.

We construct synthetic explanatory variables $\mathbf{X}^{\text{I}}\in\mathbb{R}^{1600\times 120}$ containing $1600$ observations on $120$ continuous features with hierarchical population structure as described in Fig. 1. In the context of topic modeling for survey data, these data mimic a population containing eight groups, each containing $200$ respondents. Each respondent discusses some combination of four topics and each topic consists of $30$ words. While we describe these data in the context of topic modeling, our approach also applies to continuous variables in general.

We construct $\mathbf{X}^{\text{I}}$ as the product of two rank-$4$ matrices with

where $\mathbf{W}^{\text{I}}\in\mathbf{R}^{1600\times 4}$ is a person-topic matrix and $\mathbf{H}^{\text{I}}\in\mathbf{R}^{4\times 120}$ is a topic-word matrix. Therefore, $\mathbf{W}_{i,j}^{\text{I}}$ reflects the amount that the $j^{th}$ topic is discussed by the $i^{th}$ person, and $\mathbf{H}_{i,j}^{\text{I}}$ reflects the importance of the $j^{th}$ word in the $i^{th}$ topic.

We construct $\mathbf{W}^{\text{I}}$ according to the hierarchical structure depicted in Fig. 1. At the first split, Groups 1 and 2 discuss words in Topic 1 according to zero-truncated $\mathcal{N}(64,9)$ and $\mathcal{N}(3,9)$ distributions, respectively. At the second split, Groups 1a and 2a and Groups 1b and 2b discuss words in Topic 2 according to zero-truncated $\mathcal{N}(45,9)$ and $\mathcal{N}(3,9)$ distributions, respectively. At the third split, Groups 1a1, 2a1, 1b1, and 2b1 and Groups 1a2, 2a2, 1b2, and 2b2 discuss words in Topic 3 according to zero-truncated $\mathcal{N}(3,9)$ and $\mathcal{N}(50,9)$ distributions, respectively. Finally, the entire population discusses words in Topic 4 according to a zero-truncated $\mathcal{N}(50,9)$ distribution.

We construct $\mathbf{H}^{\text{I}}$ as follows. Suppose that each of the four topics consists of 30 words. For simplicity, we assume that the words in each topic appear together in the columns of $\mathbf{H}^{\text{I}}$. For each topic, we construct 30 vectors of length four by sampling from a multinomial distribution so that the words in each topic are four times more likely to appear in that topic than in the others. This simulates a more disjoint vocabulary; words are less likely to appear in multiple topics simultaneously. Fig. 2 depicts the structure in the resulting synthetic explanatory variables matrix $\mathbf{X}^{\text{I}}$.

We construct a synthetic coefficient vector $\boldsymbol{\theta}_{g}\in\mathbb{R}^{4}$ with $1\leq g\leq 8$ for the subgroups. We obtain a synthetic response $\mathbf{y}^{\text{I}}\in\mathbb{R}^{1600}$ by multiplying the corresponding rows of $\mathbf{W}^{\text{I}}$ with $\boldsymbol{\theta}_{g}$ for $1\leq g\leq 8$. This simulates a response that is determined by different combinations of the topics for each subgroup.

We construct synthetic explanatory variables $\mathbf{X}^{\text{II}}\in\mathbb{R}^{1600\times 120}$ containing $1600$ observations on $120$ dichotomous categorical features with hierarchical population structure as described in Fig. 1. We construct $\mathbf{X}^{\text{II}}$ according to

where we employ the same $\mathbf{W}^{\text{I}}$ and construct $\mathbf{H}^{\text{II}}$ similarly to $\mathbf{H}^{\text{I}}$. We threshold the entries in the product $\mathbf{W}^{\text{I}}\mathbf{H}^{\text{II}}$ so that $\mathbf{X}^{\text{II}}$ contains only 1’s and 0’s.

We highlight some differences in the procedures for constructing $\mathbf{H}^{\text{II}}$. Rather than employing 30 words in each topic as in $\mathbf{H}^{\text{I}}$, we vary the number of words in each topic to ensure ordered topical importance. To see this, notice that a topic’s relative importance is determined by the magnitudes of its entries and its size, or the number of words it contains. For example, equal numbers of entries containing 1’s and 0’s and equal topic sizes produces equally important topics. Therefore, we vary the number of words in each topic to induce an importance hierarchy. Topic 1 consists of 65 words, Topic 2 consists of 30 words, Topic 3 consists of 20 words, and Topic 4 consists of 5 words. We again assume that words appear together by topic.

To ensure that $\mathbf{X}^{\text{II}}$ contains only dichotomous categorical variables, we threshold the entries in the product $\mathbf{W}^{\text{I}}\mathbf{H}^{\text{II}}$ as follows. Suppose that the $j^{th}$ word belongs to the $l^{th}$ topic and let $\text{med}_{l}$ denote the median value of the words in $\text{Topic}\,l$. Then we obtain $\mathbf{X}^{\text{II}}_{i,j}$ with

Fig. 3 depicts the structure in $\mathbf{X}^{\text{II}}$. We employ the same response variable $\mathbf{y}^{\text{I}}$ since the hierarchical population structure determined by $\mathbf{W}^{\text{I}}$ remains unchanged.

We employ real data containing survey responses from residents of Austin, Texas on their satisfaction with various city services [12]. Our dataset consists of the $3{,}723$ completed surveys obtained since 2018 that do not contain any empty columns.

Approximately 200 columns correspond to multiple choice questions such as the following.

1. 1.

   Overall quality of public safety services (i.e. police, fire and ambulance) (Choices: Very Dissatisfied, Dissatisfied, Neutral, Satisfied, Very Satisfied, Don’t Know)

2. 2.

   I feel safe in my neighborhood during the day (Choices: Strongly Disagree, Disagree, Neutral, Agree, Strongly Agree, Don’t Know)

3. 3.

   Have you had contact with the City of Austin Municipal Court? (Choices: Yes, No, Don’t Know)

4. 4.

   Which two items listed in Quality of Life do you think are most important for the City to provide? (Choices: ‘Feel Welcome’, ‘Overall quality of life’, ‘Place to Live’, ‘Place to Work’, ‘Raise Children’, ‘Retire’, ‘None’)

In addition to the multiple choice questions, we include eight columns corresponding to open-form response questions on respondent satisfaction levels. We exclude open-form questions that closely resemble the multiple choice questions.

We consolidate multiple choice responses on satisfaction to positive, neutral, and negative responses. We construct indicator variables for positive and negative responses and retain these as input variables. For each open-form question, we employ tfidfvectorizer from scikit-learn in Python [13] to convert text responses to a respondent-word matrix and perform NMF to identify the primary response topics. For each topic, we construct a new variable whose entries are the corresponding NMF coefficients indicating how much each respondent discusses each topic, and set the largest coefficient to $1$. The resulting dataset contains $3{,}723$ observations on $250$ continuous and categorical variables. We exclude demographic variables in performing PHNMF and employ them only to ascertain the meaningfulness of the PHNMF subgroups.

We employ real data containing multiple choice questions from a Facebook survey measuring public opinion on climate change from $2{,}432$ respondents [14]. We emphasize that these data and results do not reflect our personal views on climate change. We apply PHNMF to the following seven multiple choice questions.

1. 1.

   Do you think that global warming is happening? (Choices: Yes, No, Don’t know)

2. 2.

   Assuming global warming is happening, do you think it is mostly… (Choices: Caused by human activities, Caused by natural environmental changes, Other)

3. 3.

   Which comes closest to your own view? (Choices: Most scientists think global warming is not happening, There is much disagreement among scientists about whether or not global warming is happening, Most scientists think global warming is happening, Don’t know)

4. 4.

   How much do you think global warming will harm you personally? (Choices: A great deal, A moderate amount, Only a little, Not at all, Don’t know)

5. 5.

   How much do you think global warming will harm future generations of people? (Choices: A great deal, A moderate amount, Only a little, Not at all, Don’t know)

6. 6.

   How much do you think global warming will harm plant and animal species? (Choices: A great deal, A moderate amount, Only a little, Not at all, Don’t know)

7. 7.

   How much do you support or oppose funding more research into renewable energy sources, such as solar and wind power. (Choices: Strongly support, Somewhat support, Somewhat oppose, Strongly oppose)

For each question, we construct indicator variables for each multiple choice response. The resulting dataset contains $2{,}427$ observations on $35$ categorical variables. Again, we exclude demographic variables when performing PHNMF and employ them only to investigate the meaningfulness of the PHNMF subgroups.

Fig. 6 depicts a heatmap of the population structure discovered with PHNMF on the City of Austin Satisfaction Survey. The heatmap suggests non-trivial structure on the columns and the rows.

Fig. 7 depicts the hierarchical population structure discovered with PHNMF on the same dataset. The population splits are interpretable and also provide an understanding of differences in survey responses among respondents. The first PHNMF split divides the $3{,}723$ survey respondents into two subgroups with $1{,}312$ and $2{,}411$ respondents, respectively. Group 1 primarily responds negatively to multiple choice questions on satisfaction with city services, while Group 2 primarily responds positively to the same questions.

At the second level, PHNMF divides Group 1 into two subgroups with $391$ and $921$ respondents, respectively. While Group 1a is distinguished by high levels of dissatisfaction, Group 1b reveals affirmative responses to feeling safe in Austin and satisfaction with the fire department and garbage collection services. We compare demographic statistics on the PHNMF subgroups and find that compared with the entire population, Group 1a reports lower average income, lower full-time employment rates, and lower percentages of owning their home compared to the entire population. Meanwhile, Group 1b reports comparable statistics to that of the entire population in these measures.

At the third level, PHNMF divides Group 1a into two groups with $145$ and $246$ respondents, respectively, and Group 1b into two groups with $503$ and $418$ respondents, respectively. Group 1a.1 is distinguished by consistent dissatisfaction. Meanwhile, Group 1a.2 expresses dissatisfaction with traffic flow and city planning but also expresses satisfaction with Austin’s emergency services, and affirms feeling safe in the city. Group 1b.1 is distinguished by satisfaction with city libraries and Austin as a place to live and feel welcome, and only express dissatisfaction with traffic flow. By contrast, Group 1b.2 has higher rates of dissatisfaction, but affirms feeling safe at home and in their neighborhood, and trusts emergency services.

At the fourth level, PHNMF splits Group 1a.1 into two groups with $96$ and $51$ respondents, respectively, and Group 1b.1 into two groups with $204$ and $299$ respondents, respectively. Group 1a.1a and Group 1a.1b consistently respond negatively but Group 1a.1a is more concerned with traffic and city planning. Meanwhile, Group 1a.1b primarily expresses dissatisfaction with the City of Austin’s communications with its residents. By contrast, Group 1b.1a and Group 1b.1b mainly express satisfaction. Group 1b.1a is satisfied with the city library and learning centers, while Group 1b.1b is satisfied with Austin as a place to live and feel welcome, and report feeling safe in the city.

Since $27$ of the $250$ columns correspond to continuous variables, we convert these to categorical variables to employ LCA. However, LCA struggles to identify the best number of latent classes $k$ in this dataset. When we enable $k$ to range from $1$ to $15$, LCA reports insufficient degrees of freedom to select an optimal number of classes for $k\geq 13$. When we limit the number of classes to $k\leq 12$, however, LCA selects $k=12$. Since LCA does not detail the importance of the variables in contributing to each class, the resulting 12 classes on the $250$ columns are difficult to interpret. Since a small percentage of the variables in this dataset are continuous, we do not employ LPA.

Fig. 8 depicts a heatmap of the population structure discovered with PHNMF on the Facebook Climate Change Survey. The heatmap depicts non-trivial structure on the columns and the rows.

Fig. 9 depicts the hierarchical population structure discovered with PHNMF on the same dataset. The population splits are interpretable and also align with existing findings on climate change. At the first split, PHNMF divides the $2{,}427$ survey respondents into two subgroups with $1{,}665$ and $762$ respondents, respectively. Group 1 primarily answers affirmatively to whether global warming exists and expresses strong support for funding renewable energy. Meanwhile, Group 2 primarily indicates that global warming is caused by natural environmental changes and is unworried about these changes. We compare demographic statistics on the PHNMF subgroups and find that Group 1 is approximately 60% female while Group 2 is approximately 60% male. This aligns with findings that women are more likely to affirm and express greater concern over climate change ([20], [21]). Additionally, about 60% of Group 1 is under the age of 45 while about 60% of Group 2 is over the age of 45. This aligns with findings that younger generations are more likely to support a shift to renewable energy [22]. Finally, 44% of Group 1 identifies as Democratic and 7% as Republican whereas 4% of Group 2 identifies as Democratic 40% as Republican. This aligns with findings that Democrats tend to express greater concern over climate change [23].

At the second split, PHNMF divides Group 1 into subgroups Group 1.1 and Group 1.2 with $853$ and $812$ respondents, respectively. The distinguishing factor between these subgroups is their level of concern for global warming. Group 1.1 expresses a great deal of worry while Group 1.2 expresses being only somewhat worried. The percentage of women in Group 1.1, Group 1.2, and the population at large is 64%, 56%, 50%, respectively. This aligns with findings of greater levels of climate change concern among women [24]. Additionally, Group 1.1 consists of 51% Democrats and 4% Republicans while Group 1.2 consists of 37% Democrats and 10% Republicans. This aligns with findings that Democrats express greater concern over climate change [23]. At the second split, PHNMF also divides Group 2 into Groups 2.1 and 2.2 with $359$ and $403$ respondents, respectively. Group 2.1 acknowledges the existence of global warming to some extent while Group 2.2 denies global warming altogether.

The third split divides Group 1.1 into Group 1.1.1 and Group 1.1.2 with $415$ and $438$ respondents, respectively. Group 1.1.1 upholds that global warming is occurring and expresses concern, but only expresses moderate concern for the harm that global warming will cause them. Group 1.1.2 identifies that global warming will harm themselves, the planet, and future generations. Members of Group 1.1.2 are about 50% more likely to be under the age of 18 than those of Group 1.1.1. This aligns with findings that younger individuals are more likely to express concern that climate change will harm them [25]. Additional splits are likewise interpretable and appear to align similarly with existing findings on climate change views.

The corresponding LCA analysis finds $14$ classes within the $35$ variables on the first split. The large number of classes is difficult to interpret. When we enforce only two classes in the first LCA split, we obtain subgroups with $1{,}568$ and $859$ respondents, respectively. The distinguishing features between these two subgroups mirror the differences found between the PHNMF subgroups. However, when we run LCA again on these two subgroups, we obtain 6 and 7 subgroups for each group, respectively. This is again difficult to interpret. We do not employ LPA since it requires continuous variables.

Tab. III depicts the cosine similarity between the coefficient vectors obtained with ordinal ridge regression on the discovered PHNMF subpopulations and those obtained from the same analysis on the entire population.

We observe greater alignment between coefficient vectors in subpopulations obtained from earlier PHNMF splits. However, the directions differ more as PHNMF gleans finer subpopulation structure with additional splits. While not depicted here, we also note that the most statistically significant regression variables identified by ordinal ridge regression agree with our interpretations of the PHNMF subpopulations. For example, for Group 1.1, the variable corresponding to “Yes, global warming is happening” is not significant since the entire group is distinguished by their agreement with this statement. However, within this subpopulation, the most significant variables relate to the extent that respondents are concerned about the impact of climate change on the future. This coincides with the distinguishing feature of the next PHNMF split.