Model Review: A PROMISEing Opportunity

PROMISE will soon enter its third decade. What have we learned from two decades of PROMISE v1.0 that could shape the next decade of PROMISE v2.0?

Over the years, there have been many results where very simple models performed exceptionally well (Holte, 1993; Menzies et al., 2008; Agrawal et al., 2019; Xu et al., 2021; Tawosi et al., 2023; Kohavi and John, 1997). Using those results, PROMISE v2.0 could focus less on model creation, and more on something that requires and demands simpler and more comprehensible models. Specifically, I say PROMISE v2.0 should be about simplifying model review (using data mining).

PROMISE v1.0 was created one night in 2004 walking around Chicago’s Grant Park. Jelber Sayyad-Shirabad and I had spent the day at a disappointing workshop on data mining and SE. “Must do better”, we said. “Why don’t we do it like in ML? Make conclusions reproducible? Demand that if people publish a paper, they should also publish the data used in that data?"¹¹1 In 2023 it is hard to believe that “reproducible SE” was a radical idea. But once upon a time, there was little sharing of data and scripts– so much so that in 2006 Lionel Briand predicted the failure of PROMISE saying “no one will give you data”..

At first, the series got off to a shaky start. But once Elaine Weyuker, Thomas Ostrand, Gary Boetticher, and Guenther Ruhe joined the steering committee, the meeting earned the prestige needed for future growth. And in those early days, it was impressive to see so many researchers taking up the idea of reproducible results. Numerous papers were written that applied an increasing elaborate tool set to data like COC81, JM1, XALAN, DESHARNIS and all the other data sets that were used (and reused) in the first decade of PROMISE.

Many of those papers lead to successful research. In 2018, 20% of the articles listed in Google Scholar Software Metrics for IEEE Transactions on SE used data sets from the first decade of PROMISE. So while other research areas struggled to obtain reproducible results, PROMISE swam (as it were) in an ocean of reproducibility.

The problem was that in the second decade of PROMISE, many researchers still continue that kind of first-decade research. For example, all too often, I must review papers from authors who think it is valid to publish results based on (e.g.) the COC81 data set first published in 1981 (Boehm, 1981); the DESHARNIS data set, first published in 1989 (Desharnais, 1989); the JM1 data, first published in 2004 (Menzies and Di Stefano, 2004); or the XALAN data set, first published in 2010 (Jureczko and Madeyski, 2010)²²2Just to be clear, there is value in a publicly accessible collection of reference problems. For instance, if a PROMISE author is unable to present results from confidential industrial data, they can use the reference collection to construct a reproducible example of their technique. That said, I am usually tempted to reject papers that are solely based on defect datasets that I contributed to the PROMISE repository in 2005 (e.g. CM1, JM1, KC1, KC2, KC3, KC4, MC1, MC2, MW1, PC1, PC2, PC3, PC4 and PC5) since, in 2023 we have access to much more recent data (e.g. see the 1100+ recent Github projects used by Xia et al. (Xia et al., 2022))..

Meanwhile, AI fever took over SE. As of 2018, it became standard at ASE, FSE, ICSME, ICSE, etc. to see papers that make much use of AI. For example, the MSR conference (which in the early days looked like a sister event to PROMISE) has grown to a large annual A-grade venue. And just as MSR grew, so did PROMISE shrink. In 2008, PROMISE was a two-day event with 70 attendees. PROMISE is now a much smaller and shorter event. Without definitive results or a novel technological position, it became difficult to differentiate PROMISE from dozens of other, somewhat more prominent, venues.

Reducing the size of the model is an important part of model review. According to psychological theory (Czerlinski et al., 1999; Gigerenzer et al., 1999; Martignon et al., 2003; Brighton, 2006; Martignon et al., 2008; Gigerenzer, 2008; Phillips et al., 2017; Gigerenzer and Gaissmaier, 2011; Neth and Gigerenzer, 2015), humans can best review a system when it “fits” it into their memory; i.e., when that it comprises many small model fragments. Larkin et al. (Larkin et al., 1980) characterize human expertise by a very large long-term memory (LTM) and a very small short-term memory (STM) that contains as few as four to seven items³³3Ma et al. (Ma et al., 2014) have used evidence from neuroscience and functional MRIs to argue that the capacity of the STM could be better measured using other factors than “number of items “”. But even they conceded that “the concept of limited (STM) has considerable explanatory power for behavioral data”.. The LTM contains many tiny rule fragments that explore the contents of the STM to say “when you see THIS, do THAT”. When an LTM rule triggers, can rewrite the STM content, which, in turn, can trigger other rules. Experts are experts, says Larkin et al. (Larkin et al., 1980) because LTM patterns dictate what to do, without having to pause for reflection. Novices perform worse than experts, says Larkin et al., when they fill up their STM with too many to-do’s where they plan to pause and reflect on what to do next. This theory is widely endorsed. Phillips et al. (Phillips et al., 2017) discuss how models with tiny fragments can be quickly comprehended by emergency room physicians making rapid decisions; or by guard soldiers making snap decisions about whether to fire or not on a potential enemy; or by stockbrokers making instant decisions about buying or selling stock.

Complex models cannot fit into STM, leading to problems with model review. Green (Green, 2022) comments that when faced with large and complex problems, cognitive theory (Simon, 1956) tells us that humans use heuristic “cues” to lead them to the most important parts of a model. Such cues are essential if humans are to reason about large problems. That said, using cues can introduce their own errors: …people (including experts) are susceptible to “automation bias” (involving) omission errors - failing to take action because the automated system did not provide an alert - and commission error (Green, 2022). This means that oversight can lead to the opposite desired effect by “legitimizing the use of faulty and controversial (models) without addressing (their fundamental issues”) (Green, 2022).

By “faulty and controversial models”, Green refers to the long list of examples where detrimental models were learned via algorithmic means. For example, Cruz et al. (Cruz et al., 2021) lists examples where:

• •

  Proposals from low-income groups are are five times more likely to be incorrectly ignored by donation groups;

• •

  Woman can be five times more likely to be incorrectly classified as low income;

• •

  African Americans are five times more likely to languish in prison until trial, rather given the bail they deserve.

These are just a few of the many reported examples(Rudin, 2019)⁴⁴4See also http://tiny.cc/bad23a, http://tiny.cc/bad23b, http://tiny.cc/bad23c of algorithmic discrimination. For another example, the last chapter of Noble (Noble, 2018) describes how a successful hair salon went bankrupt due to internal choices within the YELP recommendation algorithm.

Like Mathews (Johnson and Menzies, 2023), I am not surprised that so many models are unfair. Mathews argues that everyone seeks ways to exploit some advantage for themselves. Hence, we should expect that the software we build to discriminate against some social groupings;

“People often think of their own hard work or a good decision they made. However, it is often more accurate to look at advantages like the ability to borrow money from family and friends when you are in trouble, deep network connections so that you hear about opportunities or have a human look at your application, the ability to move on from a mistake that might send someone else to jail, help at home to care for children, etc. The narrative that success comes from hard work misses that many people work hard and never succeed. Success often comes from exploiting a playing field that is far from level and when push comes to shove, we often want those advantages for our children, our family, our friends, our community, our organizations.”

Hence I assert that unfairness is a widespread issue that needs to be addressed and managed. Specifically, we need to ensure that a software system created by one group, $A$, can be critiqued and modified by another group, $B$. There are many ways to do this and at PROMISE’23, it is appropriate to focus on the data mining methods issues (see next section).

But first, we take care to stress that technical methods like data mining should not be used in isolation. PROMISE v2.0 should acknowledge its relationship and responsibilities to those affected by the tools we deliver. We must stop “flattening”, which is the trivialization and even ignoring of legitimate complaints that our institutions discriminate against certain social groups (Coaston, 2019). Bowleg (Bowleg, 2021) warns that flattening “depoliticized and stripped (its) attention to power, social justice, and (how we do things wrongly)”.

To address these issues, organizations should review their hiring practices to diversify the range of perspectives seen in design teams. Requirements engineering practices should be improved to include extensive communication with the stakeholders of the software. Software testing teams should extend their tests to cover issues such as discrimination against specific social groups (Cruz et al., 2021; Majumder et al., 2023; Chakraborty et al., 2021).

Further, on the legal front, Canellas (Canellas, 2021) and Mathews et al. (Matthews, 2023) suggest a tiered process in which the most potentially discriminatory projects are routinely reviewed by an independent external review team (as done in the IEEE 1012 independent V&V standard). Ben Green (Green, 2022) notes that reviewing software systems and AI systems is becoming a legislative necessity and that human-in-the-loop auditing of decisions made by software is often mandatory. Such legislation is necessary to move away from the internal application of voluntary industrial standards (since, as seen in the Volkswagen emissions scandal (see http://tiny.cc/scandalvw), companies cannot always be trusted to voluntarily apply reasonable standards.

How to make models smaller, more comprehensible, and easier to review? Enter all the data mining tools explored at PROMISE. In those explorations, often it was seen that

Outside of SE, I have seen keys in hospital nutrition analysis (Partington et al., 2015) and avionic control systems (Gay et al., 2010). Within SE, I have seen keys in:

• •

  Defect prediction datasets (Menzies et al., 2006) where two to three attributes were enough to predict for defects

• •

  Effort estimation models (Chen et al., 2005) where four to eight attributes where enough to predict for defects;

• •

  Requirements models for NASA deep space missions (Jalali et al., 2008) where two-thirds of the decision attributes could be ignored while still finding effective optimizations;

• •

  11 Github issue close time data sets (Rees-Jones et al., 2017) where only 3 attributes (median) were needed for effective prediction.

One way to see how many keys are in a system is to ask how many prototypes (minimum number of examples) are required to explore that system. At PROMISE"08, the keys were found in 50 examples (selected at random) since the models built from this small sample performed no worse than the models learned from thousands of additional examples (Menzies et al., 2008). In 2023 we made a similar observation. In a study that explored 20 years of data from 250 Github projects with 6000 commits per project (average). In that study, the defect models learned from only the first 150 commits were predicted, as well as the models learned from much larger samples (N.C. and Menzies, 2023).

Of course, not all data sets can be explored by a few dozen keys. Recently, we have successfully modeled security violations in 28,750 Mozilla functions with 271 exemplars and 6000 Github commits using just 300 exemplars (Yu et al., 2019)⁵⁵5Specifically, after incremental active learning, the SVM had under 300 support vectors.. Although 300 is not an especially small number, it is small enough so that, given (say) two analysts and a month, it would be possible to review them all.

Note that keys have obvious implications for software testing. Ling (Ling and Menzies, 2023) generates test suites for cyberphysical systems by first finding prototypes. In that work, $N$ candidate tests are recursively bi-clustered to leaves of size $\sqrt{N}$ (using the methods of Table 1). Test suites generated from the mode of each cluster are orders of magnitude faster to generate and just as effective as tests generated by more complex (and slower) methods.

This recursive bi-clustering method has also been applied to multi-objective optimization (Agrawal et al., 2020). Given a large enough initial population (e.g. $N=10,000$), recursive bi-clustering is faster and finds better solutions than state-of-the-art genetic algorithms and sequential model optimization methods (Chen et al., 2019; Lustosa and Menzies, 2023) (even though it only evaluates $2\log{10,000}=26$ examples while other methods might evaluate 100s to 1000s of examples).

Once we have a keys-based multi-objective optimizer, we can offer much support for reviewing models with respect to their fairness. Figure 1 comes from Cruz et al. (Cruz et al., 2021). That figure shows the effects of 10,000 different hyperparameter options applied to five machine learning algorithms (random forest; LinReg; boosted trees; decision trees; feed-forward NN)⁶⁶6The hyperparameters of Random Forests, learners include (a) how many $T$ trees to build (e.g., $T\in\{10,20,40,80,160\}$); (b) how many features $F$ to use in each tree (e.g., $F\in{2,4,10,20,\mathit{sqrt},\mathit{log2},\mathit{all}}$); (c) how to poll the whole forest (e.g., majority or weighted majority); (d) what impurity measures (e.g., gini or entropy or log.loss); (e) what is the minimum examples needed to branch a sub-tree (e.g., min$\in\{2,5,10,20,50,100\}$; (f) should branches be binary or n-arty. In all, this gives us $5*7*2*3*6*2>2,500$ different ways, just to configure a learner in Figure 1. . Adjusting tunings can change learners from low to high accuracies and fairness (measured here as the ratio of false positives between different social groups, such as men and women). But with the search methods of the Table 1, reviewing all those points would require just 20 evaluations– a number so small that (potentially) it could happen between coffee breaks of a stakeholder review session.

Having presented the methods of the last section, I rush to add that they are hardly complete. There is much here that could be better defined/ refined/ improved/ replaced by further research in PROMISE v2.0.

In my view, the goal of a PROMISE v2.0 paper could be “less is more”; that is, achieve faster, simpler, better results using some simplification of the existing technique. There are many ways this could be done; e.g. with:

• •

  Semi-supervised learning methods that let us do much more with much less data (Tu and Menzies, 2021; Zhu, 2005);

• •

  Instance or feature selection to reduce training data (Olvera-López et al., 2010; Kohavi and John, 1997);

• •

  Distillation methods reduce model size. For example, a new data set could be synthesized from the branches of a decision tree. By summarizing this new data set, do we find a smaller model? And for alternative neural approaches, see (Shi et al., 2023; Biswas and Rajan, 2023).

• •

  Variance studies that tests if the improvement of a complex method over a simpler one are statistically insignificant;

• •

  Ablation studies (Yedida et al., 2023) to see how much can be thrown away (of the modeling method, of parts of the training data) while preserving model performance.

• •

  Studies showing that (say) a 10% implementation can perform nearly as well as 100% of an entire system.

• •

  Some kind of keys-based approach (e.g. see last section).

Human-in-the-loop studies would also be strongly encouraged in PROMISE v2.0, to test if the smaller models are still acceptable and useful for people. But in line with PROMISE’s long and admirable history of reproducibility, these experiments should include human surrogates (developed perhaps via data mining) that can model the strengths and weaknesses of subject matter experts (and surrogates should be shared as part of the reproducibility package of a paper).

Also encouraged would be a broader range of performance criteria than just (e.g.) accuracy, Performance should be measured in a multidimensional manner and include much more than mere predictive performance, but also runtime, energy usage, discrimination measures, model development cost, etc.

My “less is more” proposal is antithetical to much current research in SE and AI. Data-hungry researchers in SE assume that “more is more”; i.e. if some data are useful, then even more data is even more useful. For example “Long- term JIT models should be trained using a cache of many changes” (Amasaki, 2020); and “..as long as it is large; the resulting prediction performance is likely to be boosted more by the size of the sample’” (Rahman et al., 2014).

A common problem with “more is more“ is that researchers often make that assumption without actually testing it. For example, a recent systematic review (Hou et al., 2023) of the literature on large language models in SE reported 229 research articles from 2017 to 2023. We asked the authors of that review, “how many articles compared their approach to something simpler non-neural approach?” and “in how many of those comparisons was there any hyperparameter optimization?”. They responded with a list of 13 papers ($\frac{13}{229}\approx 5\%$) which, when read, contain some questionable methodological choices. For example, one of those 13 articles reported that LLMs perform better than a text mining methods called LDA. But that article used LDA in its “off-the-shelf” configuration even though we have seen dramatic improvements in LDA performance with hyperparameter optimization (Agrawal et al., 2019, 2018).

To be clear, I firmly believe that deep learning and generative AI methods such as “chain of thought”⁷⁷7https://github.com/Timothyxxx/Chain-of-ThoughtsPapers will dramatically change the nature of science (in general) and SE (in particular). But moving away from generative tasks to classification, regression, and optimization tasks, my experimental results strongly suggest that other non-neural methods can be just as effective, particularly when combined with hyperparameter optimization. This is an important point since the non-neural methods can yield the succinct symbolic models that humans need to review and understand a model.

But rather than stating all this in an adversarial manner, it might be more useful to ask how “less is more” can be beneficial for more elaborate approaches. Lustosa (work in progress) has explored the hyperparameters of some deep learners using the recursive bi-clustering method described in the Introduction, and found that it was able to configure the deep learners more effectively than other state-of-the-art optimizers, and do so much faster.

Despite all my papers on the topic, “less is more” is mostly ignored. Perhaps this is my own fault. Initially, I argued this with simulation studies on artificially created examples, which some people find unconvincing (Menzies et al., 2000; Menzies and Singh, 2004). However, subsequent work reported results from real-world data  (Menzies et al., 2007, 2008; Chen et al., 2005; Menzies et al., 2006; Partington et al., 2015; Jalali et al., 2008; Chen et al., 2019; Lustosa and Menzies, 2023; Agrawal et al., 2020; Ling and Menzies, 2023; Yu et al., 2019; N.C. and Menzies, 2023). Furthermore, the latter work included numerous studies that demonstrate that this “less is more” approach produces smaller and better models than the current state-of-the-art.

Perhaps there is something deeply embedded in our research culture that encourages and rewards complexity. Perhaps our research is driven by the concerns of large software organizations that prefer complexity (since only those large organizations have the resources to build and maintain complex solutions).

Perhaps also, in a publication-oriented environment, researchers tend to rush out reports of complex mashups of tools, rather than refactor and reduce the size of their toolkits.

Perhaps what we need is a space where we can revisit and reflect on old results, looking for some synthesis that significantly simplifies and improves those results. To create a journal venue for such papers, I invite submissions to the “Less is More” section of the Automated Software Engineering journal⁸⁸8See https://ause-journal.github.io/simpler.html.

And as to an associated conference venue, perhaps that venue could be PROMISE v2.0?

Thanks to Hou et al. for quickly responding to a query on LLMs. Also, thanks to all the grad students and coauthors who helped mature, refine, and simplify the ideas that led to this paper.