Towards Lean Research Inception: Assessing Practical Relevance of Formulated Research Problems

The practical relevance of Software Engineering (SE) research is often limited by a disconnect between academia and industry, with studies failing to address practically relevant challenges despite their theoretical contributions (Garousi et al., 2020; Franch et al., 2020; Winters, 2024). Gorschek et al. (Gorschek et al., 2006; Gorschek and Mendez, 2021) emphasize that a well-defined problem formulation is critical for research to generate industrial impact. Many research problems are formulated in isolation, lacking input from industrial practice (Gorschek and Mendez, 2021). This disconnect hinders understanding of real and practical industry challenges. Ensuring practical relevance requires structured approaches that align research with industry needs through clear problem formulation and assessment (Ali, 2016; Molléri et al., 2023; Storey et al., 2024).

To integrate academic and industry perspectives early in the research process, we introduce the Lean Research Inception (LRI) framework to support the formulation and initial assessment of practically relevant SE research problems. In this paper, we present the complete vision of the LRI framework and an evaluation of its research problem assessment phase. The assessment employs a semantic differential scale (Heise, 1970) to assess the relevance of formulated research problems based on three criteria: valuable (whether solving the problem can generate meaningful impact for industrial practice), applicable (whether the problem leads to practical and usable outcomes in industry), and feasible (whether the problem can realistically be investigated given available resources). The scale was inspired by agile principles and Lean Startup’s Minimum Viable Product (MVP) concept (Ries, 2011), aiming to enhance the effectiveness of aligning research with industry needs.

The empirical evaluation was conducted through a workshop with 42 senior SE researchers at the 2024 annual meeting of the International Software Engineering Research Network (ISERN¹¹1https://isern.iese.de/). The study focused on evaluating the three key criteria – valuable, feasible, and applicable – used in the semantic differential scale. Participants analyzed these criteria by discussing an example research problem, capturing their insights through collaborative annotations, and individually applying the criteria to assess the problem’s relevance. Following this discussion, they completed a survey on the perceived importance and completeness of each criterion. The survey provided quantitative and qualitative insights into how well these criteria are perceived as important to support an initial assessment of the practical relevance of formulated research problems.

The results indicate that our criteria, valuable (83.3%), feasible (76.2%), and applicable (73.8%) are appropriate for assessing research problem relevance, with high levels of agreement among participants. Furthermore, qualitative feedback suggested adjustments in terminology for a clearer distinction between feasible and applicable, and refinements for valuable to include concrete examples to ease understanding, such as business value, ROI, and originality.

Software Engineering research is primarily aimed at industrial applications, which makes alignment with industry needs crucial for practical relevance (Winters, 2024). Achieving this requires strategies that balance theoretical advancements with real-world applicability (Stol and Fitzgerald, 2018). Ivarsson et al. (Ivarsson and Gorschek, 2011) proposed a model to assess the rigor and industrial relevance of technology evaluations in SE. In their analyses, the model revealed that most evaluations lacked both rigor and industrial relevance. Additionally, the study found no significant improvement in industrial relevance over time.

Garousi et al. (Garousi et al., 2020) investigate the practical relevance of SE research, synthesizing community opinions and evidence through a multivocal literature review (MLR). They identify three key factors limiting relevance: too simplistic views of practice, weak industry connections, and poor problem identification. To address this, they recommend adopting appropriate research approaches, focusing on practical problems, and strengthening industry collaboration. The study emphasizes the need for rigorous empirical research to better align SE studies with industrial needs. Winters (Winters, 2024) further highlights the misalignment between SE research and industry needs, criticizing the lack of practical context, narrow problem scopes, and limited scalability, emphasizing the need for research that delivers measurable and practical value to industry.

Petersen et al. (Petersen et al., 2024) propose a reasoning framework to enhance the design, assessment, and reporting of industrial relevance in SE research. Given the lack of consensus on defining and measuring relevance, they review key attributes such as applicability, context, and practical impact. Their framework, structured around six aspects (what, how, where, who, when, and why), provides an approach for evaluating and communicating research relevance of applied research in industry contexts. Storey et al. (Storey et al., 2024) propose a playbook for researching disruptive innovations in SE. They emphasize interdisciplinary collaboration, industry engagement, and iterative experimentation while underscoring the need for clear criteria to assess impact and applicability in industrial and social contexts.

These studies emphasize the importance of aligning SE research with industry needs, reinforcing the motivations behind our work. Ivarsson et al. (Ivarsson and Gorschek, 2011) highlight the persistent lack of rigor and industrial relevance in SE evaluations, underscoring the need for structured approaches to bridge this gap. Garousi et al. (Garousi et al., 2020) and Winters (Winters, 2024) critique the disconnect between academia and industry, pointing to weak collaboration and impractical research focus—challenges LRI directly addresses by integrating practitioners early in the research process. Petersen et al. (Petersen et al., 2024) propose a reasoning framework for industrial relevance, which helped to define LRI’s problem formulation attributes and assessment criteria. Finally, Storey et al.(Storey et al., 2024) provide valuable insights into assessing the impact of disruptive innovations.

LRI advances previous work by proposing an iterative framework that integrates academic and industry perspectives from the outset to support the formulation of research problems and an initial assessment of their practical relevance through clear and easy-to-apply criteria.

The Lean Research Inception (LRI) framework was developed to help bridging the gap between SE research and industrial practice. Grounded in agile principles and methodologies such as Design Thinking (Plattner et al., 2009), Lean Startup (Ries, 2011), and Lean Inception (Caroli, 2017), LRI fosters collaboration between researchers and practitioners from the very beginning of a research project. Its core objective is to integrate these stakeholders early on to ensure that research problems are practically relevant and applicable to real-world scenarios.

LRI complements the technology transfer model proposed by Gorschek et al. (Gorschek et al., 2006), which defines seven stages for transitioning research into industrial practice. LRI focuses on how to engage in one of the most critical stages: “Problem Formulation”. LRI refines this stage with a structured and transparent process. It includes practical recommendations (Garousi et al., 2016) to promote early alignment between academia and industry. The approach actively involves SE researchers and practitioners from the beginning. However, it acknowledges that engaging practitioners is challenging. By promoting this collaborative approach, LRI aims to enhance the relevance and applicability of SE research, fostering a more effective exchange between academia and industry.

The LRI framework comprises five sequential phases, as illustrated in Figure 1. We created an interactive board template that can be used to guide these phases and be filled out collaboratively in a shared workspace. Our open science repository (Fernandes Pereira et al., 2025) provides a blank template that allows visualizing the framework in detail and a complete example filled out based on a published research paper.

Phase 1 - Problem Vision Outline: SE researchers work collaboratively to create an initial draft of the practical problem by filling attributes of the “Problem Vision” board (problem outline (what/how/why), context (where/when), implications/impacts (why), practitioners (who), evidence (how), objective (what/how), and research questions (what)). The attributes were selected based on previous research on relevance in SE, also considering their support in the broader literature on SE, especially regarding their role in problem formulation (Garousi et al., 2020; Gorschek et al., 2006; Ivarsson and Gorschek, 2011; Petersen et al., 2024).

Phase 2 - Problem Vision Alignment: Practitioners join the SE researchers in a collaborative workshop. After introductions, the researchers present the initial “Problem Vision”. The participants then review, discuss, and refine its seven attributes to ensure a shared understanding and improve the problem formulation.

Phase 3 - Research Problem Formulation: Researchers and practitioners refine and document the research problem based on the discussions in Phase 2. The formulated research problem serves as the basis for the assessment in the next phase.

Phase 4 - Research Problem Assessment: Participants individually assess the relevance of the formulated research problem using a semantic differential scale (Heise, 1970). Adapted to LRI, this scale assesses three key criteria of practical relevance in SE, aligned with agile principles and the concept of a Minimum Viable Product (MVP) (Ries, 2011). They are: “Worthless - Valuable”, which assesses whether solving the problem can generate meaningful impact for industrial practice; “Infeasible - Feasible”, which assesses whether the problem can realistically be investigated given available resources; and “Inapplicable - Applicable”, which assesses if the problem leads to practical and usable outcomes for the industry. Each criterion is rated on a seven-point scale, enabling a detailed analysis.

Phase 5 - Go/Pivot/Abort Decision: In this final phase, the group assessment results are consolidated. Aligned with the MVP-approach (Ries, 2011), LRI seeks to ensure that the research problem is relevant (valuable, feasible, and applicable) to real-world SE scenarios. Similarly to MVPs, if the perceived relevance is high, the investigation should continue (e.g., to the ‘Candidate Solution’ stage of the Gorschek et al.’s model (Gorschek et al., 2006)). If the perceived relevance is low but adjustable, the research problem should be realigned (or pivoted), restarting at Phase 2. However, if the perceived relevance is low and cannot be fixed, the investigation should be aborted to prevent wasted effort on irrelevant research.

We conducted an empirical study with 42 senior SE researchers experienced in industry-academia collaboration to evaluate three criteria—valuable, feasible, and applicable—in assessing the practical relevance of research problems. As part of our initial evaluation strategy, we enacted the LRI framework within the ISERN workshop. Participants were introduced to a pre-filled “Problem Vision” board, based on a published research problem (Cabral et al., 2024), to explore its structure and attributes (Phase 1). They then discussed and refined the problem collaboratively (Phases 2 and 3), followed by an individual assessment using the semantic differential scale (Phase 4). The exercise concluded with the consolidation of perceptions and submission of individual feedback via a survey (Phase 5).

This study offers a complete view of the LRI framework, with a specific focus on Phase 4, which evaluates the practical relevance of formulated research problems. The analysis is based on data collected from survey responses on the perceived importance and completeness of the three evaluation criteria. While this represents a small-scale evaluation rather than a full case study (Wohlin and Rainer, 2022), this approach tends to be well-suited for the initial assessment of the prospective value of a new proposal (Wohlin and Rainer, 2022). We adhered to the case study reporting structure recommended by Runeson et al.(Runeson et al., 2012).

Quantitative results showed high levels of agreement (slightly agree and agree) on the importance of the criteria, especially for valuable (83.3%), followed by feasible (76.2%) and applicable (73.8%). Hence, our initial results highlight the importance of the semantic differential scale criteria (valuable, feasible, applicable) for an initial assessment of research relevance. Qualitative feedback refined these findings, suggesting improvements in terminology definitions. For instance, the participants suggested a clearer distinction between feasible and applicable. This could be solved by better explaining that feasibility takes the research perspective, while applicability considers the industry perspective.

Participants suggested refinements rather than entirely new criteria, focusing on aspects of valuable (e.g., business value, measurable benefits, originality, ROI, problem importance), feasible (e.g., costs, limitations), and applicable (e.g., applicability requirements, intention to use). These suggestions reflect a need to clarify value, add contextual detail for feasibility, and define measurable applicability criteria. However, as noted by some participants, evaluating these dimensions precisely during problem formulation is difficult. Thus, we consider the MVP-inspired assessment scale suitable for early-stage evaluation, enabling flexible, collective judgment before more detailed metrics are applied.

The contribution of this work complements related work focusing on assessing the practical relevance of research problems in SE. While Garousi et al. (Garousi et al., 2020) highlight gaps in practical relevance and Winters (Winters, 2024) criticizes the limited applicability of academic studies, LRI addresses these concerns by integrating practitioners early in the process of problem formulation and relevance assessment. In line with the need of assessing relevance described by Ivarsson et al. (Ivarsson and Gorschek, 2011), and differently from the more complete reasoning framework to assess industrial relevance (Petersen et al., 2024), inspired by the desired properties of MVP (Ries, 2011), LRI defines a simple semantic differential scale with three main criteria –valuable, feasible and applicable– that can be used to support an initial collective assessment of the relevance of formulated research problems.

We addressed the four categories of validity threats described by Wohlin et al. (Wohlin et al., 2024):

Internal Validity: To mitigate threats to internal validity, we randomly assigned participants to groups, and each participant individually applied the assessment scale for the selected case before answering the survey questions on the assessment criteria. Furthermore, the survey provided open-ended questions for justification and allowed participants to suggest additional criteria.

External Validity: Conducting the study with a group of participants that may limit generalizability beyond (senior) SE researchers. However, we argue that ISERN members, as per ISERN’s mission, are expected to be experts in empirical SE and industry-academia collaboration, and thus render this group highly relevant for this study. Hence, we consider these emerging results to be useful initial indications. Nevertheless, future research should involve industry stakeholders for broader generalizability.

Construct Validity: The Likert-type scale may not have fully captured perceptions, and interpretations of the criteria could vary. We addressed this by including open-ended questions, carefully reviewing workshop materials, and conducting a pilot study with master and Ph.D. students to refine study instruments.

Conclusion Validity: The sample size (42 participants) and reliance on Likert-type scale responses limit inferential statistics. To strengthen the findings, we triangulated quantitative data with qualitative analysis, ensuring a more comprehensive interpretation.

This study introduced the Lean Research Inception (LRI) framework as a structured approach to support the formulation and initial assessment of practically relevant research problems in SE. We evaluated the three criteria of LRI’s Semantic Differential Scale in terms of importance and completeness to support this initial assessment. Our findings indicate an overall agreement on the importance of the three criteria – valuable (83.3%), feasible (76.2%), and applicable (73.8%) – in aligning research problems with industrial needs. Qualitative feedback suggested refining terminology to better distinguish between feasible and applicable and expanding valuable to encompass business value, ROI, and originality.

Although LRI remains an ongoing research effort requiring further evaluation, these emerging results indicate that early practitioner involvement and the use of the semantic differential scale offer a practical approach for the initial assessment of the relevance of SE research problems. Future research could refine the terminology of the scale and include practical examples for each dimension. In addition, more industry-driven case studies are needed to evaluate the effectiveness of LRI in real-world collaboration contexts.