The PETSc Community Is the Infrastructure

PETSc serves many purposes as a software library that connects research in applied mathematics to usage within applications in science and engineering. These include the following.

• •

  A research platform targeting cutting-edge algorithmic development

• •

  A well-supported HPC library

• •

  A repository of template applications via a wealth of example codes

• •

  A compendium of algorithms, with an algorithmic management system that provides concrete, scalable implementations of a wide range of methods described in the applied mathematics literature

• •

  An application development framework

• •

  A pedagogical tool for training numerical analysts on HPC platforms [5]

• •

  A source of best-choice numerical methods in its role as an interface between academic algorithmic development and the needs of users in science and engineering

• •

  An extensible interface to complementary HPC software, such as SuperLU and hypre

Virtually all active PETSc community members are PETSc users; a smaller subset of these, who often began as PETSc users, are also PETSc developers. All PETSc users provide important contributions, including bug reports, bug fixes, improved documentation, and suggestions for new features. Individuals often move between different roles in the PETSc community. There is a “long tail” of members who contribute less frequently than the most active developers, yet collectively contribute a great deal. Over 100 people are contributors to the PETSc Git repository.

The community structure is crucial in providing a pathway to increasing involvement for interested users. One way is to characterize PETSc community members along institutional lines.

• •

  Research-oriented users are students, faculty, and staff focusing on research and development, who typically are employed by universities and research laboratories. Students may use PETSc to do homework or develop a paper or thesis code. Students often contribute code back to PETSc, and as they graduate, bring PETSc to their new institutions. Some students have become PETSc developers.

• •

  Industrial users employ PETSc in their company’s research or commercial products. PETSc’s 2-clause BSD license eases its commercial use. These users may request support that is unlikely to be funded by research grants, such as support for Microsoft Windows; fortunately, there are avenues for PETSc community members to help with these requests. Industrial users require discretion and confidentiality. They cannot always share their use cases, so we must provide general solutions without details of the specifics. Developing the trust needed for industrial users is a slow, gradual process whose importance must be recognized by both sides.

Another way to categorize PETSc community members is by the goals of their work, as shown in Figure 1.

• •

  Algorithm developers focus on devising and analyzing algorithms and hence may be less concerned about generality and usability. They use PETSc because it provides infrastructure for HPC architectures, allowing them to avoid unnecessary coding. Algorithm developers face the challenge of writing scalable implementations, which, even in PETSc, can be time-consuming with a steep learning curve. However, such people find the benefits of using PETSc, including the broad impact of their work on HPC applications, outweigh using packages with a less steep learning curve, such as MATLAB.

• •

  Application developers focus on creating a code that addresses one type of simulation well, at a low cost. They are often discipline scientists or engineers, who benefit from performance enhancements provided through PETSc composability, where upgrades in algorithms and data structures can occur seamlessly from a user perspective, yet provide significant performance increases.

• •

  Scientific toolkit developers build systems that tackle a subset of PETSc functionality but at a higher level of abstraction, with more specific support for their target class of problems. Such toolkits, including Firedrake, MOOSE, and Deal.II, leverage PETSc capabilities and introduce additional infrastructure.

Support is a crucial aspect of a healthy software community. Members of the PETSc community usually respond to requests within hours, if not minutes. This engagement helps new users feel welcomed and valued, make rapid progress, and gain confidence. Through users’ feedback, experienced community members learn what works, what doesn’t, and where improvements are needed. PETSc community members often discover new research topics from feature requests and discussions.

However, providing excellent support is a substantial effort, particularly when users encounter difficult bugs or performance issues at scale. User-developer communication on a particular topic can span weeks or even months. PETSc developers need to be patient and consistently engaged with users. One may question whether this practice is sustainable, but it has worked reasonably well. In the last section, we discuss some technical approaches to providing support.

To maintain the vitality of the library, new algorithmic developments must be rapidly integrated, bugs promptly fixed, and awkward constructions removed. These activities require the PETSc community to establish a high level of trust, communicating that the library will be well supported even in the face of rapid evolution, and that code will continue to run with help from the community. Establishing trust is a precondition to creating a welcoming environment for new users and developers. Members of the PETSc community have a wide range of professions, backgrounds, and levels of involvement, with individuals often participating in several ways over the years. Engagement is key to disseminating tacit knowledge and developing users’ skills and social support, so that people can transition to become developers and mentors. The PETSc community has developed a broad base of people with expertise and kindness to reduce and report bugs, mentor newcomers, and contribute in other ways.

Application communities often treat PETSc as a software ecosystem instead of a stand-alone package. As a result, they rely on PETSc to manage necessary low-level tools, such as MPI, BLAS/LAPACK, and vendor packages used on accelerators. Application communities also appreciate unified solver interfaces, particularly linear preconditioners, which enable application codes to access third-party libraries such as MUMPS, SuperLU, and hypre with little effort. Often “technical language” barriers exist between communities. For example, a contact mechanics expert may describe a solver convergence issue as “we found a PETSc error when contact mechanics occurs with a frictionless model.” A numerical solvers expert might find it hard to resolve such an issue. Fortunately, other PETSc community members may have domain expertise and can serve as liaisons between communities. Such individuals speak the languages of both communities, understanding both PETSc’s capabilities and the needs of their communities. Thus, they can appropriately explore, explain, and introduce PETSc features to their communities. Such liaisons help expand PETSc’s reach across disciplines and reduce the centralized maintenance burden by addressing many questions directly in their communities while contributing patches, feature requirements, and even serving as testers of software releases.

An important PETSc subcommunity is system engineers who manage institutions’ computational infrastructure. They often are the first to encounter problems that need the attention of PETSc developers. Their expertise can help rapidly debug problems and develop fixes that need to be quickly merged into the distribution. Package maintainers, for example, for APT, are also a valuable resource, as they track PETSc on particular configurations; they often have excellent suggestions for improvements to PETSc’s configuration and installation.

Communities must be able to respond with innovative and creative solutions to changing circumstances. For numerical software, this includes the continual emergence of new science drivers and techniques, including data science and artificial intelligence, as well as new hardware architectures. For example, a large shift in HPC is currently underway with incorporating graphical processing units (GPUs) into both high-end and moderate-scale scientific computing. Major organizations such as DOE have responded with, for example, the Exascale Computing Project, where community open source projects, including PETSc, are aggressively developing innovations in data structures and algorithms for new architectures [8], [12]. The PETSc community empowers developers to be creative by providing the autonomy to be innovative while still maintaining guidelines for development [1] and town squares to organize overall development plans. This approach, along with PETSc’s wide variety of contributors, enables a level of agility that might not otherwise occur. This approach also naturally promotes project-specific planning (for example, as needed for work proposed and funded in particular grants) and coordination among development communities overall.

PETSc’s community helps members identify funding opportunities, access expertise, and transition between roles in the project. One of the greatest difficulties in maintaining a coherent software project over decades is providing career paths for contributors. PETSc gives academic contributors a solid foundation for advancement via awards (e.g., SC Gordon Bell prizes, SIAM prizes), the highly cited users’ manual, professional recognition, and productive collaborations born from PETSc development, maintenance, and support. In addition, PETSc provides resource sharing from collaborative grants and collaboration opportunities that extend beyond the development group. Sometimes, PETSc affiliation may be more important than departmental affiliation, especially since modern academic departments are often atomized, with little internal collaboration. The community provides strong academic connections for industrial and laboratory members, tangible outputs recognized by future employers, and active participation in the wider computational science community.

Even the smallest community open source projects cannot exist without some funding and institutional support. Usually, it is a combination of grants from governmental or non-governmental agencies, in-house funding within particular institutions, and less formal systems that allow employees to contribute to open source packages during a portion of their regular employment. Members of the PETSc community are actively engaged with program development, including communicating with program managers at the U.S. Department of Energy and the National Science Foundation and with management at their institutions, to ensure that support is provided and maintained. This form of interaction is crucial to the long-term viability of all open source software communities; PETSc users and community members have played important roles in various local, national, and international conversations, including recent DOE activities related to software sustainability [11, 9].

HPC software systems have complex execution environments, including great variance in hardware and software. Software developers must expend considerable effort to configure and build their code for different situations. Then, if failures occur, the software developers have to diagnose them. Configuration failures are the most common support issues the PETSc community faces. Thus, having a debuggable configuration system with comprehensive logging is critical. Rather than using a standard configuration system such as GNU Autotools or CMake, PETSc has a bespoke configuration system with extensive checking, written in Python, which logs everything during configuration in a single file configure.log. When a check fails, it generates clear error messages and a Python stack trace. PETSc users attach configure.log and another file make.log generated by make when they meet configuration errors. By examining the two files, PETSc developers can quickly determine why the configuration system made specific choices and what went wrong. In addition, since the configuration system is bespoke, the PETSc community can easily add new checking, testing, and logging. CMake is notoriously difficult to debug by email because it logs information in various directories and does not log much of its process.

The PETSc library strives to provide descriptive error messages that explain why and where errors have occurred, making it easy for PETSc developers to diagnose by email what went wrong and assist users with fixes. PETSc has extensive code to assist in this regard. See Listing 1, which shows code that adds two vectors with y += alpha*x. Every PETSc function returns a PetscErrorCode, indicating whether the function is successfully executed, and if not, what type of error occurred. In PETSc source code, every function call is error-checked, in a style similar to lines 12–14. We make errors manifest early rather than later to avoid obscure error messages. The default error handler prints the stack trace leading to the error, including function names, file names, and line numbers. The stack trace is built inside the two macros PetscFunctionBegin and PetscFunctionReturn(0); see lines 3 and 16. PETSc also provides utilities to check the integrity of function parameters; see lines 4–10. All application programming interfaces (APIs) shown here are public; users are encouraged to apply the same strategy in their code.

PETSc also has APIs to assert properties of the code so that useful error messages are generated promptly if the code behaves unexpectedly. For example, once a matrix is preallocated or assembled, one can set a property of the matrix to indicate that in subsequent insertions one will insert only to existing nonzero locations.

Memory corruption problems are common; therefore, memory allocations are done through a PETSc-specific API that records information and sets sentinels around the allocations in debug mode. With command-line options, PETSc will initialize the allocated memory with NaN (not a number); using the uninitialized memory in floating-point operations will generate an appropriate error message. Also, PETSc can check the integrity of the entire heap of PETSc-allocated memory at every allocation. PETSc codes also can output information about memory that has never been freed during the PETSc finalization stage, to detect memory leaks. As lightweight Valgrind-like features, the output can be shared with PETSc developers to help understand a code’s misbehavior. But, of course, we also recommend using more sophisticated tools, including Valgrind and debuggers.

Another challenging support task, which the PETSc community also handles routinely, is debugging performance problems, particularly for high levels of parallelism. PETSc provides APIs to set stages in their code and log the performance of events of interest. For example, lines 12 and 14 in Listing 1 are for the VEC_AXPY event, rendering a lightweight, integrated logging system that allows users to quickly gather timings. Listing 2 shows a snippet of the stdout output. From the top, we know that, besides the PETSc’s default stage (Main Stage), the test created another stage (Solve). PETSc summarizes the computation and communication statistics of the two stages. Below that, it lists detailed statistics of events (usually functions) within each stage (only stage 1 is shown here), including the number of times an event has been called, time and floating-point operations (flops) an event has spent, MPI messages and reductions an event has incurred, and other statistics. Interested readers are referred to the PETSc/TAO Users Manual [3] or hints in the log view message itself. Because MPI processes typically have different statistics in parallel, PETSc shows maxima over all processes and ratios of maxima to minima. This information is useful because whenever we encounter a large ratio in time or flops in output, we know a load imbalance in the corresponding event might exist. Sometimes, imbalance in one event can distort the timing of other events (for instance, processes might wait for messages from a lagging partner), giving confusing results.

Having the profiling tools integrated with the numerical algorithms in use, outputting by default to stdout, is crucial because it allows all users to provide quickly information on their usage, independent of what computational systems they may use or which additional analysis or logging tools they have available. This allows PETSc developers to quickly and directly view timings on the user’s system and facilitate performance debugging of scalable solvers at “production” scale, by email, where direct reproduction of a user’s issue is infeasible.

PETSc includes an extensive suite of parallel preconditioners, linear solvers, nonlinear solvers, and time integrators. Composable and nested solvers are among the most powerful PETSc features since they facilitate numerical experimentation on novel, highly complex problems, but keeping track of them can be difficult. PETSc developers must see the detailed solver configurations to spot potential problems. Hence, PETSc provides APIs to display all solver options being used. Listing 3 shows a snippet of a longer output from a nonlinear solver. With indention reflecting levels of the composite solvers, we can clearly see the nested solvers used and key parameters employed at various levels of the solvers.

PETSc provides flexible monitors to be used in conjunction with solver views, that print the residual or function norm at each iteration of an iterative solver so that users can check the convergence of the solver and compare different algorithms. Listing 4 shows output of nonlinear and linear solver monitors.

In summary, PETSc offers a wide set of complementary options to aid debugging by email with the following common themes: users can enable debugging regardless of their computing environment; errors appear as early as possible; and output is printed in well-formatted plain text for copy-and-paste or file attachments to emails and GitLab issues.