Application-level Benchmarking of Quantum Computers using Nonlocal Game Strategies

Running simple instances of quantum algorithms with a provable advantage is difficult given the current state of quantum hardware [1, 2]. For this reason, it is important to develop benchmarking tools and techniques that can test and validate the unique aspects of quantum hardware that are consistent with the predictions of quantum theory. In particular, recent work on quantum benchmarking has highlighted the importance of developing benchmarking metrics that can measure progress toward quantum utility of useful quantum algorithms [3].

Low-level benchmark metrics such as randomized benchmarking [4, 5, 6] aim to measure the average error rates of a gate set independent of the initial state or measurement scheme, but are limited, for example, in that it cannot help specify sources of error in an algorithmic pipeline [7] and can overestimate gate fidelity in the presence of errors [8]. High-level benchmarks such as Quantum Volume [9] aim to measure the performance of the entire quantum computing stack, including all classical control systems, but this can be too broad a metric and does not necessarily capture the performance of useful quantum algorithms. In addition, both of these benchmark metrics are difficult to compute at scale and fail to capture the ability of a specific hardware platform at attaining some quantum advantage.

Recent work on nonlocal games has begun to shed light into their utility for quantum hardware verification, quantum advantage, and self-testing [10, 11, 12, 13, 14]. In a nonlocal game, two noncommunicating players cooperate to convince a referee that they possess a strategy that does not violate the rules of a game. When players are allowed to use entanglement as a resource in the development of their joint strategy, they are able to perform computations that no classical computer can replicate without communication and can win the game with higher probability. Nonlocal games have been historically important and provide a unique setting to explore the relationship between classical physics, quantum theory, and other non-signaling theories [15, 16, 17, 18]. An extensive body of research links these games to foundational problems in quantum physics, conjectures in operator algebras, and computational complexity theory [19, 20, 21]. Moreover, advances in quantum information theory and combinatorics have revealed broad classes of games with a provable quantum advantage when players are allowed to incorporate quantum resources into their strategies, such as graph coloring and graph homomorphism games [22, 23], making them exciting experimental candidates for testing quantum hardware [24]. Moreover, nonlocal games are classically verifiable, i.e. given a strategy, you can check in polynomial time if the answers satisfy the rules of the game.

Despite many breakthroughs in our theoretical understanding of nonlocal games, constructing optimal strategies for general nonlocal games remains a challenge. In our work, we propose a new methodology for constructing strategies using variational methods and outline the utility of the strategies found for benchmarking. We begin Section 2 with an introduction to nonlocal games and some definitions. In Section 3, we propose the use of a dual-phase optimization technique to find the resource state and the measurement scheme of a quantum strategies for a nonlocal game. In Section 4, we demonstrate how our method is able to successfully find optimal strategies for CHSH, an N-Partite Symmetric game, and the graph coloring game. For the graph coloring game, we were able to find a short-depth perfect quantum strategy for a graph on $14$ vertices shown to be the smallest graph instance where there exists a strict separation between classical and quantum strategies [25, 26]. We then proceed to test the performance of this novel short-depth strategy on $14$ superconducting quantum computing devices and highlight some potential sources of errors causing decreased performance on some of the devices we tested. In Section 5, we outline how we can use quantum strategies to benchmark quantum devices, their desirable noise robustness properties, and win rate estimation procedure in the presence of device shot noise.

A nonlocal game of $N$ players $\mathcal{G}=(Q_{1},...,Q_{N},A_{1},...,A_{N},\lambda)$ (illustrated in Fig. 1) consists of a set of possible questions $Q_{j}$ that player $j$ receives from a referee and a set of answers $A_{j}$ that player $j$ is allowed to send to the referee, which the referee then evaluates against a rule function $\lambda:Q_{1}\times...\times Q_{N}\times A_{1}\times...\times A_{N}\to\{0,1\}$. Each set of questions $Q_{j}$ and the set of answers $A_{j}$ has cardinality $m_{j}$ and $k_{j}$, respectively; however, in our work, we assume that there are $m$ questions and $k$ answers for each player. The game proceeds in the following steps:

1. 1.

   The players are informed about the rules of the game $\lambda$, and can collaborate to create a joint strategy, modeled as a conditional probability density between the questions and answers, to maximize their chances of satisfying the rules of the game before it starts.

2. 2.

   Players are then separated or isolated to prevent them from communicating. This is referred to as non-signaling, or in other words, each player’s actions are independent of each other.

3. 3.

   The referee tests the strategy by asking questions to each player $\mathbf{q}=[q_{1},...,q_{N}]$ and receiving their responses $\mathbf{a}=[a_{1},...,a_{N}]$, where $q_{i}\in Q_{i}$ and $a_{i}\in A_{i}$, respectively.

4. 4.

   The players win a round if $\lambda(\mathbf{a}|\mathbf{q})=1$, and lose if $\lambda(\mathbf{a}|\mathbf{q})=0$. Multiple rounds are played with different questions to establish that players have a valid strategy.

It is common that all players share the same set of possible questions $Q$ and answers $A$. In particular, Synchronous games have rules that require that the answers of two (or more) players be identical when asked the same questions $\lambda(a_{1},\dots,a_{i},a_{j},\dots,a_{N}|\tilde{\mathbf{q}})=0$, for all $a_{i}\neq a_{j}$, where $\tilde{\mathbf{q}}$ is a vector of questions. In our work, we only consider computing strategies for synchronous games, although the optimization procedure we propose in Section 3 applies for more general strategies.

Using the rules, we can define the value of the game as

where the sum is taken over all possible values of $q\in Q^{N}$ and $a\in A^{N}$ (we drop the vector notation for convenience). The distribution $p(q)$ of the questions asked is typically chosen to be uniform, and the behavior $p(a|q)$ is determined by the strategy that the players construct. Notice that this is the only term that players can control to maximize their win rate. A strategy is said to be perfect if $\lambda(a|q)=0\implies p(a|q)=0$ and, consequently, $\omega(\mathcal{G})=1$.

Classical strategies consist of a lookup table that indexes each player’s response to a particular question. It suffices to consider deterministic strategies since stochastic strategies involving shared randomness between the players cannot outperform deterministic strategies due to the linearity of the value of the game [17].

Suppose that players share a quantum state $\ket{\psi}\in\otimes_{i}\mathcal{H}_{i}$, and each player has a set of positive operator-valued measures (POVMs) with elements of the form $\{\mathcal{M}_{a|q}\}$, which they perform on their subspace. Using this setup, players can generate the following conditional probability density,

where $\rho=|\psi\rangle\langle\psi|$. These densities are known as quantum strategies.

In addition to the above definition of a quantum strategy, there are a variety of competing definitions for a quantum strategy depending on the choice of axioms that describe how joint measurements between two parties should be performed [27]. In our work, we will only consider strategies as defined above, but the study of quantum strategies is a very active area of research [28, 29, 30, 31]. Note that in [32], the authors prove that for synchronous games, a maximally entangled state is sufficient for a quantum strategy to win the graph coloring game.

A game exhibits a quantum advantage if there exists a quantum strategy that performs better than the best possible classical strategy, in which case there is a Bell inequality $\mathcal{I}$ that is violated for some quantum strategies. The inequality has historically served as an experimental test of local realism [33]. Such inequalities are constraints satisfied by classical (local hidden-variable) models, and are often linear inequalities derived from the local realism assumption. More specifically, a Bell inequality consists of a function $\mathcal{I}$ with respect to the probabilities $\{p(a|q)\}$ such that

for some $\xi\in\mathbb{R}$. Bell inequalities are a central object for self-testing of states [12]. The construction of such functions $\mathcal{I}$ and constants $\xi$ are as follows: for a given Bell inequality $\mathcal{I}=\sum\limits_{q,a}w_{q,a}p(a|q)$, where $w_{q,a}$ are weights, there is a corresponding Bell operator $\mathcal{B}=\sum\limits_{q,a}w_{q,a}\bigotimes\limits_{i}\mathcal{M}_{a_{i}|q_{i}}$, such that a violation is obtained as $\xi=\mathrm{Tr}(\mathcal{B}\rho)$. If the maximal achievable violation is obtained by using quantum resources, denote $\xi_{Q}$ for this distinction and consider the shifted Bell operator $\xi_{Q}\mathbbm{1}-\mathcal{B}$. If the shifted Bell operator admits a decomposition

where each $P_{\gamma}$ is a polynomial with respect to the measurement operators $\{\mathcal{M}_{a_{i}|q_{i}}\}$, then we call the decomposition a sum of squares (SOS) for the Bell inequality. Such a decomposition is extremely hard to find [34].

In this section, we present a variational quantum algorithm for computing quantum strategies of nonlocal games. Let $\ket{\psi}$ be the shared entangled state between the players and $\mathcal{M}_{a|q}=\bigotimes_{i}\mathcal{M}_{a_{i}|q_{i}}$ be the joint POVM applied to that state for question $q$, returning $a$ with probability $p(a|q)=\langle\mathcal{M}_{a|q}\rangle$. It was noted in [35] that fixing these measurement operators gives a Hamiltonian whose ground state is the optimal shared state for this measurement setting. This fact has been used with reinforcement learning to optimize measurements while selecting the optimal shared state through exact diagonalization [36].

To evaluate the performance of DPO, we apply it to several nonlocal games with known quantum bounds: CHSH, the N-partite symmetric (NPS) game [36], and the odd-cycle game [14]. Then, we use DPO to explicitly compute an optimal quantum strategy for the coloring game of a $14$ vertex graph called $G_{14}$ [25]. This strategy is then evaluated on quantum hardware, demonstrating that it can be used to benchmark the nonlocal capabilities of quantum devices and find sources of errors.

Nonlocal games with perfect strategies can serve as hardware benchmarks by assessing and analyzing the empirical win rates when executed on near-term hardware. Under certain assumptions about the structure of quantum noise, nonlocal games can exhibit quantum advantage in shallow circuits, even with noisy qubits [43]. The ‘noisy entanglement’ generated in shallow circuits enables correlations that classical circuits fundamentally struggle to reproduce. This is seen in [43]: their classical circuits of constant depth cannot simulate the long-range correlations.

In this section, we demonstrate the effectiveness of this benchmark by analyzing hardware noise and its strong correlation to strategy performance. We proceed backwards,from the unentangled readout measurements, to the independent Bell state measurements, to the initial entangled resource state preparation. By investigating the effects of hardware noise on the empirical win rates we seek to establish: a) which questions are most affected by noise, b) which components of the circuit are most affected by noise, and finally, if classical correlations, or quantum noise, could be misinterpreted as a winning quantum strategy.

In addition to the classical bounds provided in Section 4.4, we also consider the worst outcome on hardware: a nearly uniform distribution over all bitstrings. This would skew the win rates in the $G_{14}$ game as follows: for any vertex question would only be $1/4=0.25$ and the average win rate of any edge question would be $12/16=0.75$. Thus random guessing would return an overall win rate of 59%. In Figures 11, 12, 13 and 14 we include these values as a reference.

Quantum hardware is affected by many sources of noise. The noise profile is time dependent and there are many strategies developed to optimize the scheduling and execution of quantum circuits. Using superconducting qubit platforms from IBM and Rigetti, we investigate the robustness of the original $G_{14}$ strategy, and the Bell pair strategy on superconducting qubit platforms with respect to hardware noise fluctuations over several days, and also to changes in the circuit made during the transpiration step.

On near-term quantum devices, the win rate of each circuit (question) is estimated by statistical sampling, using independently drawn samples to estimate the probability that the players return the correct answers. Finite sample effects lead to statistical fluctuations. In this section, we derive an upper bound on the number of individual samples (shots) to draw from a prepared state to sufficiently assess whether a circuit has correctly answered the referee’s question.

In the interactive nonlocal game setup, the scenario is repeated with random questions until the referee is satisfied with the outcome. We consider how to obtain a low error estimate of the win rate with high probability using a finite number of repetitions.

Let $n$ be the number of rounds performed, where each round consists of the referee asking the players all $m$ possible questions once and checking their answers using the rule function $\lambda(a|q)$. In the context of quantum hardware, this can be viewed as the execution of $m$ quantum circuits with $n$ shots per circuit.

Because the outcome of each question is binary, i.e., $\lambda(a|q)\in\{0,1\}$, we model the outcome of question $q_{j}$ as a Bernoulli random variable $\lambda_{j}$ with an unknown success probability $p_{j}$. The random variable describing the game value of a single round is $\omega=\frac{1}{m}\sum_{j}^{m}\lambda_{j}$. We denote the empirical estimate of the win rate with $n$ rounds as $\bar{\omega}=\frac{1}{n}\sum_{i}^{n}\omega_{i}$ where $\omega_{1},\dots,\omega_{n}\sim\omega$ are i.i.d. samples. Under these mild assumptions, we derive an expression for the number of samples needed to accurately estimate the win rate within error $\epsilon$.

From (20), we see two possibilities to achieve asymptotic $O(\log(1/\delta)/\epsilon)$ sampling: with a large number of questions $m$ and when $\bar{\sigma}\simeq\epsilon$. The first case is not practical because increasing the number of questions counterproductively increases the total number of circuit samples $mn$.

The second case is also hard to achieve (at present) because it requires a near-perfect strategy on high-fidelity quantum hardware. This results from $\bar{\sigma}$ being directly linked to the win rate of the questions, which depends on both the strategy and quantum hardware. Assuming all questions have equal win rate $p$ for simplicity, this requires (again taking the positive solution) $p\approx\frac{1}{2}(1+\sqrt{1-4\epsilon^{2}})$, which is approximately $p\approx 1-2\epsilon+O(\epsilon^{2})$. We expect that $O(\log(1/\delta)/\epsilon)$ sampling may become feasible for perfect strategies with improved gate fidelity, quantum error correction, or amplitude amplification. Table 2 contains all the win rates of our executed experiments with confidence intervals derived from Corollary 6.3.

We present a variational algorithm to compute novel quantum strategies for nonlocal games by encoding the rules of a nonlocal game into a Hamiltonian and employing a two-step optimization procedure. Our key insight is to optimize separately the state preparation circuit and the measurement scheme while leveraging robust circuit initialization and general techniques, such as ADAPT, during optimization. The proposed algorithm successfully reproduces known quantum strategies and has also discovered new short-depth, perfect quantum strategies for a graph on $14$ vertices using four qubits. This demonstrates that variational techniques can be effectively used on classical computers to identify short-depth, optimal strategies for small examples of nonlocal games where analytic methods fail. Moreover, these techniques extend to a quantum setting, where sample-based gradient estimation is employed. However, the presence of barren plateaus is a known challenge with the training objective function, suggesting that “warm starts” or other techniques to mitigate vanishing gradients may be necessary for scaling these methods to larger nonlocal games.

We further illustrate how the execution of a nonlocal game strategy can serve as an application-level benchmark for quantum devices. By evaluating the win rates of both vertex and edge questions in these games, the win rate of vertex questions reflects a device’s ability to perform nonlocal operations and maintain gate fidelity, while the win rate of edge questions can help confirm the utilization of entanglement across a device. Although none of the devices we tested surpassed the quantum advantage threshold, primarily due to noise in circuit execution, we believe our results can be improved by optimizing the transpilation of the individual circuit before execution and control of the device calibration schedules. It is also worth noting that although our experiments do not provide a full proof of quantum advantage, given that the particles are not spatially separated enough to guarantee that classical communication does not happen during the experiment, it does provide validation that the quantum hardware in question outputs results consistent with the hypotheses of quantum theory. Recent work has begun to outline ways of guaranteeing a “loop-hole free” full verification of quantum advantage by compiling a multi-prover nonlocal game strategy into a single prover strategy [50, 10, 11] and we leave it to future work to investigate the feasibility and implications of these schemes for the games we studied. In a recent survey [51], the authors outlined five desirable properties for a good quantum benchmark and in our work we argued how the win rate from nonlocal game strategies fit all five points:

• •

  Relevant: The win rate measures the ability to prepare, control, and manipulate entangled states.

• •

  Reproducible: Strategy and questions are fixed.

• •

  Fair: Device independent and the executed circuits are shallow.

• •

  Verifiable: Straightforward to calculate the win rate via sampling.

• •

  Usable: Circuits can be made accessible via QASM files and can easily be ported to other quantum devices.

We believe that the continued study and extensions of nonlocal games, in particular graph-based games, can enable the design of more appropriate quantum benchmarks as quantum devices scale and hardware architectures become more complex. Ultimately, our research not only advances the understanding of variational quantum strategies but also lays the foundation for leveraging quantum machine learning techniques to explore other nonlocal games strategies beyond the reach of classical methods.

Thanks to David Roberson and Eleanor Rieffel for providing valuable feedback. NW, JF, and COM were funded by grants from the US Department of Energy, Office of Science, National Quantum Information Science Research Centers, Co-Design Center for Quantum Advantage under contract number DE-SC0012704. JF and COM were partially supported by the Laboratory Directed Research and Development Program and Mathematics for Artificial Reasoning for Scientific Discovery investment at the Pacific Northwest National Laboratory, a multiprogram national laboratory operated by Battelle for the U.S. Department of Energy under Contract DEAC05- 76RLO1830. S. C. is supported in part by the DOE Advanced Scientific Computing Research (ASCR) Accelerated Research in Quantum Computing (ARQC) Program under field work proposal ERKJ354. K. H. was supported by the DOE Advanced Scientific Computing Research (ASCR) Pathfinder Testbed Program under FWP ERKJ418.

This research used resources of the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725.

This manuscript has been authored in part by UT-Battelle, LLC, under Contract No. DE-AC0500OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for the United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan.