DISCD: Distributed Lossy Semantic Communication for Logical Deduction of Hypothesis

We begin by illustrating how the state of the world can be represented using First-Order Logic (FOL). Consider a finite world $\mathbf{W}_{F}$ characterized by a finite set of entities $\mathbf{E}$ and a finite set of predicates $\mathbf{P}$. The set of all possible states of $\mathbf{W}_{F}$ is denoted by $\mathbf{S}_{F}=\{\mathbf{s}_{1},\mathbf{s}_{2},\dots,\mathbf{s}_{n}\}$, where $n=2^{|\mathbf{P}|\times|\mathbf{E}|^{2}}$. Each state $\mathbf{s}_{i}$ provides a complete description of the world using the logical language $\mathcal{L}$, which includes the predicates $\mathbf{P}$, entities $\mathbf{E}$, quantifiers $\forall,\exists$, and logical connectives $\land,\lor,\neg$.

In this setup, each state is a conjunction ($\bigwedge$) of a specific enumeration of all possible predicates (or their negations), $R_{r}\text{ for }r\in\mathcal{I}_{P}$, applied to all ordered pairs of entities, $(a,b)\in\mathbf{E}$. Any FOL sentence $m$ in this finite world can thus be expressed as a disjunction of these state descriptions.

When extending to an infinite world $\mathbf{W}_{I}$ where entities $\mathbf{E}$ becomes a countably infinite set, a more intricate approach is needed. We introduce the concept of Q-sentences to handle this complexity.

Building upon Q-sentences, we define attributive constituents to capture the relationships of individual entities.

To represent the entire infinite world, we define constituents as combinations of attributive constituents.

Any FOL sentence $m$ in $\mathbf{W}_{I}$ can then be expressed as a disjunction of such constituents.

As an illustration, consider the statement ”Every person owns a book.”, or more explicitly, ”For every x, if x is a person, then there exists a y such that y is a book and x owns y.”, formally expressed as:

ensuring that all entities are accounted for. Therefore, $m$ can be expressed as:

State descriptions and constituents act like basis vectors in a vector space (as they partition the space and hence any two are mutually exclusive), allowing us to define a probability measure over them. The inductive logical probability of any sentence in the FOL language $\mathcal{L}$ is then the sum of the probabilities of its constituents as per axioms of probability. In the next section, we will discuss how to define this probability measure using the frameworks proposed by Carnap and Hintikka.

In FOL-based world $\mathbf{W}\text{ (could be }\mathbf{W}_{F}\text{ or }\mathbf{W}_{I})$, we introduce a probability measure $\mathcal{P}_{I}$ over the fundamental elements—either state descriptions $\mathbf{s}_{i}$ or constituents $C^{j}$, collectively referred to as ”world states” $S_{i}$. Let $S_{i}\in\mathcal{S}$ where $\mathcal{S}$ is the state space. To define this measure, we make use of inductive logical probabilities, which serve as inductive Bayesian posteriors, built upon empirical observations and a prior distribution.

Then, the inductive logical probability measure $\mathcal{P}_{I}$ over $\mathcal{S}$ is defined as $\mathcal{P}_{I}=\{c(S_{i},e)\mid i\in\mathcal{I}_{\mathcal{S}}\}$ where $\sum_{i\in\mathcal{I}_{\mathcal{S}}}c(S_{i},e)=1$. In the following section, we make use of inductive logical probabilities to define a semantic information metric to effectively quantify semantic informativeness regarding states of the world.

To quantify the remaining uncertainty after new observations, Carnap proposed the concept of ”cont-information”. This metric assesses how observations influence an observer’s understanding of the SotW by evaluating the extent to which they decrease observer’s uncertainty.

In simple terms $\text{cont}(S_{i};e)$ measures the uncertainty remaining about state $S_{i}$ after observing e. Next, the cont-information measure is used to devise a communication framework. This measure captures the dynamic and cumulative nature of evidence accumulation in the communication and, as a result, the refinement in an observer’s understanding of the world state.

We start by formalizing a hypothesis deduction problem at each user node $N_{j}$ in the wireless network. Let $\mathcal{H}_{j}=\{h_{i}\mid i\in\mathcal{I}_{\mathcal{H}}\}$ be a finite or infinite set of self-consistent statements representing possible hypotheses about the world $\mathbf{W}$ at node $N_{j}$. There exists a hypothesis $h^{*}\in\mathcal{H}$ which is most consistent, among all other hypothesis, with the actual SotW. The statements $h_{i}$ can be represented by a non-empty disjunction of state descriptions or constituents, as per Theorem 1 and 2. At each node $N_{j}$, the task is to deduce the hypothesis $h_{j}^{*}$ (the most consistent hypothesis with the SotW). However, as each node has incomplete information, this deduction cannot be concluded reliably. The aim of the communication is, then, to receive most cont-informative information regarding world state so as more accurately determine which hypothesis is supported the most by world (entity) population.

As shown in Fig.1, we consider a distributed network comprising a set of edge nodes $\mathcal{N}=\{N_{1},N_{2},\dots,N_{v}\}$ and a central server $CS$. Each edge node possesses evidence $e_{j}$ obtained through observations of the world $\mathbf{W}$, which can be finite ($\mathbf{W}_{F}$) or infinite ($\mathbf{W}_{I}$). Each node $N_{j}$ maintains a local inductive probability distribution $\mathcal{P}^{j}_{I,t}$ over the state space $\mathcal{S}$, reflecting its view of the State of the World (SotW) at round $t$. Each node $N_{j}$ aims to solve a hypothesis deduction problem $\mathcal{B}_{j}$, involving determining the hypothesis $h_{j}^{*}$ most supported by evidence from a set of possible hypotheses. Since the nodes have incomplete observations about SotW, they wish to collaborate with each other through a central server to share some information with each other to improve each node’s local distribution over states. However, due to communication constraints, every node can transmit only a limited amount of information to the server in each communication round. The central server $CS$ aggregates messages from all nodes to update its own perception of the SotW, denoted by $\mathcal{P}^{CS}_{I,t}$. The server then selects a message and broadcasts it back to the nodes to aid them in refining their local distributions. In the next section, we turn our attention to, for each round t of communication: (i) how a node chooses its message for transmission to the server, (ii) how the server updates its perceived SotW upon receiving all incoming messages from the nodes, (iii) how the server selects which message to broadcast to all nodes, (iv) how a node updates it’s perceived SotW upon receiving the message from the server, and (v) how the hypothesis deduction problem is solved.

The communication occurs in iterative rounds, each consisting of two phases: an uplink phase and a downlink phase.

In this section, we examine the convergence behavior of the communication framework under finite evidence to determine the relationship between empirical and true distribution over state space $\mathcal{S}$. To preserve generality, we assume an infinite universe $\mathbf{W}_{I}$ and conduct the analysis over constituents.

Let $e_{j,l}^{c}$ describe a finite sample of $l$ individual observations, containing $c$ distinct attributive constituents accumulated at node $N_{j}$ after $T$ rounds of communication. A constituent $C^{w}$ of width $w$ is compatible with $e_{j,l}^{c}$ only if $c\leq w\leq K$, where $K=4^{|P|\times|E|^{2}}/2$.

To compute the posterior distribution

one needs to specify the prior $c(C^{w},\emptyset)$ and the likelihood $c(e_{j,l}^{c},C^{w})$. A natural method to assign the prior $c(C^{w},\emptyset)=P(C^{w})$ is to assume it is proportional to $\left(\frac{w}{K}\right)^{\alpha}$, reflecting the assumption that there are $\alpha$ individuals compatible with a particular constituent $C^{w}$. This approach captures the intuition that constituents with larger widths (i.e., involving more attributive constituents) are more probable if we believe there are more individuals exemplifying them. The explicit equation for the prior $P(C^{w})$ is:

where $\Gamma(\cdot)$ is the gamma function, and $\alpha\geq 0$ is a parameter reflecting our prior belief about the number of individuals compatible with $C^{w}$.

The likelihood $P(e_{j,l}^{c}\mid C^{w})$ is computed based on the number $l_{z}$ of observed samples confirming each attributive constituent $C_{z}(x)$ in the evidence $e_{j,l}^{c}$. It is given by:

where $\lambda$ is a prior coefficient, possibly dependent on $w$. This likelihood reflects how well constituent $C^{w}$ explains the observed evidence $e_{j,l}^{c}$.

Using the prior and likelihood, the posterior probability of constituent $C^{w}$ given $e_{j,l}^{c}$ is:

where the sum in the denominator runs over all constituents compatible with the language and $\mathcal{I}_{\mathcal{C}}$ is the index set of constituents.

Using the prior in equation (16) and the likelihood in equation (17), we can analyze the convergence of the posterior probability $P(C^{w}\mid e_{j,l}^{c})$ as the number of observations $l$ increases. Specifically, these equations are used to derive a Probably Approximately Correct (PAC) bound that determines the minimum number of samples required to consider the minimal constituent $C^{c}$ to be approximately correct with high probability. (Despite omitted from this manuscript due to space concerns, it had been proven that asymptotically, the constituent with smallest width, a.k.a. the minimal constituent, receives probability 1 while all other constituents receive probability 0. See [31] for further discussion and proof.)

This PAC bound provides a theoretical guarantee on the convergence of the posterior distribution $\mathcal{P}^{j}_{I,T}$ to true distribution on the state space $\mathcal{S}$ as more evidence is accumulated. The detailed derivation of this bound relies on the prior and likelihood expressions, as also discussed in the next section. Now, we turn our attention to showing that DISCD is superior compared to random transmissions.

In this section, we compare two communication strategies under sentence level communication constraints, from the perspective of a node $N_{j}$, as detailed by the next theorem.

Elaborating on Theorem 4, it is observed that evidence $e_{j,cont}$ affects the posterior in two ways. It provides more specific information and as the evidence is more informative (larger $l$, larger $l_{w}$) and the likelihood function $P(e_{j,cont}\mid C^{w})$ will often be higher for constituents that align closely with the evidence and lower for others. Also, $e_{j,cont}$ may render some constituents impossible (assigning them a likelihood of zero) because they are inconsistent with the evidence.

Under $e_{j,rand}$, the likelihoods $P(e_{j,rand}\mid C^{w})$ are less discriminative because random evidence provides less specific information. The differences in likelihoods between the true constituent and incorrect ones are smaller. As the likelihoods under $e_{j,cont}$ are more discriminative, the posterior distribution $P(C^{w}\mid e_{j,cont})$ is more peaked around the true constituent $C^{c}$, assigning higher probability to $C^{c}$ and lower probabilities to others.

In other words, $e_{j,cont}$ improves discrimination. By providing more specific observations, evidence $e_{j,cont}$ with high degree of confirmation enhances the differences in likelihoods between the true constituent $C^{c}$ and incorrect constituents. The increased likelihood $P(e_{j,cont}\mid C^{c})$ and decreased likelihoods $P(e_{j,cont}\mid C^{w}\neq C^{c})$ result in a higher posterior probability for $C^{c}$. The posterior distribution over constituents becomes more concentrated around $C^{c}$, assigning lower probabilities to incorrect constituents and leading to a more accurate posterior. The improved likelihood ratios lead to a tighter PAC bound, reflecting a better estimation of $C^{c}$. In the next section, we comment on the impact of improved posterior estimation on hypothesis deduction task success.

Albeit discussions in this section are for a single node $N_{j}$, they can be trivially extended to multi node case. Each node $N_{j}$ aims to minimize its risk (expected loss) $R_{\mathcal{P}^{j}_{I,T}}(h_{j})$ over $\mathcal{B}_{j}$:

where $L(h_{j},h_{j}^{*})$ quantifies the discrepancy between $h_{j}$ and $h_{j}^{*}$ and can be selected as any appropriate distance metric.

As nodes exchange information and receive informative messages, the distributions $\mathcal{P}^{j}_{I,T}$ converge towards $\mathcal{P}^{j*}_{I,T}$ (true distribution), reducing expected loss and enhancing their ability to identify $h_{j}^{*}$, as $h_{j}\equiv\bigvee_{\begin{subarray}{c}w\in\mathcal{I}_{\mathcal{C}}\\ \text{such that }h_{j}\models C^{w}\end{subarray}}C^{w}$ and $c(h_{j},e_{j}\land\bigcup_{z=1}^{T}m_{cs,z})=\sum_{w}c(C^{w},e_{j}\land\bigcup_{z=1}^{T}m_{cs,z})$.

Therefore, as evidence accumulates and as per Theorems 3 & 4, $e_{j,cont}$ leads to tighter converge towards the true (minimal) constituent $C^{c}$, it better minimizes the Bayes risk $R_{\mathcal{P}^{j}_{I,T}}(h_{j})$ compared to random selection of messages $e_{j,rand}$ (e.g., $R_{\mathcal{P}^{j,cont}_{I,T}}(h_{j})\leq R_{\mathcal{P}^{j,rand}_{I,T}}(h_{j})$), yielding larger hypothesis deduction success rate. In the next section, we experimentally verify these theoretical observations.