fybrrStream: A WebRTC based Efficient and Scalable P2P Live Streaming Platform

A signalling server is used to establish RTC Connection between peers. Connection establishment includes exchange of SDP (Session Description Protocol) and ICE (Interactive Connectivity Establishment) candidate. If the ICE candidate connects successfully then RTC Connection is setup for stream delivery. The SDP contains the session description of the two peers, with configuration of the media to be sent by the source peer and the response description of the receiver peer.

To setup RTC Connection, STUN servers are used to retrieve public IP of a device for initialising RTP over User Datagram Protocol (UDP) . WebRTC uses UDP for stream delivery. If the ICE candidate fails to connect due to any possible reason (if a device is behind symmetric Network Address Translation (NAT), or is protected by a firewall which discards UDP data packets), then Traversal Using Relays Around NAT (TURN) servers are used to relay the stream around NAT.

When a new RTC Connection is established between an old and a new peer, the new link extends the overlay network which actually is a tree. The predefined rules of peer selection (request-based protocol for parent assignment) protocol says that every node in a level must have at-least 2 children until the level is complete. This makes the worst case complexity for a packet to reach any node from the source $\mathcal{O}(\log{}n)$ for a total of $n$ nodes in the network.

Our proposed architecture is using WebRTC to make connections and transfer stream between the peers. Internet Connectivity Establishment (ICE) service will find the possible candidates to connect to a peer. Although we are proposing a distributed model for live streaming but maintenance of overlay topology is taken care of by a centralised entity i.e., signalling server. New peers will be added to the topology according to their score and available slots as we know that nodes can be heterogeneous in terms of internet connectivity and thus have varying capacity.

Video stream will be pushed from the parent node to their children in this tree overlay topology. In case, children fail to receive the stream (e.g., parent node failure), they utilise auxiliary connections to continue the stream.

The base full mesh topology is utilised to the fullest to provide seamless stream delivery. When a peer-to-peer RTC Connection in the overlay network is broken, then an auxiliary peer starts sending stream to the orphaned nodes temporarily until the normal overlay is established. The server finds that a node has failed. It then informs every orphaned child so that they can create an auxiliary RTC Connection with an auxiliary peer to minimise the interruption in service.

The auxiliary links are established with the peer that can provide the stream with the least number of hops possible. WebRTC enables the quickest transition between stream sources. The auxiliary peers will be stored for every node such that there is no lag in the transition. Siblings of parent and grandparent will be the preferred for auxiliary nodes. The list of auxiliary nodes will be sorted to get the best auxiliary node. The list is required to be kept prepared with the best node identified.

The hybrid approach constitutes of a full mesh based topology and an overlay tree network of RTC Connections. The reason for going with this approach is predominantly to lower the reconnection delay after a network abruption. RTC Media Channels are very fast to establish in WebRTC. During network churns, the links in the full mesh become auxiliary RTC Connections to deliver the stream in case of multiple node failures. WebRTC implementation has optimal bandwidth requirements and has been tested to work under low bandwidth network conditions as well. Our approach has mobility of an unstructured mesh during a churn and also the low latency stream delivery of a push-based stream in a tree topology. The overlay network is created with the best parent-child RTC connections with a global outlook on the current network conditions. No peer is over-populated with children nodes if there already exists another peer with unused capability.

Once a source peer has joined the network and it is willing to stream the media, parents will be assigned to new peers on their respective joining as per the suggested Joining Protocol (See Section IV-B for details). The score of a node is calculated after taking uploading bandwidth, video streaming rate, latency, and stability of that node into consideration. Eqn. (1) has been proposed for the calculation of score.

First term in Eqn. (1) represents the serving capacity of a peer or the number of children a particular peer can forward stream to without much impact on its own performance. $upload\_bandwidth$ is the upload bandwidth of the node and $streaming\_rate$ is the rate at which the data packets are being generated, both given in Mbps. Next term depends on the time taken for the stream to reach from source to the peer. $latency$ is the latency observed on the node when establishing the connection and for further data packet transmission. Last term signifies the stability of the peer i.e., the measure of node failures. $active\_duration$ is the time duration for which the node has been active and $num\_of\_failure$ is the number of times the node has failed or lost connection. The second and third term is used to update the score of a peer during the stream. Where, $k_{1}$, $k_{2}$, and $k_{3}$ signifies the weights of the terms and values of these constants varies according to applications. The weights sum up to one, i.e., $k_{1}+k_{2}+k_{3}=1$.

Also, the peer is assigned slots i.e., the number of children nodes it can handle. The slots are assigned according to the upload bandwidth of the peer. Slots of a peer $P$ can be calculated using Eqn. (2).

Whenever a new peer wants to join the streaming, score and available slots will be calculated on the client side with the help of the Eqn. (1). It will then send a “JOIN” request to the signalling server. Score and number of available slots will be embedded in this joining request only. Server finds the most suitable peer to be the parent of the new peer based on the score and available slots of the peers that are already present in the network. Newly joined peer can even replace a direct child of source if it has a better score than that node in terms of forwarding capabilities and serving power. Algorithm 1 is the pseudo-code for the Peer Joining Algorithm. The algorithm is explained further with few possible joining scenarios as examples.  
Scenario 1 : New peer joins. Room contains none except Source.

New Peer A has joined the network and has initiated Stream receiving protocol by sending “JOIN” request to signalling server. Server recognises Peer A being the only node in the room except Source and connects Peer A with Source directly. An RTC Peer Connection Channel is setup between Source and Peer A.

 
Scenario 2 : New peer joins. Source has children with lesser capacity.

Source with two children A and B with capacity 3 and 4, respectively, are present in the room along with respective sub-trees. New Peer C joins the network. C has a capacity of 6. Server finds the child of Source with the minimum capacity lesser than that of Peer C using the function findMinSlotChildOfSource(). In this case server finds Peer A. It makes Peer C the direct child of Source and Peer A child of Peer C. This is done to keep high capacity nodes in the front line of the overlay such that stream delivery has the least latency possible.

Scenario 3 (General scenario): New peer joins. Source has children, all with higher capacity than new peer.

Joining request is sent to the signalling server from a peer X with score and slots values. Server checks that the source peer is working at its full capacity and none of its children capacity is less than the new peer. As per our proposed algorithm, signalling server finds the peer with highest score S using the function findBestParent() which is available to server new peers i.e., currently not working at its full capacity and assigns it to new node. Peer X can start receiving streams from the assigned parent.

A peer has left the streaming network can be concluded from two ways. Firstly, if the server has received a “LEAVE” request from the peer. Secondly, a heartbeat ping-pong protocol runs on the server at regular intervals. If a server misses a pong message of a peer that would signify a node failure. In both the cases, leaving peer will be replaced by its best child and will then be removed from the network. If leaving peer was a free rider and its parent was the source node, then replace it with the most suitable peer (any node from the grandchildren sub-trees of Source) in the overlay network for efficient utilisation of uploading bandwidth of source and to maintain maximum width tree topology.

Till the new parent is assigned to the orphaned children (if any), these children will continue to receive the stream from the auxiliary connections (as explained earlier in Section III-E) to avoid any glitches that might occur because of peer leaving. Algorithm 2 is the pseudo-code for the Peer Leaving Algorithm and has been explained with possible scenarios as examples.  
Scenario 1 : Node with non-zero children fails and leaves. Node A was an intermediate node in the overlay network.

Server will find the child with the maximum score from the set of direct children of the leaving node. Maximum score child will now receive the stream from the parent of leaving node and the rest of the children of leaving node will join the network as per the joining protocol.
Scenario 2 : A leaf node fails or leaves the network. Node was also a direct child of Source.

Leaving node will create a vacancy of a direct child of source. Signalling server will find a peer with the maximum score which is not a current child of source and will put it (keeping subtree of this node intact) directly under the source to fill that vacancy.

Signalling server will detect peer failure with the unacknowledged heartbeat pings. Once any failure is detected, node leaving protocol will be executed with only one difference of incriminating node failure count by one every time it fails. Node failure count will have an impact over the stability value of node and so score also changes in the similar fashion. New score will be recalculated and used only when this node again joins the streaming network.

We designed fybrrStream with main focus on reduced latency, packet loss, and startup delay. Therefore, to evaluate our approach and to assess the impact of our approach on the mentioned parameters, we define performance evaluation metrics as

• -

  Packet delivery ratio (PDR): It is the ratio of the number of packets received by a node to the maximum number of packets that could have been received.

• -

  Latency: It is the time difference in receiving the packet by a node after it was uploaded by the source node. Calculated as defined in Section 6.4.1. of RFC 3550 [17].

• -

  Jitter: It is the variation in inter-arrival time between consecutive data packets. Calculated as defined in Section 6.4.1. of RFC 3550 [17].

• -

  Startup delay: It is the time taken by a node, after joining the session, to start receiving the stream.

Implementation of fybrrStream is hosted on Heroku [18] servers and the same was used for the experiments performed. Performance of proposed approach and algorithms are analysed in a real-life scenario with a field-test. In this field test, around 56 students from our institute participated in the performance evaluation as an individual peer from their home, thus we were able to get such diverse geo-location (as shown in Fig. 7) of the peers. The nodes were heterogeneous in nature with respect to Internet speed and computation power. This was helpful in showcasing a more realistic scenario where the nodes are on different networks, experience different Internet bandwidths and are using different mediums for communication (Ethernet, WiFi, 3G, 4G) with different signal strengths. Both mobile and desktop users were allowed to participate. Institute TURN servers were utilised to relay stream in case direct peer connection was not possible because of firewall or symmetric NAT. Our participants were distributed across the map of India as shown in Fig. 7, out of which few peers shared the same city/town/village. The source node is responsible for creating a room and sharing the stream to other participants in the room. The users joined the room through a web-link which was provided to them prior to the experiment. The source node started the stream after all the users joined the room to maintain consistency in the collected data. As the users were joining the room, fybrrStream was simultaneously constructing the peer topology and measuring the startup delay for each joining node. WebRTC statistics was accessed with the help of getStats() API [19] which returns a WebIDL dictionary [20] containing the values of several monitored metrics at a specific moment in time. During the live stream, we collected data for packet loss, packet received at each node, along with it, jitter and latency is calculated for each leaf node of the constructed overlay tree topology.

Fig. 10 shows the upload network bandwidth distribution of participated users in the field test. The values are in megabits per second obtained through Ookla Speedtest or Google Speedtest. Fig. 10 shows the number of slots assigned to the number of nodes present in the session and from the plot it can be seen that the maximum number of nodes have 2 slots whereas some nodes have 3 and 4 slots. For this experiment, we have restricted the number of slots to 4 due to network and computational limits on the user-end. Fig. 10 shows the overlay network topology formed by the nodes in the fybrrStream scheme. As can be clearly seen in the figure that the source node is connected to 2 nodes, which means the source node has 2 slots which is then followed by the 2 children nodes of the source node, with 4 slots each. It can also be observed that the nodes with higher value of slots are placed nearer to the source node compared to the nodes with lower number of slots. This figure represents one instance of the whole session, as the topology varies slightly with each node joining and leaving mid-session.

Table III defines the parameters and values considered for the experiment.

The proposed scheme is compared with two other tree-based schemes - ’binary tree’ (2 children nodes per peer) and ’quaternary (quad) tree’ (4 children nodes per peer). The reason behind choosing these schemes to compare with fybrrStream was the similarities between the schemes (binary and quad are tree based, and fybrrStream has a tree overlay network) but the heuristics causing massive differences in the results. Binary tree has been suggested in multiple papers [7], [8]. We also wanted to reduce the height of the tree to decrease latency in the leaf nodes. So we considered a quad tree, in which every peer would have 4 children nodes (regardless of bandwidth constraints), and that would decrease the height of the tree.

The height of the topology tree with increasing number of nodes for different schemes is shown in Fig. 12 and can be related through Eqn. (3):

where $height_{quad}$, $height_{fybrrStream}$ and $height_{binary}$ is the height of the topology tree in case of quad tree, fybrrStream and binary tree, respectively. In fybrrStream, peers are assigned the maximum number of children nodes they can support based on their upload bandwidth. For this experiment, we considered to keep a minimum of two slots for every peer as all had an upload bandwidth of around 5 Mbps, so a peer could afford forwarding the stream to atleast 2 peers. Therefore in the worst case an overlay network created will have been the same as that created in the binary tree scheme.

Now, if $n_{1}$ nodes have $2$ children nodes assigned to them, $n_{2}$ nodes have $3$, and so on till $n_{i}$ nodes have $i+1$ children nodes assigned to them, and

Therefore we can say,

Hence, height of tree in fybrrStream would always be less than or equal to height of tree in binary. Due to full utilization of stream forwarding capacity of a peer, even for a high number of live stream viewers, the height of the tree in case of fybrrStream will be quite low.

fybrrStream follows the scoring function for each new node and assigns children to the nodes accordingly. This ensures that maximum utilization of node capacity is done. For this experiment, compared to fybrrStream, binary tree topology under-utilizes the capacity by $23.6\%$, whereas quad tree topology over-utilizes the capacity by $52.8\%$. These values can be clearly justified by the fact that node capacity is not fully utilized in case of binary, whereas nodes are over-utilized by forcing 4 children to each node in case of quad tree.

Fig. 12 shows the plot for startup delay for each node joining the session. The startup delay is defined as the time taken for a node to start receiving the stream data packets from the moment when it joined the room. An average of 1.56 sec startup delay was observed for fybrrStream, which is excellent for such a system. We can also see in the graph that the startup time does not increase with increase in the number of nodes, which shows that the proposed Peer joining algorithm works well and gives stable results for any number of nodes in the system. This startup delay can be considered as a loading time for streaming services and thus we can say that a loading time of 1.5 sec is comparable or even lesser than some of the state-of-the-art streaming services which use dedicated CDN servers. Moreover, fybrrStream performs better than the proposed model in [21] as well as Fast-mesh [22] in terms of the startup delay.

Fig. 13 shows the same frame of streamed video received by one of the leaf nodes in the quad, binary, and fybrrStream overlay topology. It can be seen from the images that fybrrStream provides a better quality stream than the other two schemes.

Fig. 16 shows the plot for average latency at leaf nodes of the overlay network during the stream. As we see, fybrrStream gives a much lower latency than binary as it minimizes the number of hops from source node. However, at some moments, quad tree has lower latency readings than fybrrStream as it has a much lower number of hops in the network which helps to reduce the latency at leaf nodes. In Fig. 16, we can see that the median value of latency for all the nodes in quad tree is lowest followed by fybrrStream which is followed by binary tree. It is also evident that the median value of latency in fybrrStream is lower than minimum value obtained in the case of binary tree which suggests that fybrrStream performs much better than binary tree with respect to latency and quad performs better than fybrrStream. However, quad suffers in average PDR of the nodes (see Fig. 19) which implies if we increase number of children nodes more than what a node can support then there would be more packet loss although the latency might go down due to decrease in number of hops in the overlay tree.

Fig. 16 shows the plot for average jitter at the leaf nodes with respect to the time duration of the session. As can be seen in the plot that the jitter at leaf nodes in case of quad tree is high and also has huge variation through the session whereas for binary tree case the average value is slightly higher than that of fybrrStream, which showcases that fybrrStream has a more stable network topology. Fig. 19 provides a more comprehensive study of the jitter. The median and the range of values for jitter in case of fybrrStream is lower than both of the other two schemes.

Fig. 19 shows the plot for an average PDR at any node in the session. The PDR is defined as the ratio of packets received by a node and the total number of packets transmitted for the node. The total number of packets is equal to the sum of packets received by the node and packet lost by the node. We can see in Fig. 19 that the PDR for binary tree approach is highest, followed closely by the PDR of fybrrStream and then lastly by quad tree approach which is more evident in Fig. 19. It supports the fact that binary tree and fybrrStream are closely related with respect to PDR, whereas quad tree lacks behind by a significant margin, thus it is safe to conclude that fybrrStream provides sufficient PDR for better QoS. The PDR is higher in the case of a binary tree as each node only needs to forward the data packets to two other nodes. However, the height of the binary tree will be larger than fybrrStream and we would receive a higher latency, which can be seen in Fig. 16.

For fybrrStream, we assign the number of slots to a node according to its bandwidth and by this we achieve an equilibrium between latency and PDR. For the quad tree approach, the PDR is much lower, which can be explained by the fact that each node is required to forward the stream to four child nodes. Here, the quad tree approach requires a lot of uplink network bandwidth therefore the parent node is unable to match the PDR of fybrrStream and binary tree approaches.

Table IV lists the mean values of latency, jitter and PDR for all the three schemes. As can be seen in terms of latency, the binary tree has 64% higher latency and the quad tree has 64% lower latency than fybrrStream. Binary tree shows 60% higher jitter than fybrrStream whereas quad tree shows 207% higher jitter. In terms of PDR, binary performs 0.013% better than fybrrStream and quad performs 0.063% worse than fybrrStream.

By this real-time field test experimentation, we tried to cover all the major factors which affect the quality of service for a user streaming through a peer-to-peer streaming service. fybrrStream worked efficiently to utilize the maximum capacity of each user node. PDR and latency plots show how the subtle choices made by the algorithm for assigning parents affect the final results whereas the comparison with the other two schemes suggests the importance of utilizing the capacity of a node efficiently.