CWcollab: A Context-Aware Web-Based Collaborative Multimedia System

A crucial feature distinguishing our design from other collaboration systems is the general-purpose distributed collaboration with a uniform interface. As events are transmitted in a standardized message format, this design significantly relieves the difficulty of supporting a new type of media. Figure 1 shows the structure of collaboration based on a uniform interface.

Sender flow. For the client who initiates a media event for collaboration, there is a media event capturer module to monitor and capture the event. A media state recorder will be invoked when necessary to record the current media state for possible resynchronization. For example, the state of a video may include the play/pause state, volume, progress, playback rate, related annotations, etc. At the same time, the event is sent to a message serializer, which encapsulates the event into a standardized message and sends it to the backend service.

Receiver flow. After the message is routed to one target client via the backend service, a message deserializer takes charge of unwrapping the message and sending information to the media event replayer to update the current media state. Simultaneously, the change of media state may also be recorded in the media state recorder module.

With the uniform interface, if a developer would like to add collaboration mechanisms for a new type of media, he/she just needs to follow the structure by adding functions in the media event capturer, media state recorder, and media event replayer. Message serializer and message deserializer should be left intact if the format of transmitted messages is not changed. The feature of extendability significantly relieves the burden of implementing new functionalities for collaboration.

The main functionality of the media event capturer component is to capture an abstract media event via DOM events. In our design, by using event propagation, especially event bubbling, which is now supported in all of the modern browsers, we create another layer on top of original media objects to capture these events. We leverage an object-prioritized hybrid approach to capture media events. Specifically, it consists of:

• •

  Object-based. Events are captured as DOM events on the media object or a UI component. This provides more flexibility than the traditional position-based approach, as no matter where a media event fires, we only trace the object effected, isolated from the influence of positions. This type of capture scenario is much more often seen than others. Figure 2 illustrates how to capture object-based video events. In the figure, every control is triggered through a UI component. For example, when a user plays/pauses a video, he/she clicks the video control button, which is eventually captured by the media event capturer as a click button DOM event. similarly, when a user seeks to a timestamp in the video player, he/she clicks the progress bar, and the media event capturer captures this event as an update slider DOM event.

• •

  Proportion-based. Events are captured as DOM events related to the positions proportionally to the display. Although object-based capture handles most scenarios, some media events may not be object-based, especially those related to mouse events. For example, we use the relative position of an image to the screen to synchronize an image moving event.

• •

  Value-based. Events are captured as DOM events related to value changes. For example, when scrolling a mouse to zoom in/out an image, the event can be captured via a wheel, mousewheel, or DOMMouseScroll event. This type of event is fired with a value to represent how much the mouse has scrolled.

The media event recorder is used to track the current state of the media. The state of a media block depends on the type of media. A state is a set of key-value pairs. For example, a state of a video contains the video source, current timestamp, muted or not, volume, playback rate, annotations, etc. The key-value pairs can be compressed into messages for further transmission.

The main functionality of a media event replayer is to replay events attached with complete media event information. To achieve that efficiently, we design a hierarchical tree structure-Handler Tree-to handle messages, as shown in Figure 3.

An event is propagated in the handler tree via parent-child chains until arriving at a leaf node. During the propagation, the received message is parsed gradually, and finally, in a leaf handler, the message is totally consumed to update the UI if necessary. An example of handling a video play/pause event in CWcollab is demonstrated in Figure 4.

When an event is received by an audience, it is first sent to the root node, also acting as a dispatcher in the handler tree. The next level in the handler tree contains various handlers for corresponding types of media. The message is sent to the appropriate media handler at this level. Then, a specific action handler is invoked to fire the target DOM event. For the example in Figure 4, the message is first handled by the root handler and dispatched to the video handler according to its media type. Because the event type is button click, the message is next sent to the button click handler for further process. Then the handler reads the target id and triggers the click event on the element with this identifier. The handler also loads the value of current time in the data field to update the current progress of the video to that timestamp. Considering that our system is real-time, the tree is flat to reduce the overhead.

With the help of the media event replayer, the events occurring in a collaboration session can be replayed sequentially according to their timestamps. The events occurring in a session are pushed into a queue structure. These events are scheduled to be popped from the queue and sent to the handler tree for replay.

In CWcollab, collaboration events are captured and represented by a standardized format of strings, and the strings are transferred among collaboration subjects. There are two different types of messages: media events and control events. For a media event message, it represents an action related to a specific media block, such as scrolling a web page. Table II demonstrates the possible fields in a media event message. By contrast, a control event message represents an action involving controls in a collaboration session, such as resynchronizing the media state. Table III shows the fields for a control event message.

The responsibilities of a message serializer and message deserializer are to wrap a media message into a standardized format of string and to read the message from such a string, respectively. To maintain consistency, the two components share the same message format.

We created a web application¹¹1 Demonstration video: https://youtu.be/X-IWFSBvfSc. based on CWcollab to provide a collaborative presentation service, including account management, materials preparation, presentation synchronization, and replay. The backend service is implemented in Node.js, which is an event-driven, non-blocking, and cross-platform JavaScript run-time environment. We use MongoDB, a document-oriented NoSQL database, to store presentation data. We also use Redis, which is an in-memory key-value database, as a message broker.

The participants of a collaboration session can download media materials beforehand or the presenter can dynamically add materials in a session. In the application, manipulations on the material in the presentation panel are synchronized to other users’ panels.

Currently, screen sharing products such as Zoom and Google Hangouts are widely used to provide basic collaboration support in video conferencing. In this section, we compare Google Hangouts with CWcollab from multiple perspectives. Although we didn’t evaluate other video conferencing products in details, we reckon that they share similar mechanisms for screen sharing.

We performed a series of experiments on multimedia, including videos, PDF documents, images, and web pages, to measure the network bandwidth usages in one minute of Google Hangouts and CWcollab, on the presenter side. Instead of transferring video chunks during a presentation, the static materials can be loaded beforehand, and only minimal events information is transferred in a session. We found that CWcollab requires a much lower bandwidth usage, as illustrated in Figure 5, 6, 7, and 8. Note that the y-axis units are different in these figures.

Figure 5 shows the network bandwidth consumption of both Google Hangouts and CWcollab for presenting a video. Figure 5(a) shows the Google Hangouts network bandwidth consumption is around 60KB/s - 100KB/s, which is mainly for transferring video frames. However, CWcollab presets the static media-the video-in this experiment, and the network bandwidth consumption is only around 0B/s - 1KB/s in Figure 5(b). The bandwidth consumption comparison is summarized in Figure 5(c). Similarly, Figure 6 shows Google Hangouts utilizes around 60KB/s bandwidth while CWcollab only needs a maximum 3KB/s for presenting a PDF document; Figure 7 shows Google Hangouts utilizes around 70 KB/s and CWcollab only needs a maximum 2KB/s for presenting an image; Figure 8 shows Google Hangouts needs around 100KB/s and CWcollab needs almost 0B/s most of the time (the two peaks are for loading web pages). To summarize Figure 5-8, we can see that in the measurement of one minute, most events only require hundreds of bytes per second. The most expensive action is the free drawing event whose bandwidth usages can be as high as around 3KB/s. However, the usages are still much lower than those of Google Hangouts.

We also summed up the bytes transmitted in this one minute and showed the results in Table IV. CWcollab transmits very few bytes in a presentation, especially noticing that in the collaboration session except web pages, in one minute, the total bytes transmitted in CWcollab is less than 10 KB, because CWcollab leverages an object-prioritized approach to capture media events and represent them in simple messages. As the web pages have to be loaded dynamically, even though loading pages costs a large amount of network bandwidth, CWcollab still requires much lower bandwidth than Google Hangouts.

Besides bandwidth usages, there are some other differences between screen sharing tools and CWcollab. Because the mechanism of screen sharing tools is to capture and broadcast the display, it cares nothing about what application the user is synchronizing. As a result, it is straightforward for these tools to support multimedia in web browsers and desktop environment. By contrast, our platform is web-based. To support another type of media, we need to instrument necessary control to achieve the collaboration. Additionally, CWcollab can be a supplement to current screen sharing products, considering that it provides an efficient platform to support message-based collaboration on multimedia.