Architectural Design Alternatives based on Cloud/Edge/Fog Computing for Connected Vehicles

In comparison to smartphones, the amount of data generated from a vehicle is huge in its volume. A high-end vehicle is typically equipped around a hundred sensors or more to monitor correct system operation, and enhance safety and driving comfort. Even though not all raw sensor data need to be transferred to the remote cloud, it is generally expected that each vehicle needs to send at least 20 GB of data per month to the cloud to achieve practical automotive applications according to Automotive Edge Computing Consortium (AECC) [128]; in contrast, in spite of varying statistics, a typical smartphone user consumes around 2 to 5 GB of cellular data these days²²2These statistics exclude Wi-Fi usages..

In addition, the dataflow pattern of vehicle data is quite different from smartphones. Most smartphones are dedicated for downloading services; that is, the remote cloud server is typically a data producer that creates contents, and sends it to the smartphone, which is a data consumer. Due to this characteristic, many Internet Service Providers assign higher network bandwidth to the downlink services than uplink services. On the other hand, a vehicle is more likely to become a data producer, which generates various raw data, and send it to the cloud for being used by additional services.

One typical type of the service response time is a duration from the moment a user requests a data or computation to the moment it has been completed or received by the user. Both smartphone and CV services need to meet diverse granularity of timing requirements; for example, real-time multi-user game (smartphone service) or road-side object recognition (CV service) typically need to meet the response time in the order of milliseconds; on the other hand, storage backup application (smartphone service) or HD map generation (CV service) need to meet the response time in the order of seconds or minutes.

However, the consequences of violating such expected response time is significantly different each other, so designing those systems also become different. Many CV services are safety-critical services where delayed response time has a safety impact on drivers or others. For example, a vehicle platooning service that needs to guarantee a constant distance among a group of vehicles may end up crashing each other unless a series of positions of other vehicles do not arrive on time. For this reason, many CV services are typically equipped with a fail-safe mode that is activated when such abnormal condition arises. Hence, the architecture should be designed more robustly to cope with such abnormal delays and to provide sufficient information to activate such a fail-safe mode. On the other hand, most smartphone applications do not have safety implication on the users when the response time is delayed, so the supporting architecture typically do not accompany with such a consideration in place.

Both smartphone and vehicles may be equipped with various services that require different degrees of network availability depending on their service requirements. Some services require high network availability to provide proper functionalities such as video streaming (smartphone service) or vehicle platooning (CV service). On the other hand, other services may only require intermittent network availability as their local compute unit and storage can support the continuation of the services without continuous network connectivity, such as downloadable standalone games (smartphone service), downloadable navigation map (CV service).

Even though there is an ongoing debate as to the best way to provide connectivity for future vehicles [129, 130, 131], vehicles may be exposed more heterogeneous wireless networks that have different costs and latencies than smartphones while they are moving; a cellular network is typically the only option for a smartphone to maintain the connectivity while it is moving at a similar speed with a vehicle. In United States, some of ongoing V2I services [122] are provided by government via DSRC that allows DSRC-compatible vehicle to freely receive the public services. At the same time, a vehicle is also equipped with a cellular modem that can transfer other types of data via cellular network, which incur costs in most cases. This requires a moving vehicle to make a unique design decision, which does not arise in a moving smartphone, as to when multiple network options are available on the vehicle route, how to schedule the service execution by considering various aspects, such as latency, cost and so on.

Unlike smartphones’ impact resulted from security or privacy attack, the CV service attack has safety implications as those services are linked to safety-critical control applications. For example, a roadway signal infrastructure may broadcast safety messages (e.g., pedestrian positions or speed limits) in intersections for a crash-mitigation service [122]; when the information is compromised by attackers, a vehicle that utilizes the fake information may trigger unexpected control operation resulting in safety issues such as unexpected hard braking due to fake pedestrian crossing information or unexpected speed increase due to fake speed limit.

However, it is also challenging to achieve the necessary degree of security and privacy as it typically negatively affects system performance and convenience. For example, adding strong encryption to all data from cloud or out from vehicles may add extra complexity to the vehicle system design such as latency, extra compute and storage power. Therefore, it is necessary to impose different types of security measure depending on their criticality levels accounting for their interaction with control-related systems. Note that a smartphone also provides multi-level security and privacy measures, but the burden to achieve the required level of security and privacy is mainly imposed to the users by asking more credential (e.g., multi-factor authentication). However, it is not possible for drivers to follow such complex procedure during driving, so it is desirable to perform such procedure more seamlessly.

Vehicle data has a higher locality than the one used for smartphone services. Many CV services utilize data that is only consumed in the areas where it is originating, such as positions of other vehicles, semantics of road signs, local HD map information and so on; that is, such data is meaningless in other remote areas irrelevant to the CV services. Therefore, it is not desirable to send all data to remote clouds as it consumes the network and compute resources unnecessarily such as network bandwidth or cloud storage. The system architecture should be able to support the unique characteristics of data locality for CV services so that the infrastructure-wide resources can be efficiently utilized for other non-CV services as well.

In addition, as data is increasingly an important asset to each country, it is necessary to follow the local rules and regulations imposed by each nation. For example, some countries may restrict some type of data to physically stay in their territories depending on how they are used. If a vehicle is used for a certain that falls in such a restrictive category, the data transmission should strictly follow the local regulation. These days, OEMs typically do not have that level of customization as it increases the manufacturing cost significantly. However, this situation will arise as more data is shared with remote cloud or vehicles, so it is necessary to consider a system architecture design to enable data to be transmitted conforming such local regulation via support from either in-vehicle system or infrastructure.

Due to the reason that vehicular connections are usually uncertain and frequently changing in topology, and thus the reliability of vehicular networks is still challenging. At the same time, the bandwidth resources of cellular networks are limited and BSs’ signal cannot extend to all the urban and suburban areas. Therefore, we need to design a heterogeneous vehicular network that combines the best of cellular networks and V2V ad hoc networking. In such a heterogeneous vehicular network, resource sharing and co-scheduling among different networks is still an open issue. A lot of work, currently, has investigated in resource sharing in 5G-enabled vehicular networks [132]. However, so far co-scheduling mechanism design is still lacking for CVs with heterogeneous communication network support[133].

Since co-located vehicles often require shared content, such as navigation or environment recognition, the broadcast nature of V2V communications can improve the performance of content sharing in vehicular networks. However, due to the fact that IEEE 802.11p utilizes the carrier sense multiple access with collision avoidance (CSMA/CA) mechanism, the request-to-send/clear-to-send (RTS/CTS) handshake is disabled in broadcast [29]. Thus, V2V communications are subject to a severe hidden terminal problem that will cause potential collisions at the receivers. Therefore, an efficient medium access control (MAC) mechanism should be designed to avoid such collisions. According to the space-constraint deadlines, the data demands have different urgent levels, even for the same content. Thus, in a certain area, sharing different data contents will have different gains. In order to increase the amount of data transmitted through vehicular networks as much as possible, three issues should be carefully planned: (i) which content to broadcast; (ii) when to broadcast; and (iii) which vehicle to broadcast. Edge servers that know the served vehicles’ content demands and positions, can coordinate transmissions among vehicles to improve the network gain and make sure no collisions among the served vehicles. However, a single edge server cannot realize collision avoidance among vehicles in different locations. Timely and frequent interactions among multiple edge servers should be introduced to avoid such collisions. Furthermore, the frequent control messages among multiple edge servers and served vehicles will incur extra overhead for vehicular networks. Thus, it is necessary for vehicular networks to share contents in a distributive and cooperative manner [29].

Vehicular applications can be deployed in cloud centers, edge/fog nodes, or vehicles. The application deployment depends on factors such as the vehicular network topology, users’ delay tolerance, and the vehicles’ and users’ mobility predictions. In paper [134], the authors focused on application deployment on the rented cloud nodes or the own fog nodes, and proposed a heuristic-based algorithm that tries to make a trade-off between the makespan and the expenditures of cloud nodes. In paper [135], taking the mobility of mobile devices into consideration, an adaptive content reservation scheme, which reserves the resources on the cloud centers and fog nodes for real-time video streaming to mobile devices, is proposed. In paper [136], the authors proposed to develop a new fog that treats the idle devices of game players or organizations as fog nodes, rendering game streaming to the nearby players. Still, it is challenging to consider the mobility of edge/fog nodes in application deployment. On the one hand, both the vehicles and the data sources may move at the same time. On the other hand, it is complex to coordinate the computation and communication systems simultaneously [133].

Most existing researches focus on the potential attacks or threats in cloud/edge/fog-assisted vehicular applications. Attacks can be generally classified into two types, active attacks and passive attacks. The functionality of a vehicular cloud/edge/fog system cannot be destroyed by passive attacks which only want to eavesdrop the private information. However, the active attacks are more damaging than the passive ones, due to the reason that the active ones attempt to interrupt the operations of the cloud/edge/fog computing systems, or modify the sharing data. Active attacks are usually easy to be detected when it induces huge damages to the vehicular system. However, it is hard to be noticed if the attacks are performed in an inconspicuous manner within a very limited time [117]. Additionally, the attackers generally falls into two categories: insider and external attackers. An insider attacker comes from inside the vehicular system, and are usually equipped with key materials. An internal attacker may induce more potential risks than an external attacker, since the internal attacker knows the existing security control policy and can circumvent it.

Specifically, in paper [137], the authors show that in a connected car environment, a real vehicle and malicious smartphone application can be used to perform a long-range wireless attack. Then, a security protocol for controller area network is proposed as a countermeasure. In paper [138], the authors identify the security challenges that are specific to vehicular clouds (VCs), including authentication of high-mobility vehicles, scalability and single interface, the complexity of establishing trust relationships among multiple players caused by intermittent communications, and tangled identities and locations. A security scheme is proposed to address several of the aforementioned challenges. In paper [139], security issues in service-oriented vehicular networks are elaborated, i.e., minimizing V2I authentication latency and distributed public key revocation. These two security issues are considered as among the most challenging design targets in service-oriented vehicular networks. Accordingly, a fast V2I authentication based vehicle mobility prediction scheme and an infrastructure-based short-time certificated management scheme are proposed to address the aforementioned two challenges. In paper [117], the authors discuss several key security and forensic challenges and their potential solutions. A secure VFC implementation should provide multiple baseline security and forensic properties, including confidentiality, integrity, authentication, access control, non-repudiation, availability, reliability, and forensics. Most of the security requirements can be achieved by cryptographic techniques. Moreover, the authors investigate the compromise attack and selfish attacks, and their potential countermeasures. In paper [72], the authors holistically analyze the mechanisms, challenges, and security threats existing in all edge scenarios, while highlighting the collaboration and synergies among them. In paper [140], the authors consider the security and privacy aspects of CVs, including security of communication links, data validity, security of devices, identity and liability, access control and privacy issues.

Paper [141] proposed a three-layer VC architecture from the perspective of communication support. This architecture is composed of the device level, communication level, and service level, as shown in Fig. 3. At the device level, various devices ranging from sensors, actuators, Global Positioning System (GPS) devices, and smartphones are used for collecting data such as temperature, pressure, image, and driver’s bio-medical information. Then, these collected raw data are stored in a repository and wait for further processing at the upper level. Based on the pre-processing techniques, those stored raw data can be classified into high-level context (such as human activity and gesture) and low-level context (such as pressure and temperature). The communication level is divided into in-car communication modules (e.g., BASNs), V2V communication modules, and V2I communication modules (e.g., satellite and cellular networks). The service level includes various services such as context-based services (e.g., driver status monitoring), communication-based services (e.g., road traffic monitoring, weather information, and Internet access), and customized services (e.g., parking, health-care, and dining booking). Context-based services are given charge of tasks that include drivers’ health and safety improvements, while tasks like drivers’ convenience and comfort degree improvements are allocated to communication-based services. As stated above, this proposed three-tier architecture collects a wide variety of data on the device level. Thus, customized CV services that require various data and high accuracy are suitable for being executed in this architecture.

In paper [142], a four-layer agent-based intelligent traffic cloud architecture from the perspective of computation support is proposed. This proposed architecture mainly focuses on the scenario of intelligent transportation systems and aims to handle a large amount of computing and storage resources that are required to use traffic strategy agents and massive transport data. It includes the application layer, platform layer, unified source layer, and fabric layer. The application layer shown in Fig. 4, contains all the applications that run in the clouds, including agent management, generation, optimization, testing, traffic decision support, and agent-oriented task decomposition. Customers can obtain the services that they need through a pre-defined standard interface. The platform layer is made of artificial transportation systems, providing platform as a service. Components such as weather simulator, population synthesizer, $3$D game engine, and path planner are contained in this layer and provide services to the traffic applications in the upper application layer. The unified source layer maintains the hardware resources in the lower fabric layer and provides infrastructure as a service. In the unified source layer, VMs are utilized to protect users’ data and equipment safety. In addition, a unified access interface is established for the upper distribute computing resources. These features described above can help the intelligent transportation system efficiently mine useful knowledge from the massive urban traffic data. Furthermore, hardware-level resources (e.g., storage, network, and computing resources) are contained in the fabric layer, which will help the intelligent traffic cloud supply the peak demand of urban-traffic management systems. CV services with requirements such as large data size, high security, multiple users, latency-insensitive, and high computing power are recommended for using this four-layer architecture.

As stated above, one of the advantages of centralized cloud architectures is data aggregation. By using the cloud storage techniques, the cloud can provide a variety of stored data for the private and government agencies (e.g., department of transportation, the meteorology department) to perform various studies. Therefore, several papers described various studies in vehicular systems based on centralized cloud computing architectures. For example, paper [143] proposed a metropolitan air quality monitoring service. In this service, vehicles act as the air quality data gathering sensors under a similar architecture described above. Vehicles offload their gathered data to a remote centralized cloud for performing an air quality estimation. The main contribution of this paper is the investigation of the trade-off between monitoring accuracy and data offloading overhead, where dynamic grid partition and probabilistic reporting approaches are proposed to adjust data sampling rate and avoid redundant data.

However, with the development of the latency-critical and computation-intensive applications in CVs, centralized architectures with a remote cloud server described above are not efficient. Therefore, architectures of CV networks with MEC are proposed, where besides a centralized cloud server, additional computing infrastructures (e.g., RSUs, BSs, and/or edge/fog servers) are also given the role of computing units, forming a hierarchical computing architecture. The main objective of MEC is to extend the cloud computing functionality to the edge of networks, which saves network bandwidth and reduces the communication latency. For example, several papers [181, 132, 184] proposed to utilize 5G and MEC together to help CVs improve task transmission efficiency. Among these papers, paper [132] proposed a paradigm of $5$G-enabled vehicular networks to improve the network capacity. The cloud server is extended by integrating geographically distributed cloudlets that are responsible for local services. In addition, the matrix game theoretical approach is exploited to operate the resource sharing and allocation among cloudlets.

In addition, other variations of hierarchical computing architectures for CVs were also proposed. Paper [144] proposed a cloud-based MEC offloading framework in vehicular networks. Considering the time consumption of the computation task execution and the mobility of the vehicles, data are adaptively offloaded to the MEC servers through a direct uploading mode, i.e., V2I, or a predictive relay mode, i.e., V2V. Furthermore, the proposed framework does not have a remote core cloud server and all computing tasks are offloaded to different MEC servers, which reduces the transmission cost and the latency of the computation offloading. However, since MEC servers are deployed at the edge of networks, e.g., RSUs, their computation capacities, storage, and service ranges are limited. Thus, applications require high computation resource, a large amount of feed data, and constant offloading environment with fast mobility may be seriously restricted by this proposed framework. In papers [145, 147], fog servers are proposed to co-locate with BSs, forming a BS-fog node. However, all of these proposed architectures face a common challenge – mobility management issue. In order to mitigate the impact of mobility, paper [146] proposed a mobility prediction mechanism where RSUs exploit mobility predictions to decide which data they should fetch from the Internet and to schedule the further transmission to vehicles.

Papers [149, 148] proposed a four-layer architecture for urban traffic management with the convergence of $5$G networks, VANETs, MEC, and SDNs technologies, as shown in Fig. 5. It contains the environment sensing layer (e.g., traffic data are derived from the roadside infrastructure and on-board sensors), communication layer, including SDN global controller, 5G BSs, SDN RSUs, and SDN wireless nodes (e.g., vehicles), MEC server layer (MEC servers are deployed in 5G BSs), and remote core cloud server layer. The communication layer with two emerging network paradigms, 5G and SDN, provides several advantages. First, since the data and control planes are separated, forwarding policies can be exploited to balance the traffic flows, and provide a more flexible path selection strategy with the network’s programmability. Second, in order to support different approaches of communication, several wireless modules are deployed on vehicles. Thus, the SDN-based communication layer can provide a path selection strategy based on the cognitive radio and channel allocation policy, which may reduce the communication latency and increase the communication bandwidth in CV networks. Thus, the channel and frequency selection in CV networks can be more flexible. Third, the 5G cellular network with multiple-input multiple-output (MIMO), wide spectrum, and ultra-dense network technologies can achieve $1.2$ Gb/s data rate in a mobile environment, where a vehicle is at the speed of $100$ km/h, based on 28 GHz spectrum [149]. Fourth, because the current cellular network is designed for mobile broadband traffic, it lacks support for V2V communications. Although IEEE 802.11p is proposed for V2V communications, its limited bandwidth and peak data rate (i.e., up to 27 Mbps [108, 107]) might not satisfy the diverse requirements of vehicular applications [194]. The 5G Public Private Partnership (5G-PPP) proposed device-to-device (D2D) communications that data can be directly exchanged among mobile users by bypassing infrastructure within 1 ms delay [195], which demonstrates that 5G is a promising approach for V2V communications. In addition, paper [194] presents three salient features of 5G-enabled communications in vehicular scenarios, including proximity service, integration of MEC, and network slicing. For example, (i) proximity service provides a solid foundation for vehicular safety communications and identifies the source of autonomous vehicle attacks; (ii) MEC plays a fundamental role in 5G [63], which can improve the user experience of vehicular applications such as the traffic information system that have flexible latency requirements; and (iii) variant network slices can be designated based on the diverse requirements of vehicular applications, which simplifies the design of vehicular systems. Compared with the centralized cloud computing architecture, mobile edge servers are deployed closer to end users in this architecture. Thus, it reduces the delay of data offloading and is suitable for implementing latency-sensitive applications. A rapid road accident rescue system, for instance, can be built under this architecture with the support of the low-latency and high-bandwidth SDN-based heterogeneous network and the fast-response MEC server.

Paper [150] proposed a three-layer hybrid computing architecture, including the micro layer (vehicles and users), meso layer (transmission), and macro layer (cloud services). However, the vehicular cluster under this architecture only performs the function of sharing data, e.g., entertainment resources and traffic accident information among vehicles in this cluster, instead of providing computing services. Papers [153, 177] introduced a new concept, fog computing, into the cloud computing architecture for CVs, where a cooperative fog architecture, shown in Fig. 6, is proposed. The cooperative fog architecture mainly contains two layers: edge layer and fog layer. The edge layer may include the components such as VANETs (i.e., a VANET can be applied for V2V communications and traffic information broadcasting), IoT (i.e., lots of IoT application are widely used in city transportation systems such as video traffic surveillance systems, range finders, and wireless sensors), and mobile cellular networks. The fog layer is a federation of geographically distributed local fog servers and may include entities such as fog servers (i.e., the long data transmission latency between CVs and the cloud server can be reduced via offloading the computation and data in the edge layer to local fog servers), access control routers (i.e., they are responsible for controlling or migrating the input data flow), cloud server (i.e., it is deployed out of the fog layer and has strong computation and storage capacity), and coordinator server (i.e., it is responsible for the federation and autonomy of fog networks). Four potential functions might be achieved in this architecture: mobility control, multi-source data acquisition, distributed computation and storage, and multi-path data transmission. Additionally, a flexible hierarchical resource management methodology that includes Intra-fog and Inter-fog resource management is designed for reducing the system maintenance cost and improving the QoS in areas with high population density. Paper [152] proposed a hybrid architecture that offloads the vehicular communication traffic in cellular networks to V2V paths based on the SDN. A centralized V2V path selection approach, a lifetime-based network state routing algorithm, is developed based on the SDN inside the MEC architecture, where each CV reports its location, speed, direction, and IDs of the neighboring vehicles to a context database implemented in the MEC server. The proposed approach can not only find the V2V routing path that has the longest life time but also recover a broken V2V.

Several papers proposed multiple CV services in hybrid architectures. For instance, paper [151] described a photo surveillance service, named Pics-on-Wheels, that a group of vehicles in a certain area are selected to take camera images of a required urban landscape based on the requirements of a customer. CVs that participate in this service should periodically offload their GPS location to the cloud manager. The customer who requests the service has to send a message containing the time and location to the centralized cloud manager first. Then, the cloud manager will search for available vehicles in the requested time and location by using the best vehicle selection algorithm proposed in this paper. This service can assist with forensic purpose where an accident occurred.

The security and privacy issues in computation-aided hybrid architectures can be divided into two categories, VANET and V2I. As we presented, the VANET is one of key components of computation-aided hybrid architectures. Thus, it is crucial to meet the critical security requirements of VANETs for designing computation-aided hybrid architectures. Firstly, one of the most prevalent security issues is how to maintain the availability of each V2V connection in VANETs, where Denial of Service (DoS) is considered as one of the potential attacks that may affect VANETs [196]. Several possible solutions for the DoS attack is discussed in paper [157]. Secondly, in order to secure the integrity and ensure the reliability of applications trust must be developed among vehicles in VANETs. In paper [158], a fuzzy trust model is proposed to secure vehicles to receive correct and credible information from surrounding vehicles. A series of security checks is conduct by the proposed trust model to ensure the correctness of the received information. Lastly, papers [154, 155, 156, 197] jointly investigates the reputation management and privacy protection in VANETs, where reputation management is responsible for rewarding the complying vehicles and punishing the misbehaving ones. A joint privacy and reputation assurance scheme is proposed to reconcile the requirement conflicts of the privacy protection and the reputation management in VANETs. In paper [154], edge servers are adopted to perform local reputation management tasks for vehicles, while, in papers [155, 156], vehicles’ reputation values are updated by themselves. Thus, the solution proposed in paper [154] is said to provide more reliable reputation manifestation and more accurate reputation update than the other two solutions.

Furthermore, V2I is also considered as an important component of computation-aided hybrid architectures, where V2I communication enables vehicles to access services from the cloud/edge/fog computing infrastructures. Preservation of the confidentiality in the V2I communication is one of the most important security requirements. Paper [31] proposes a secure and efficient mutual authentication and key agreement scheme for V2I communications to defend against the RSU replication attack and to prevent all entities from eavesdropping. Paper [159] investigates security challenges in wireless communications between CVs and RSUs and proposed a protocol which enables CVs to download data securely from RSU with privacy preservation in VANETs. In addition to that, a smart security framework for VANETs equipped with edge computing nodes is proposed in paper [160] to provide secure V2V and V2I communications by using the Quotient filter which is a probabilistic data structure.

VC is new technological shifting, where a cluster of vehicles are in the role of corporate computing, sensing, communication, and data sharing units. In other words, “a group of CVs whose physical resources can be coordinated and dynamically allocated to authorized users” [161]. Since every CV can be a computing unit, we consider VC as distributed computing. There are three major objectives of VC: (i) it provides low-cost computational services to the authorized users (e.g., vehicle drivers); (ii) it helps minimize road traffic congestion, travel time and accidents; (iii) it offers real-time and low energy consumption services of software, platform, and infrastructure with QoS to drivers. Architectures of VC can be classified into two categories, dynamic VC and static VC.

Regarding dynamic VC, papers [163, 165, 202, 203, 164] proposed a distributed computing architecture that provides computation services in dynamic vehicular environments via managing the idle computational resources on each vehicle and utilizing them efficiently. This proposed architecture contains three types of vehicles named requesters, processors, and forwarders. Vehicles that generate jobs are requesters, while others that are responsible for processing these jobs are processors. Forwarders are responsible for relaying the generated jobs to nearby available processors. In addition, a vehicle may be a processor for one job and also be a requester for another job that it does not have sufficient capacity to process. Paper [168] investigated a VC service, content-based routing, that allows VC applications to store, share, and search data within the cloud. Regarding static VC, papers [178, 166, 162, 169] proposed a static architecture of VC to augment the computation and storage power of fog computing. Under this static VC architecture, a pool of smart vehicles parked at a shopping mall, or parked vehicles on the roadside are composed of a computing cloud. In addition, paper [198] investigated the service migration issue among different VCs.

However, the network capacity is considerably limited in a vehicular environment, which may significantly constraint the data sharing and cooperations among vehicles in VC scenarios. In paper [163], a VC framework that focuses on processor discovery and job scheduling is proposed to decrease the job transmission and processing latency and improve the total utility. The proposed framework consists of three submodules: a job queue module, a resource management module, and a scheduling module. The job queue module is responsible for caching jobs in a CV, which avoids channel contention caused by multiple simultaneous job transmissions. The resource management module controls the available on-board computation resources of a CV, while the scheduling module is responsible for communicating with other CVs, determining the job assignment, offloading jobs and receiving feedbacks. In addition to that, a MAC layer protocol named Vehicular Cooperative Media Access Control is proposed in paper [167] to improve the network throughput and decrease the channel collision.

Compared to security systems in traditional clouds that are not associated with vehicles, security systems in VCs face more complicated challenges. In the VC scenario, it is difficult to locate an attacker because it is physically moving with a high speed, which may cause several security issues, such as secure location and localization, authentication, data security, and VC access control [113]. In addition to that, attackers in VCs can pretend to be both computation providers and requesters, which increases the complexity of designing a secure scheme to identify the attackers in VCs. Furthermore, security schemes for VCs must be capable of overcoming a dynamically changing number of vehicles [138], where the number of vehicles in a CV may dynamically change due to the traffic volume, time, terrain, etc. Lastly, security issues in VCs includes security of both networks (e.g., VANET, V2V, V2I, etc.) and cloud computing. Although some security approaches for vehicular networks are applicable in VCs, few specific solutions are developed for VCs. Therefore, in order to make the VC become reality, the challenges of assuring trust and security in VCs need to be addressed [161]. However, to the best of our knowledge, few of existing work investigate the security challenges presented above.

Papers [170, 171, 115, 208, 172, 174, 173, 209, 176] propose to merge VCs with cloud computing to form a hybrid computing architecture, as shown in Fig. 7, where RSUs act as gateways for VCs to access the centralized cloud. High-speed wired communications can be used for connecting RSUs with the centralized cloud. VCs are further divided into two cases, moving VCs (i.e., a cluster of vehicles on the road) and static VCs (i.e., a cluster of vehicles in a parking lot). For example, some vehicles may need specific applications that require a large number of computing resources or storage space. Therefore, vehicles that have unused storage space can share their computing resources or storage space as a cloud-based service. In addition, in a VC, vehicles can be either the service providers to enrich existing cloud services by providing various on-road information or be the service consumers to enjoy existing centralized cloud services. Therefore, a user can acquire cloud services from either the centralized cloud or the distributed VCs. In papers [175, 120], the proposed architecture consists of two hierarchies, permanent cloud, e.g., a powerful and stationary server, and temporary clouds, e.g., vehicles and drivers’ devices. In the permanent cloud, three different types of services (i.e., infrastructure-as-a-service (IaaS), platform-as-a-service (PaaS), and software-as-a-service (SaaS)) are provided for CVs. The permanent cloud has powerful and stationary computing and storage capacities and can provide computing, storage, and network resources to the CV system entities. While in temporary clouds, CVs, vehicles’ on-board modules, and the drivers’ devices are temporarily integrated together to expand the computing capacity. However, this architecture does not meet some requirements such as scalable, reliable, and secure in a large-scale deployment, especially considering vehicle mobility and dynamic participation of mobile computing resources.

In addition, paper [173] proposed a four-layer hybrid architecture which consists of end users, mobile buses, public infrastructures, and remote cloud, as shown in Fig. 8. Each layer differentiates each other from communication capacity, computation capability, and their inherent properties (e.g., buses have mobility). There is an AP near each bus station, and it serves as the gateway from the users to different kinds of computing resources via the corresponding network interfaces. When a task arrives, users will transmit the task directly to the nearby AP via the wireless network. Then, the collected tasks by the AP can be further offloaded to the mobile fog (i.e., buses in the figure), the fixed fog/cloudlet, or the cloud. The user layer includes people who wear intelligent devices such as smartphones or intelligent glasses, as well as sensor devices such as roadside cameras. They will generate tasks and may not have enough computation resources to execute these tasks. End users may connect to the APs near the bus stations using different network access technologies. The mobile fog layer is formed by a large number of mobile buses that are routinely driving in the city. When buses stop at a station or pass through a station, they will connect to the APs using the millimeter wave technology, offloading tasks from APs. Since these mobile fog nodes are usually located close to end users, they can achieve low latency for uploading and downloading tasks. Once a bus completes the computation tasks, it will return the results directly, if the AP is still in the effective transmission range; otherwise, it will transmit the results to the next AP. The fixed fog layer refers to the infrastructures, such as smart buildings, parking lots, or BSs which usually have fixed locations. Compared with the mobile fog layer, the fixed fog layer may have higher computation capability but longer communication latency. The fixed fog nodes can connect with the mobile fog through the WLAN created by the APs. The cloud layer refers to the cloud data center. It can be physically located in remote areas far from the users. It is usually equipped with powerful computation units but with a high cost of access delay. In traditional cloud applications, users can offload computation tasks to the cloud data center via the wide-area network (WAN). The characteristics of different layers are summarized in Table XI.

As shown in Fig. 8, the concerned task scheduling period starts when there are buses approaching a bus stop. In case there is no bus or vehicles approaching the stop, the tasks will be processed by the fixed fog or the cloud. According to IEEE 802.11ad standard, time division multiple access (TDMA) can be adopted for mmwave transmission among devices [214]. When two transmitters (i.e., buses) are close at the bus stop, the corresponding MAC protocol of the adopted wireless access network can manage the interference by coordinating the transmission among different devices towards two transmitters [215]. In addition, highly directional antennas can greatly decrease the interference of concurrent transmissions among mmwave transmitters. Therefore, the interference caused by two buses transmitting simultaneously can be managed. Note that the number of the allocated slots for a task determines the transmission rate of the task. When a bus is approaching a bus station, the achievable data rate increases rapidly.

Security issues in hybrid computation-enabled architectures include security challenges in both computation-aided and distributed computation-enabled architectures, such as trustworthiness between the VC members, misbehaving vehicles, vehicular privacy, etc. Although some of these security challenges are investigated in VANETs and VCs, there are still no solutions that treat all these challenges. For example, in a hybrid computation-enabled architecture, a vehicle can interact with a lot of entities, such as neighboring vehicles, RSUs, conventional cloud/edge server, etc. The vehicle therefore face more danger from the data stealing, hostile attack, and virus infection. Thus, it is crucial to develop a suite of elaborate and comprehensive security solutions for hybrid computation-enabled architectures.

1. *

   Centralized: Advantages: (i) Under centralized computing architectures, V2I, V2V, V2D, and V2C data sharing are all available for CVs. For example, in paper [143], a metropolitan air quality monitoring service is proposed, where CVs act as the air quality data gathering sensors. Vehicles offload their gathered data to a centralized cloud for performing an air quality estimation. (ii) In addition, centralized architectures provide relatively stable network connections, where CVs can obtain cloud services from anywhere in the world at any time. (iii) Furthermore, centralized computing architectures usually offer greater security over decentralized systems because all of the user data is stored in a central location.

   Disadvantages & Challenges: In centralized computing architectures, all of the shared data is directly transmitted to the remote cloud server or MEC servers without any pre-processing or filtering at local. In spite of potential advances in wireless communication technologies (e.g., 5G), “the bandwidth required for efficient transmission of such a big volume of data is not guaranteed due to a wide range of logistical, political, and geographical factors” [117]. For example, “assuming 20 GB per month per vehicle and three million vehicles ($12\%$ market share and $25\%$ regional ratio of 100 million vehicles), 60 petabytes of vehicle data will come to the cloud every month. Assuming the data transaction rate at the cloud is 10 GB per second, it will take 70 days just for the transactions” [128]. Therefore, the constrained bandwidth will be a bottleneck for most vehicular services/applications that transmit a large amount of data or require frequent data sharing.

2. *

   Distributed: Advantages: Under distributed computing architectures, direct wireless communications from vehicle to vehicle allow to data to be exchanged fast even where there is no communication infrastructures such as BSs of cellular networks or APs of WLANs. Defined specifically to VANETs, the DSRC that operates in the $5.9$ GHz band is intended to provide high-speed and secure wireless communication for V2V and V2I communications. The U.S. Federal Communications Commission (FCC) allocated $75$ MHz bandwidth at $5.850$-$5.925$ GHz spectrum band for DSRC, while ETSI allocated $70$ MHz in the $5.855$-$5.925$ GHz band. The DSRC can support a CV with a speed up to $200$ km/h, covering a range of $300$ m and reaching up to $1000$ m, and the default data rate is up to $27$ Mbps [108, 107]. IEEE 802.11p wireless access in vehicular environments (WAVE) is the specification of DSRC [216]. In addition, there have been extensive research work to explore communication properties of DSRC [217, 218, 219, 220].

   Disadvantages & Challenges: (i) Under distributed computing architectures, CVs only possess very limited data transmission approaches, such as V2V and V2D. Furthermore, the small effective network diameter of VANET leads to a weak connectivity in the communication between CVs. These significantly constrain the data sharing efficiency in many situations. For example, if a CV, named “red”, in vehicle cluster A intends to share its information with another CV, named “yellow”, in vehicle cluster B, its data can only be successfully transmitted to “yellow” through multi-hop V2V connections. Furthermore, in urban areas, the line-of-sight (LOS) path of V2V may often be blocked by buildings at intersections. (ii) One of the major challenges of distributed computing architectures for CVs is the instability of communication connections among CVs. Because of the high mobility, there is no guarantee on the behaviors of the vehicle and the highly dynamic topology results in frequent changes in its connectivity. Therefore, the connection between two CVs can be quickly interrupted when they are transmitting data. Additionally, high vehicle mobility also leads to Doppler effects, which may cause severe wireless loss. (iii) In addition, there are several challenges that threaten CVs’ security under distributed computing architectures. Providing secure network connections in distributed architectures is more difficult than in centralized architectures because of the high mobility of CVs [110]. For example, in VANET-based distributed architectures, the security issues can be classified into five categories [113]: authentication [221, 222, 223], non-repudiation [224, 225], confidentiality [226], verification of data [227], and localization [228, 229].

3. *

   Hybrid (Computation-aided): Advantages: Hybrid architectures also offer comprehensive data sharing approaches including V2I, V2V, V2D, and V2C.

   Disadvantages & Challenges: (i) One of the major characteristics of VCs is the high mobility, which may cause several challenges in different fields including routing protocol designing, security, and data transmission reliability, especially in hybrid architectures. (ii) Due to the large diversity of networks in the hybrid computing architecture (e.g., mobile ad hoc networks, wireless sensor networks, VANETs, etc.), comprehensive policy and operational management should be established. For instance, we are expecting that several types of networks will emerge and they will interact with each other seamlessly. (iii) Security issues in hybrid CV systems are more complicated than in other networks (e.g., wireless sensor networks) due to the high mobility of CVs and the large diversity of networks. For example, it is difficult to verify the integrity of messages and authentication of users when vehicles move fast [113]. Therefore, security issues in both network (e.g., VANET) and transmission (e.g., wireless communication channel) layers in hybrid computing architectures need more consideration.

4. *

   Hybrid (Computation-enabled): Advantages: Localized computation resources such as fog nodes and on-board computing units can pre-process the collected data of CVs. Thus, the collected data can be aggregated/processed/filtered before uploading to the cloud, which reduces the offloaded data volume and meanwhile improves the efficiency of the network.

   Disadvantages & Challenges: (i), (ii), and (iii) presented in Hybrid (Computation-aided).

1. *

   Centralized: Advantages: (i) The remote cloud server and MEC servers under centralized computing architectures possess large storage space and huge computation power. Therefore, centralized computing architectures can provide CVs the facility of unlimited computing power accessible on demand, which is suitable for vehicle applications that need large storage capacity and computation power. (ii) Under centralized architectures, CVs are capable of accessing servers’ computation resources from anywhere at any time. Because of the high mobility characteristic of vehicles, this benefit is significantly important for CVs to obtain stable and uninterrupted services. (iii) Furthermore, centralized architectures do not require early planning of computation resource provisioning.

   Disadvantages & Challenges: As we mentioned previously, the current trend of concentrating data processing at centralized cloud servers will cause huge data transmission traffic. This will directly incur unnecessarily long response delay and in turn, will increase the computation latency [128]. Therefore, CV applications, such as intelligent driving and high resolution map creation and distribution [128], which are latency-sensitive cannot be handled effectively under centralized computing architectures.

2. *

   Distributed: Advantages: In centralized architectures, the cloud data server consumes a huge amount of energy each year [230]. In contrast, distributed architectures enable green computing by efficiently using the spare on-board computation resources on CVs. Since vehicles are self-powered, efficiently using on-board computation resources on CVs will help minimize the energy consumption of cloud/edge servers.

   Disadvantages & Challenges: (i) The computation resource density in distributed architectures varies depending on the traffic density, which can be very high in the case of a traffic jam, or very low, as driving on the highway. However, when a vehicle stops or moves slowly, it does not require more computation power (e.g., a CV does not need to rapidly process or refresh its high-definition map for supporting intelligent driving). In contrast, when the vehicle is moving fast, it requires much computation power to accurately, frequently, and rapidly process its sensors captured data (e.g., accurately localize its surrounding dynamic objects). (ii) A VC has the issue that its computation resource availability is not reliable. Since the computation resource availability in a VC and the behavior of each CV are not guaranteed. For instance, vehicles might unexpectedly leave or join a VC. In fact, this is also one of the major differences between the distributed architecture and the centralized architecture. In addition, different CVs in a VC usually have different characteristics or capabilities (e.g., processor speed and memory volume) upon their manufacturers, models, and applications. For example, CVs can be classified into private vehicles and buses. Private vehicles have unpredictable routes, whereas buses have fixed routes. (iii) Because of the variable computation resource density and the unreliability of the computation resources in distributed architectures, it is critical for every CV to frequently monitor its cooperated CVs and to periodically search for potential candidates.

3. *

   Hybrid (Computation-aided): Advantages: (i) Compared to the centralized and distributed architectures, the hybrid architecture provides CVs with more potential computation sources, which improves the resilience of CV systems. (ii) The hybrid architecture also can provide low response latency for CVs that require running latency-sensitive applications. Instead of offloading collected data to a remote cloud, CVs can choose to process their data at neighboring fog nodes or RSUs.

   Disadvantages & Challenges: (i) High deployment cost of heterogeneous computing infrastructures. (ii) Interoperability and standardization required. Since the hybrid architecture is based on diverse stationary and mobile computing resources, many steps should be taken to address the interoperability challenge to allow these different entities to work together. In addition, standardization is a potential solution to address the interoperability issue so that a consensus can be reached among developers, vehicle manufacturers, etc.

4. *

   Hybrid (Computation-enabled): Advantages: (i), (ii), and (iii) presented in Hybrid (Computation-aided) and greenness presented in Distributed.

   Disadvantages & Challenges: (i) and (ii) presented in Hybrid (Computation-aided). (iii) A comprehensive task scheduler is required in hybrid computing architectures. As we presented above, a variety of computing units/sources coexist in a hybrid CV system, which reflects the situation in which a task generated by a CV can be processed by different approaches. Such a situation will lead to an issue that what is the best computing approach for each task or what are the criteria of the processing decision making. From the perspective of the CV application, for instance, it may depend on the complexity of the application, requirement of the response latency, transmitted data volume, required computation capacity, energy consumption, monetary cost, etc. While, from the perspective of the system, it may depend on the position and role of the CV in the hierarchical system deployment, load of each computing sources, accessibility of each computing units, etc. Therefore, designing a comprehensive task scheduler that can cover multiple criteria from different perspectives is essential and challenging in hybrid architectures.

1. *

   Centralized: Advantages: (i) Since all vehicles under centralized computing architectures are covered by a cloud server or several MEC servers, it is efficient to monitor the presence and experience of CVs under this architecture. For example, global warning services, such as warnings of traffic jam, accident, road condition, and predicted weather event, can be provided by the remote cloud server. (ii) In addition, by using the centralized cloud storage that can gather a huge amount of user historic information, various governmental and private agencies can use the gathered data to provide various monitoring and warning services or perform diverse studies. Furthermore, control services that need historical information of CVs can be implemented under this architecture. For example, the centralized server can make the control decision on whether a vehicle should enter the automated driving mode or not, based on not only the current collected data about the vehicle, such as the speed, but also the historical data, such as road conditions.

   Disadvantages & Challenges: since the centralized servers are located relatively far away from the CVs, the delay of receiving control messages is relatively long, which means that centralized computing architectures may not be suitable for delay-sensitive services, such as autonomous driving.

2. *

   Distributed: Advantages: (i) Comparing to the centralized architecture, the distributed architecture, such as VC, has great advantage in emergency warning scenarios. Disaster management, for instance, is an emergent warning application, where CVs send or broadcast the warning messages to all resources such as nearby vehicles and authorized authorities in case of any disaster. (ii) Distributed architectures provide flexibility for CVs to design user-self-defined monitoring/warning/control services. For example, it is easier for a user to set up a driver’s health monitoring service with its own concerns, such as driver’s medical records and habits. In addition, this monitoring service can still work even without an Internet connection.

   Disadvantages & Challenges: (i) Unlike the previously described global warning services under centralized computing architectures, distributed computing architectures can only provide one type of local warning services, driver health warning, because of the constraint of VC coverage. (ii) The limited storage space constrains the ability of storing and analyzing a large amount of user historic data.

Localizing data traffic: In order for OEMs to provide diverse and personalized connected services, it is essential to collect various types of sensor data from each vehicle such as camera images, acceleration/deceleration or hard braking. Those services may include personalized insurance service, predictive vehicle maintenance, high-definition (HD) map generation, intelligent driving and so on. Some forecasts that each vehicle will generate data in the order of gigabytes every day. If those data are directly sent to cloud servers, it will add a huge amount of network traffic in the existing infrastructure. Even though a vehicle can produce the raw data at a high frequency and volume, it may not be necessary to forward every single data. For example, some data are only meaningful in a particular geographic region (e.g., a hard braking signal to avoid frontal collision in the highway); it may be also fine to send data less frequently to the server (e.g., analyzing drivers’ behavior for a personalized insurance service). One research direction is to precisely characterize various types of connected services to determine the sufficient amount of frequency and volume of data enough to achieve service-specific QoS. Such knowledge can then be distributed across the multiple layers of the system architecture to appropriately localize data to avoid excessive and unnecessary traffic in the overall network.

Highly heterogeneous vehicular networks: Due to the diverse QoS requirements of vehicular services and unique characteristics of vehicular network connections, such as high mobility, frequent topology changes, and unreliable connectivity, homogeneous radio access technology (e.g., either DSRC or LTE) cannot satisfy the performance requirements of vehicular services. For example, DSRC is capable of providing low round-trip time (RTT) (i.e., below 10 milliseconds [130]), which satisfies the latency requirement of safety-related vehicular services. However, its limited peak data rate and signal coverage decline the efficiency and reliability of data sharing among CVs and infrastructures. Furthermore, although cellular networks such as LTE offers a much wider signal coverage and a higher peak data rate for CVs compared to DSRC, stringent latency requirement cannot be guaranteed due to its long RTT (i.e., over 300 milliseconds [130]). Therefore, in order to meet different QoS requirements of vehicular services, future vehicular networks tend to be highly heterogeneous. However, several open issues are imposed by the network heterogeneity. (i) Developing radio access method selection, link adaptation, and radio resource management schemes in heterogeneous vehicular networks are crucial and complicated. Thus, the main challenge is how to balance performance and complexity. (ii) Maintaining a seamless connectivity across diverse radio access technologies is complex, which includes inter- and intra-radio handoffs. Most traditional handoff approaches for cellular networks are centralized and provide a single trigger mechanism, which is not sufficient to support distributed and hybrid vehicular environments.

Support mobility in heterogeneous architectures: Mobility is one key feature of the automotive system; a vehicle continuously moves around different regions expecting to receive services seamlessly. However, it is practically difficult to expect the vehicle to communicate with a uniform infrastructure or architecture everywhere for several practical reasons. For example, different organizations (e.g., government, network providers, OEMs) may want to own their infrastructures exclusively due to security concerns; an infrastructure may be installed in a limited area only due to a budget issue or lack of profitability. Such situations will cause a vehicle to communicate with very different infrastructure as it moves. Another research direction is to develop a way to provide seamless CV services across heterogeneous infrastructure or architectures. This requires each infrastructure to provide the open interface that specifies which types of services can be supported or prohibited more explicitly, instead of hiding such information. Then, each CV service provider may have a better understanding as to where the infrastructure support for the service is available, and if it is not available, what are the alternative way to provide the service, such as service hand-over and migration. Making such interfaces available across different organizations will also boost the adoption of CVs.

Computing resource management: Each individual subsystem (e.g., cloud servers, fog/edge servers, RSUs, and vehicles) in the surveyed architectures has a different limitation of the computing resources such as memory or computation power. On the other hand, each connected service may require different types and sizes of computing resources. For example, the over-the-air (OTA) update service may require fog servers to reserve a portion of memory storage that can be used to distribute the latest version of the software patch to the vehicles in a particular area; the HD map generation requires relatively a higher computation power to synthesize local maps at real-time. Guaranteeing QoS of various services should take into account such computing resource constraints imposed on each subsystem. The relevant research direction is to manage computing resources in a more rigorous way, and the example research may include the following. If a particular area experiences a relatively higher request on a certain service, one can allocate more resources on the infrastructure in the area (resource allocation); depending on the current workload on the infrastructure, one can decide to (or not to) offload a certain computation (data/computation offloading); one can predict the future usage of the resources by reading the past trend of the requested connected services (resource provisioning). If this type of research is combined with the surveyed architectures, we expect a better quality of CV services can be provided.

Other hybrid architectural alternatives: As we discussed in Section IV, architectural design is significantly important for acquiring diverse system features and satisfying different service requirements. Although a large number of architectural alternatives for CV systems have been proposed in existing work, there should be more possibilities, especially for the hybrid architectural design. For example, as shown in Fig. 9, most of existing hybrid architectures choose to put the centralized computing in the highest layer, which acts as a remote cloud. However, besides this stereotypical architectural design, we may have another alternative, as shown in Fig. 9, where distributed computing is in the highest layer in the CV system. The centralized computing units are located closer to CVs (e.g., RSUs) rather than located in remote cloud servers. Such design will (i) make the CV system fully utilize the computation and storage capacities of centralized computing units; (ii) reduce both communication and computation latency. Furthermore, distributed computing units, such as spare computing resources in neighbouring CVs and fog nodes, can be combined as a virtual localized controller which is responsible for allocating the computing resources in its governed centralized computing units to CVs. Therefore, based on the above discussion, there might be a set of interesting research opportunities in CV system computing architectural design, which are not yet tackled in the reviewed literature.