Open government geospatial data on buildings for planning sustainable and resilient cities

We identified datasets on all inhabited continents. The majority of the datasets is found in high-income countries, predominantly from North America and Europe (Figure 2). Very few datasets are from low- and middle-income economies in Africa, Asia or South America, e.g. from Columbia or South Africa. Several high-income countries fail to release data, indicating cultural or administrative barriers. In many countries, only a few, usually large, cities are covered, indicating independent initiatives by resource-rich local governments.

Most datasets are fully accessible without cost, registration, or enquiry. However, access to data is not always without issues, and various practices in term of openness and access to the data make the collection of the data challenging and time consuming. Barriers to access include broken links, invalid files, and complex APIs — diminishing accessibility to stakeholders without advanced technical skills. For example, certain download services e.g. Web Feature Service (WFS) are primarily meant to download small local samples and may require convoluted workflows to download large areas. Further, some datasets advertised on open data portals turn out not to be open data; a common practice is to display data in a web map without download options.

International visibility is important for global or comparative studies including multiple geographies. In non-English speaking countries, open data portals are rarely available in English, inhibiting the identification of data for comparative studies. Another impediment are attributes, metadata and documentation available only in local language(s). While it is understandable that governments publish the data according to the official language(s) of the country, these datasets may be left out of international comparative studies due to unintelligibility.

There tends to be an order of magnitude more datasets at the city level than on regional or national levels (Figure 2). Datasets at the national levels seem to be currently only existing in Europe, where there has been several important releases in the last year, with the exception of Japan and New Zealand. In several instances, datasets do not cover all buildings in the jurisdiction, which may not be indicated in the metadata. For example, some datasets have large swaths of land entirely omitted, while some include only buildings of a particular class or with certain criteria (e.g. only commercial buildings or those with the largest footprint areas).

About half of the datasets (53%) only inform about the 2D geometry of buildings, whereas the other 47% contains at least one attribute about buildings (Figure 2). The most common attribute (about one third of datasets) informs on the type of the building. Building height, elevation, address, year of construction, and number of storeys follow as additional less frequent attributes. The relatively large share (28%) of datasets containing either height or storey information indicates a potential for generating 3D city models using extrusion [22]. A few datasets, e.g. in France and Finland, go beyond the most common attributes to include information on building characteristics such as roof type and amenities including electricity, water supply, heating system, elevator, etc.

Most datasets contain polygons representing the building footprint, but in a few cases, especially in low-income countries, only coordinates or addresses were provided, which are of limited use in spatial analyses. In some cases, different levels of detail are available (e.g. points and polygons) with the lower level of detail available for free, while the higher is available at a cost. The footprints may contain more or less architectural details (see Figure 3). Numerical attributes can also have different level of precision, due to aggregation of different datasets with different estimation methods. This may be indicated in the data, e.g. for building heights in France, where the altimetric precision is given. Categorical attributes are also far from being harmonised and may have quite different granularity. For example, building type may contains only few classes (e.g. public/commercial/residential), but also 10+ classes (e.g. school, museum, hospital, etc.).

Nearly all (more than 90%) datasets provide information on the year of last update. However, due to the difference in website configurations, as well as user experience/interface design (UI/UX) formats, it often takes time to locate where updates and acquisition methods are described. License status is also often not straightforward. Sometimes, there are links provided to government-specific data sharing rules due to the sensitivity of the data. A number of the datasets (28%) is openly available under the Creative Commons License for sharing. The metadata may not be conclusive in describing what certain attributes mean.

Half of the datasets has been either created or last updated within the last one year (Figure 2). Practices regarding updates vary dramatically. On the one hand, there are instances released a decade ago with no update since then. Further, some metadata claim yearly updates, while we were not able to find evidence of this. On the other hand, some datasets are updated on a weekly basis, e.g. Estonia and Netherlands. However, we have found very few instances of versioning, e.g. in New York City, enabling to study urban development.

Validating the geometric content of the datasets reveals that they are virtually free of geometric errors. Less than 0.1% of buildings have invalid geometry and more than half of the datasets do not have a single error. Such quality is in stark difference from 3D datasets on buildings, in which the share of errors is often more than an order of magnitude higher [39]. Nevertheless, geometric consistency does not warrant the positional accuracy of the data.

Although there are efforts to develop building information standards, using international standards is not a common practice for 2D building datasets, limited to a small share of the datasets encountered. The main example is the INSPIRE standard, which led to harmonisation of practices in the EU on dimensions such as metadata, licences, download methods, or data encoding. While some data are modelled according to national standards, the lack of international standards impacts interoperability, as it forces users aiming to perform comparative studies to verify the above-mentioned dimensions in each dataset.

About 86% of the datasets are available in ESRI shapefile, and the remaining datasets are stored in more than five different formats. While widely supported, shapefile is a proprietary format. Some datasets are available in custom formats, greatly reducing accessibility. 40% of the datasets are available in two or more formats, out of which an open and non-proprietary formats are common.

Building footprints can be represented either as whole blocks, as whole buildings, as buildings parts, and in some cases several layers are provided (e.g. in Spain). In most cases, it appears that whole buildings are represented. What defines an individual building is often unclear and may differ across regions, an issue that may cause interoperability issues and noisy results in large-scale models.

The OSM community has leveraged the open nature of many government datasets and integrated them into OSM via bulk imports. We find that only a quarter of government datasets we have identified has been integrated into OSM. Data imports usually serve well rural areas, when a dataset is released at the state or regional level, as these tend to be undermapped in OSM in comparison to urban areas [40]. Figure 3 shows an example where buildings in OSM has been sourced from the government data. Imports in OSM entail a stringent process that involves investigating potential impact of the mass import on the existing data and suitability of the licence [41, 42]. It is unclear why more government datasets have not been taken advantage of. In some areas, a possible explanation is that government data, while available, is outdated, and OSM already provides data of higher quality.

In many instances, government data is distributed by a private company, e.g. hosted on their servers or integrated in their products. While it appears that such data are certainly originating from a government body, we could not find any official repository hosting the same instance, possibly hinting at cases of outsourcing, or cases that the government dataset first released by a government, with the company mirroring it on their services. The dataset may have been removed from their repository due to being out of date, without a newer counterpart released, and with the commercial host remaining available; or the data may not open, but has been purchased by a company.

We find compelling cases where universities are complementing government activities with multiple purposes: standardisation, validation, and enhancement of data. First, they are regularly involved in developing standards and harmonisation approaches [7, 18]. Second, universities provide methods and tools for validation, quality control, and benchmarking spatial data [43]. Our paper is also an example how academia can contribute to gauging the data. Finally, universities leverage their expertise to integrate data, multiplying their instances. For example, in the Netherlands, a research group has conflated multiple government datasets, and released the processed and enriched dataset as open data [27]. Further, being research-driven users of data, they are spearheading new use cases, possibly leading to influencing future data releases and enhancing data requirements.

Buildings represent key challenges for urban sustainability, resilience, and transformations [1], and new data are required to study relevant determinants and design solutions involving buildings around the world [44, 45, 46, 47, 48, 49, 50]. Key gaps in knowledge include the sustainability outcomes from different urban forms, the impact built-up land use on other systems [45], and how to translate this knowledge into actionable policies [45, 46, 49].

In this context, attaining a complete coverage of building footprints globally is a key priority, but our study indicates that open government data remain currently far from such objective. The lack of building attributes limits the range of questions that can be addressed with the data, and the number of dimensions that can be examined simultaneously [47, 48]. Attributes may not be necessary for applications such as urban morphology based solely of spatial arrangements, but can substantially enrich them for social, environmental, and economic modelling. Thus, it is necessary to achieve a high coverage of basic attributes, e.g. building heights [45] as well as increasing the availability of important by rarely available attributes such as building envelope materials.

Well-designed buildings and neighbourhoods, as part of the critical infrastructure for basic well-being, are key to achieving multiple Sustainable Development Goals in the Global South [50, 51, 52]. In particular, the construction and operation of new buildings in fast urbanising areas in Asia, the Middle East, and Africa are projected to consume most of remaining carbon budget in business-as-usual scenarios, thus early strategies for low-carbon developments are pivotal [51, 44]. Although cities in the Global South face important constraints, they also offer an overlooked capacity to innovate and experiment for sustainability [52].

Our study reveals that there are barely any government sources from the Global South and the fastest urbanising regions, which is an important concern. Building data from the Global South is necessary, given that they exhibit distinct and statistically different issues from the Global North [52], and to enable the assimilation of narratives and local knowledge with technical data [44]. Up-to-date information and available archives are of particular importance in these regions to study urbanisation trends, see for example the NYU Atlas of Urban Expansion (http://www.atlasofurbanexpansion.org/). Finally, data on informal settlements is difficult to generate but critical [44].

There is a broad agreement in the urban sustainability literature on the need for more studies across scales and geographies to tackle regional and global sustainability challenges [53, 1, 48, 50, 51, 54]. The lack of such studies leads to various knowledge gaps. Much of current research is at the individual city level or aggregate level [53], while there is a need for studies spanning multiple geographic and administrative scales. These enable to generate and act on clear targets of desired diversity at different scales [53]; they address the spatial scale between human and natural systems, as well as the trade-offs between social and environmental outcomes [48].

Enabling such research requires building data that is easily accessible both for the local and global user, with minimal barriers regarding data access, processing or licensing. Our analysis indicates that the multiplicity of small-scale datasets with diverging practices generates substantial barriers even for experienced users. Further standardisation and consolidation of datasets is needed [50]. The lack of data for in many world regions is also a key bottleneck for enabling comparative work [48] and city typologies [55, 56] that help understand common patterns and diversity, and find solutions that may transfer across contexts [50, 54].

The very small number of datasets found in low- and middle-income countries seems to indicate that creating and releasing data often remains a luxury. Cultural and administrative barriers include technical, organisational or budget-related issues. These may include the cost of training staff internally or need to hire sub-contractors, difficulty to gather information that is fragmented and not necessarily digitalised across services, or concern of divulging sensitive information. In certain regions, there may not be clear perspectives of rapid progress and therefore a need for other actors to step up. But rapid progress can be observed, especially if data products are already existing, for example for internal use or commercial purposes, as this has been the case recently in the European Union with the EU directive 2019/1024, which increased pressure on governments to release data.

In Table 1, we provide recommendations that can substantially increase the value of the data for users. Many of them are easy to implement, e.g. providing basic information through metadata and naming, or publicly archiving previous versions of a dataset instead of keeping only the latest version online. Some other best practices can represent a larger expenditure, e.g. achieving a complete coverage of key attributes, but such investments offer large societal returns [57, 58]. When providing complete datasets for attributes may be too big of an investment, recent data generation literature, e.g. using machine learning, also shows that partially complete datasets are already beneficial, given that even limited local data can be used to estimate missing data [16, 59].

Many of the most impactful applications for urban sustainability come by fusing datasets on building and on other land uses, or possible satelite imagery [45, 47]. When several of such datasets have been gathered by an administration, releasing them together and faciliating matching, e.g. with common ‘id’ increases their individual relevance. A role model here is for example the French cadaster that releases openly a single structured data product, BDTOPO, which includes buildings, various infrastructures including transportation, electric grid or water pipes. When these datasets have been generated by other entities, interoperability could be mitigated by widespread implementation of best practices.

Non-governmental entities can have an important role in improving data quality and convergence in practices, as demonstrated in the European Union by the project INSPIRE. Initiated in 2006, this project provided technical guidelines and support to EU state members as well as a repository where user can access the data in centralised manner. Our results suggests that the EU is the region that offers the best data coverage and quality overall, although there is still substantial room for progress in all dimensions analysed. Similar initiatives at the regional or global level, for example the ISO standard 19115:2013 ”Geographic Information – Metadata” may help improve the current data situation. Leveraging local partners, such as universities or other research organisations, which have demonstrated their interest and expertise [27], is a promising pathway that could be followed by more governments.

The main advantages of OSM include a globally consistent framework with a uniform and liberal licence, a large community of contributors and increasing corporate editing [61]. This enables OSM to provide rapid updates (see example on Figure 3), flexible new attributes thanks to the freeform tag-value system possibly leading to more detailed data [62], and, a versioning approach generating historical data. OSM plays a key role in the Global South through humanitarian mapping efforts and embedding local knowledge in maps. The main drawbacks of OSM include potential quality and accuracy issues, inconsistent and scarce attributes, low budget overall for maintenance and enhancement of the data services, the fact that OSM is subject to vandalism (although there is an increasingly developing ecosystem for OSM quality control), and generally a urban/rural divide in term of completeness. However, in some cases, it has been noticed that the quality of OSM can be higher than that of government datasets [63, 64].

The main advantages of government data are their potential full completeness in term of footprints and sometimes attributes, homogeneous data collection by a single actor that can ensure quality control, and sometimes an extended set of attributes not available elsewhere thanks to the incorporation of administrative data. The main drawbacks include the potential difficulty to update and correct regularly the data, which together with the lack of historical data can hinder analyses in rapidly developing cities, the fact the data are locally limited to a jurisdiction, often available in scattered repositories with variable licenses and language barriers.

As both government and OSM data have distinct advantages and limitations, they do not conflict and can complement each other (Figure 4). There can be a two-way interaction between governments and VGI [65, 66].

Governments can benefit from OSM and have been engaging with VGI communities. VGI may be used by governmental bodies to supplement or facilitate their operations, as VGI enables governments to capture and integrate local knowledge, which might not be considered by official acquisition processes. For example, in Indonesia, the OSM community was engaged to map buildings in areas where previously such was not available, and the resulting dataset was adopted by the government to produce scenarios for different disasters and offer a tool to develop contingency plans [65, 66]. It may also be useful here to differentiate across governments’ level of development in term of data acquisition and maintenance. In cases where governments have highly-constrained means and that a VGI community is emerging, engaging with and investing in VGI may enable to facilitate government operations at a lower cost while generating some other benefits, e.g. related to education. In cases where both OSM and government have a high coverage, interactions may enable to shape how to develop future data products that of high value for the broader society.

Availability of open government data also seems beneficial to OSM. To illustrate this point, we compared the coverage of OSM and open government data in the European Union (see Supplementary Information). Countries that have highest coverage in OSM tend to have open government data available at the country scale and countries that have the lowest coverage in OSM have no open access to government data. There are rare cases of either high OSM coverage with low official data, possibly due to a particularly active community, and low OSM coverage with high authoritative data, perhaps due to government data being only newly released and not yet integrated into OSM. Another benefit is that using government data as ground truth is the most common way to carry out quality checks of OSM [67, 68, 69].

Ideally, open government data and OSM could be integrated in one centralised repository, providing multiple benefits. First, this combination should contribute towards maximising completeness of geometries and attributes, using the higher level of detail available in case of matches. Second, it should enable rapid updates with permanent record of data accompanied by versioning. Third, there should be a streamlined process with uniform data format and harmonisation of attribute values. Such initiative could be undertaken at different scales and could result in higher data availability and quality both for local and comparative regional or global research.

In conclusion, although open government data have become increasingly available and accessible globally, high-quality data remains mostly limited to high-income countries and main urban areas. Data currently available provide often incomplete or basic snapshots of building stocks. There are large data gaps (e.g. no buildings at all for many areas or little information on building attributes) that may be filled by further releases, VGI, or ML-based data generation approaches, and there is a high potential of morphing into integrated data. Large disparities in practices among local and national governments worldwide in terms of several aspects such as update frequency, modelling guidelines, and metadata availability require further government efforts to provide a timely and needed empirical basis for transforming the building stock globally. While it has a limited geographic coverage in comparison to OpenStreetMap, its crowdsourced counterpart, government data usually reign supreme where they are available, often providing the level of quality that is highest among datasets from other parties; but their quality should not be taken for granted, and in some instances there may be better options available. This overview of the trends in open government building data may serve policy-makers, researchers, and practitioners to understand the current landscape of open data on buildings and plan further actions.