DoRA: Domain-Based Self-Supervised Learning Framework for Low-Resource Real Estate Appraisal

Real Estate Features. Real estate has 39 features, including 16 categorical features and 23 numerical features to represent the metadata, for instance, location, the layout of the real estate, the current condition of the real estate, and household facilities.

PoI Features. The original data is in spatial distribution format. It was collected from a third-party map information company. We designed a PoI converter to transform the data into a tabular format by dividing the facilities into YIMBY (Yes In My Back Yard, i.e., desirable adjacent public facilities) facilities, e.g., park, school, and NIMBY (Not In My Back Yard, i.e., non-desirable adjacent public facilities) facilities, e.g., power station, landfill. Then, the number of PoI of the real estate property was calculated with the Euclidean distance. For instance, the feature YIMBY_100 denotes the number of YIMBY facilities within 100 meters.

Economic and Geographical Features. We added 7 external socio-economic features based on the real estate transaction quarter to represent the global view, including the house price index, unemployment rate, economic growth rate, lending rate, land transactions count, average land price index, and steel price index. We also incorporated the land area and population density by the town name to consider the number of residents and the demand.

Embedder. The heterogeneous input features can be categorized into 4 types: numerical real estate features, categorical real estate features, numerical economic and geographical features, and numerical PoI features. The numerical real estate features $NR_{i}$ of $i$-th real estate are embedded as follows:

where the dimensions of $E_{NR_{i}}$ and $NR_{i}$ are $d_{NR}$ and 23, respectively.

Encoder. To encode the $i$-th embedding of real estate, the encoder is introduced to learn the contextualized feature representation $Z_{i}$ during both the pre-training and fine-tuning stages:

where $Z_{i}$ is a $d_{Z}$ dimension vector. Since existing work on SSL for tabular data mainly focuses on the strategies of pretext tasks (Yoon et al., 2020; Ucar et al., 2021), we also adopt MLPs with $N$ layers to align the comparison, where the number of layers $N$ of the encoder is optimized to ensure it is able to perform competitive performance of the pretext task (Section 5.2). The output of the encoder can then be used to learn prior knowledge from the pre-training stage and downstream tasks from the fine-tuning stage.

Intra-Sample Pretext Task: Located Town Prediction. To enrich the feature representations from the unlabeled set, an intuitive method for designing the pretext task is to add noise and reconstruct the input, e.g., (Yoon et al., 2020), which treats all features with the same importance but neglects the domain knowledge of the meaning of the features. Inspired by a recent study (Lee et al., 2021) on the conceptual connections of features between pretext and downstream tasks benefiting the downstream tasks, we introduce a domain-based pretext task: predict the located town of the given real estate, which can also be adopted in various geographic-related tasks (e.g., house rental). The input features of the real estate do not include the located town to avoid cheating. In this way, the model is equipped with fine-grained domain knowledge to distinguish what might be the composition of real estate for each city, which benefits downstream tasks with limited transactions.

Pre-Trained Predictor. The pre-trained predictor takes the feature representation of the $i$-th real estate property as the input, and predicts the corresponding town $\hat{Y}_{i}\in\mathbb{R}^{N_{Y}}$ with an MLP and the softmax activation function, where the number of $N_{Y}$ is 350:

Pre-Training Loss. During the pre-training stage, the embedders, encoder, and pre-trained predictor are jointly trained in the following optimization problem:

where $\alpha$ adjusts the weight between the two losses, $y$ is the ground-truth of the located town, and $M=\{Embedder_{NR},Embedder_{CR},Embedder_{Econ\&Geo},Embedder_{PoI}$, $Encoder,Predictor_{pretext}\}$. $\mathbb{L}_{ce}$ is the cross-entropy loss:

where C denotes the number of the unlabeled set.

Inter-Sample Contrastive Learning. Since cross-entropy is sensitive to noisy labels and lessens generalization performance (Zhang and Sabuncu, 2018; Elsayed et al., 2018), we extend contrastive learning (CL) to incorporate label information to consider the similarities and discrepancies between real estate across towns. $\mathbb{L}_{cl}$ is defined as:

where $P(i)$ is the set of indices of all positive pairs, $A(i)$ is the set of all positive and negative pairs, $\tau$ is a temperature parameter, and $I$ is the instances (i.e., real estate) in a batch. Positive pairs represent the two instances located in the same town, while negative pairs represent the two instances located in different towns. As a result, $Z_{p}$ is one of the contextualized embeddings that is sampled from the same town within the batch as the positive pairs for CL. By optimizing $\mathbb{L}_{cl}$, embeddings with the same class are closer, and embeddings from different classes are pulled over.

House Price Predictor. To train a regressor for the house price prediction, the feature representations of the pre-trained encoder of the $i$-th real estate property are fed into the house price predictor:

where $\hat{P}_{i}$ is the estimated price.

Fine-Tuning Loss. To optimize the estimated prices of real estate, the pre-trained embedders, pre-trained encoder, and the house price predictor of DoRA are jointly fine-tuned by minimizing the mean square error loss $\mathbb{L}_{mse}$:

where $S$ is the support set and $P_{i}$ is the $i$-th ground truth price.

Implementation Details. The dimensions of $d_{NR}$, $d_{Econ\&Geo}$, and $d_{PoI}$ are 16, the dimension of $d_{CR}$ is 10, and the dimension of $d_{Z}$ is 256. The number of layers of the encoder $N$ is 6 and is designed with hidden dimensions $(2d_{Z},4d_{Z},8d_{Z},4d_{Z},2d_{Z},d_{Z})$ in order. We set weight $\alpha$ as 0.7 and temperature $\tau$ as 0.1. For the numerical features, we impute them with standard normalization. In the fine-tuning stage, both DoRA and other baselines use an MLP as the house price predictor. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) using the learning rate of 0.005, and the batch size is 512. During the pre-training stage, we use all property types of unlabeled sets to learn a pre-trained model to enforce the generalizability and then fine-tune based on various property type datasets. The training epochs of the pre-training and fine-tuning stages are 150 and 200, respectively. All of the hyper-parameters in the experiment were tuned based on the validation set. All experiments were repeated 5 times with different random seeds for sampling support sets to reduce the bias of few-shot sampling and to report average metrics with standard deviations for each evaluation metric.

Evaluation Metrics. Previous work mainly focused on regression metrics for evaluating house price prediction (Zhang et al., 2021; Li et al., 2022). We extended these metrics with hit rate k% (Goodman and Thibodeau, 2003) to measure the accuracy of target properties within a tolerance error percentage k conforming to real-world financial requirements. Therefore, we adopted mean absolute percentage error (MAPE), mean absolute error (MAE), and hit rate 10% (HR10%) to comprehensively evaluate the results.

Baselines. The baselines can be categorized into four groups: 1) Statistics model (STA): Historical Average (HA), 2) Supervised models (SUP): Linear Regression (LR), XGBoost (Chen and Guestrin, 2016), DNN, and DNN with contrastive learning (DNN + CL), 3) Self-supervised models (SSL): DAE (Vincent et al., 2008) and SubTab (Ucar et al., 2021), and 4) Graph-based models (Graph): MugRep (Zhang et al., 2021) and ReGram (Li et al., 2022). The SSL baselines are also pre-trained using the unlabeled set and are then fine-tuned to the house price prediction task.

Model Ablation. We study a comprehensive component ablation with the building dataset in the 5-shot scenario in terms of MAPE. As shown in Table 3, only using the unlabeled set of the corresponding type (building type in row 1) significantly degenerates the downstream performance, which signifies that DoRA is able to leverage the unlabeled set from various types to improve model performance on the house price prediction. Rows 2, 3, 4, and 5 present the sensitivity analysis of different hyper-parameters, which shows that removing contrastive loss and changing the weight of $\alpha$ reduce the performance more substantially. In addition, freezing the pre-trained encoder and using the different dimensions of feature representations also negatively impact the house price performance.

Feature and Pretext Task Ablations. To investigate the relative effects of different feature sources, we evaluate DoRA with full features and its three variants: 1) RF includes only real estate features, which is the basic feature to describe real estate; 2) RF+PoI includes both real estate features and PoI features; 3) RF+Econ&Geo includes both real estate features and economic and geographical features; 4) All includes the complete set of features. Figure 3(a) reports the performance with the building dataset in the 5-shot setting. Removing either PoI features or economic and geographical features leads to inferior performance in terms of all metrics. Moreover, PoI features affect the performance more considerably compared with economic and geographical features, which indicates that the neighboring facilities are critical factors for real estate appraisal. These observations suggest that considering only the metadata of real estate is insufficient for house price prediction, while various sources of describing real estate from a global viewpoint enhance the capability of the model. We also examine various pretext tasks as shown in Figure 3(b), which replaces the original pretext task with another pretext task from the real estate metadata. We can observe that all metrics performance of replacing with different pretext tasks is degraded, which empirically showcases the importance of the pretext task objective.

Deployed System. E.SUN Bank is a commercial bank encouraging AI-driven solutions for businesses in Taiwan. In the past, real estate appraisal tasks were primarily manual, high-cost, and subjective to the appraiser. Moreover, it is challenging to estimate real estate if there are limited historical transactions. To that end, we partnered with the fintech team to deploy automated DoRA to perform an online mortgage calculator platform, which requires real estate appraisal for suggesting the mortgage. In the prototype system, the user is required to enter the information (e.g., address, house age, parking space, etc.) of the property that is to be mortgaged. DoRA will then execute an online real estate appraisal by extracting the PoI features based on the house address as part of the inputs. Afterwards, the appraised price will be incorporated with other internal entities to compute the approximate loan.

Case Study. As the distribution of transactions for each city is a long-tail distribution, the great majority of the cities only have a few transactions. Therefore, we simulated three cities with extremely limited transactions and compared DoRA with XGBoost. As shown in Table 4, we can observe that DoRA is significantly superior to the baseline for all metrics, particularly in property types where it fails to appraise real estate with 0 hit rate scores. The performance of the house type in Wugu District shows that DoRA appraises effectively while the baseline deteriorates all metrics more substantially. These cases confirm that DoRA is capable of low-resource scenarios due to the incorporation of the unlabeled set. In partnership with the fintech team, such an improvement definitely improves resource utilization and performance of the real estate appraisal.