ABSTRACT
Classical spatial econometric models excel in interpreting spatial autocorrelation but struggle with scalability on resource-limited platforms. In contrast, deep neural networks offer high predictive accuracy yet demand significant computational resources. To bridge this divide, we propose a unified knowledge distillation (KD) framework that integrates four established spatial econometric models – Spatial Durbin Error Model (SDEM), Spatial Lag Model (lagsarlm), Spatial Durbin Model (SDM), and Spatial Error Model (SEM) – as teacher models to distill spatial knowledge into lightweight neural student models. Enhanced by AI-driven optimization techniques, our approach reduced mean squared error (MSE) by 20% compared to baseline neural students without distillation in geospatial regression tasks and improved the area under the curve (AUC) by 0.05 in classification tasks. Moreover, the distilled models required approximately 93% fewer floating-point operations (FLOPs) and 50% less memory, enabling real-time applications such as disaster response and urban planning. An open-source Python toolkit accompanies this work to foster further advancements in efficient geospatial intelligence.
Introduction
Geospatial data from satellites, sensors, and surveys have revolutionized fields like disaster response and urban planning (Tao Citation2013). Yet, processing this data in real time remains computationally demanding, particularly on resource-limited devices like edge systems. Traditional machine learning models, despite their predictive power, often require substantial computational resources, making them impractical for large-scale or rapid-deployment scenarios. Knowledge distillation offers a solution by transferring knowledge from sophisticated “teacher” models to simpler “student” models, maintaining accuracy while slashing computational demands (Hasan, Rahman, and Mohammed Citation2023). While this technique has proven effective in deep neural networks (Kiwelekar et al. Citation2020), its application to geospatial regression – particularly with AI-driven optimization – remains largely untapped.
This study addresses this gap through a three-pronged approach:
Evaluating Teacher Models: We assess four spatial regression models – Spatial Durbin Error Model (SDEM), Spatial Lag Model (Lagsarlm), Spatial Durbin Model (SDM), and Spatial Error Model (SEM) – as teacher models to distill spatial knowledge.
Enhancing Distillation: We integrate AI optimization techniques, including Fruit Fly Optimization (FOA), Bayesian optimization, surrogate modeling, and distributed computing, to improve the distillation process.
Benchmarking Performance: We evaluate the resulting student models using diverse geospatial datasets, such as Boston Housing and Rustbelt election data, to measure efficiency and accuracy trade-offs.
Unlike previous efforts centered on neural architectures (Janowicz et al. Citation2020), our work focuses on spatial regression, capitalizing on its strengths in modeling spatial autocorrelation and heterogeneity. The teacher models capture complex spatial relationships, producing soft labels that guide a lightweight student model optimized for speed and scalability.
Our contributions include: (i) pioneering knowledge distillation in geospatial regression using classical spatial econometric teachers; (ii) leveraging AI optimization for enhanced knowledge transfer; and (iii) providing a detailed analysis of complexity versus performance trade-offs. outlines our workflow – from teacher model training to student evaluation – illustrating the distillation and optimization pipeline.
Literature Review
Knowledge distillation in machine learning
Knowledge distillation utilizes a teacher-student framework, where a high-capacity model transfers its learned knowledge to a smaller model (Hinton, Vinyals, and Dean Citation2015). The teacher-student architecture in knowledge distillation involves transferring knowledge from a larger, more complex teacher model to a smaller, more lightweight student model. The teacher model provides guidance to the student model during training, allowing the student to learn from the knowledge and expertise of the teacher.
The teacher-student architecture works by using the knowledge from the teacher model to guide the learning process of the student model. This knowledge transfer can take various forms, such as transferring response-based knowledge (e.g., final predictions), feature-based knowledge (e.g., intermediate layer activations), or relation-based knowledge (e.g., relationships between different layers or data samples).
The teacher-student architecture is designed to compress the knowledge and expertise of the teacher model into the student model, enabling the student to achieve comparable performance to the teacher model while being more lightweight and efficient. This process allows the student model to benefit from the insights and expertise of the teacher model, leading to improved performance and efficiency.
As for knowledge distillation, there are three categories of knowledge for knowledge distillation (Gou et al. Citation2021):
Response-Based Knowledge distillation (L. Song et al. Citation2023): This category refers to the neural response of the last output layer of the teacher model. The main idea is to directly mimic the final prediction of the teacher model.
Feature-Based Knowledge distillation (Ji, Heo, and Park Citation2021): This category involves using the activations, neurons, or features of intermediate layers of the teacher model to guide the learning of the student model.
Relation-Based Knowledge distillation (H. Cheng, Yang, and Liu Citation2021): This category explores the relationships between different layers or data samples, such as the relationships between different feature maps or pairs of samples. These categories provide different types of knowledge that can be transferred from the teacher model to the student model during knowledge distillation. The recent trend of knowledge distillation is the neural architecture search.
The cutting-edge technique of knowledge distillation involves regularizing the teacher by stopping its training early and stopping knowledge distillation close to convergence to allow the student to fit the training loss better. This simple solution has been shown to be effective in improving the efficacy of knowledge distillation across different scenarios. As for the efficacy of the knowledge distillation model, more accurate teachers do not necessarily make them good teachers for knowledge distillation. Larger models do not always make better teachers due to capacity mismatch, making it challenging for small students to mimic large teachers (Gao et al. Citation2023).
The temperature parameter is the recent research topic of the knowledge distillation model. In knowledge distillation, the temperature parameter controls the smoothness of probability distributions and determines the difficulty level of the distillation process. A lower temperature value sharpens the distribution, enlarges the difference between the distributions, and makes the distillation focus on the maximal logits of the teacher’s prediction. On the other hand, a higher temperature flattens the distribution, narrows the gap between the two models, and makes the distillation focus on all logits (Li et al. Citation2023).
Geospatial regression primer
Geospatial regression extends traditional regression by explicitly modeling spatial dependence among observations. At its core lies a spatial‐weights matrix , which encodes the structure of spatial proximity or connectivity (e.g., contiguity, distance‐based k‑nearest neighbors, or inverse‐distance weighting). Each row of W is often row‐standardized so that neighboring influences sum to one. Two canonical formulations prevail: in a lag model, the response itself “spills over” via
where p quantifies how much neighboring y-values drive each outcome; in an error model, spatial autocorrelation resides in the residuals:
with controlling the propagation of unobserved shocks. Extensions such as the Spatial Durbin Model (SDM) and the Spatial Durbin Error Model (SDEM) augment these with spatially lagged covariates WXγ, capturing both outcome and covariate spillovers (Anselin Citation2021).
Foundations of spatial econometric models
Our approach builds on foundational methods in spatial econometrics for teacher model construction. Specifically, we utilize classical spatial models such as SDEM, SDM, SEM, and Lagsarlm, all well-documented in the work of (Anselin and Rey Citation2014), who provide both theoretical foundations and practical applications through GeoDa and PySAL. These models provide interpretable spatial dependence structures which are suitable for distillation into lightweight learners. Spatial regression models are typically estimated by maximum likelihood or generalized method of moments, yielding parameter estimates β, ρ, λ, γ. Their predictive performance is evaluated using standard metrics – mean squared error (MSE) for continuous targets and area under the ROC curve (AUC) when recast as binary classification (e.g., election outcomes). Crucially, residuals from spatial regressions must also be checked for leftover autocorrelation (e.g., via Moran’s I) to ensure the model has captured the geographic structure. These metrics and diagnostics provide the baseline against which we compare “student” models distilled from high‑capacity spatial “teacher” models. While classical spatial‐econometric models (e.g. SDEM, SDM) have not previously been distilled, the broader machine‐learning community has begun exploring knowledge distillation on graph‐structured data and spatially indexed deep networks. For instance, Song et al. propose Tree-like Decision Distillation (J. Song et al. Citation2021) to enable the student with the same problem-solving mechanism as that of the teacher. Similarly, Rong et al. (2020) develop DropEdge‑KD, teaching a student GCN to mimic a teacher under randomized edge removals, which improves robustness on semi‑supervised node classification benchmarks (Rong et al. Citation2020). In the spatial‐deep‐learning realm, a distillation framework helping student MLPs capture graph-structured global spatio-temporal patterns while alleviating the over-smoothing effect with adaptive knowledge distillation (Q. Zhang et al. Citation2025) (J. Zhang et al. Citation2024). proposes a simple RT-TFP model, Spatio Temporal-MultiLayer Perceptron (ST-MLP) to establish a spatio-temporal MLP model to replace the STGNN model for conducting the TFP, which is much faster and simpler. To our knowledge, however, there are no prior efforts to distill classical spatial‐econometric teachers (e.g. SDEM, SEM, SDM) into neural or other lightweight students while retaining explicit econometric parameters such as ρ or λ. This gap motivates our work on spatial‐econometric knowledge distillation, which uniquely bridges econometric theory and AI‑driven model compression.
Geospatial AI and lnowledge distillation
Geospatial AI leverages machine learning techniques for spatial data modeling. Studies have explored spatial-aware deep learning models, probabilistic forecasting, and knowledge distillation for remote sensing (Ji, Heo, and Park Citation2021). The literature review of geospatial AI, spanning from spatial-aware deep learning models to probabilistic forecasting and knowledge distillation techniques. For example, Nejadshamsi et al. introduced a hybrid CNN-GCN model enhancing commuting flow predictions by integrating geographical semantics (Nejadshamsi et al. Citation2025) Cai et al. proposed Hyper Quaternion E, a hyperbolic embedding model improving qualitative spatial and temporal reasoning (Cai et al. Citation2023). Ahmadi et al. leveraged knowledge distillation for efficient building damage assessment from satellite imagery (Ahmadi et al. Citation2023). Ma et al. developed an oncology-based BERT model for geological disasters (Ma et al. Citation2023). Lastly, Cheng et al. addressed overfitting in remote sensing scene classification with an ensemble-based residual network (Q. Cheng et al. Citation2021). Collectively, these studies contribute to robust geospatial AI applications, improving predictive accuracy, computational efficiency, and interpretability across domains.
The AI-inspired knowledge distillation models are utilized in the natural language processing (NLP) models optimization. It’s because the knowledge distillation approach can enable the small size and pretrained models to function smoothly on the edge devices (Wang et al. Citation2024). However, there is a research gap for current development. Traditional machine learning models struggle with spatial autocorrelation, heterogeneity, and computational inefficiency. The SDEM, Lagsarlm, SDM, and SEM models help mitigate these issues, and their role in knowledge distillation is further analyzed in this study.
Methodology
Knowledge distillation design
Knowledge distillation workflow for spatial models
In our framework, we treat each spatial model (e.g., SDEM, SDM, Lagsarlm, SEM) as a teacher that captures spatial relationships explicitly through the spatial weights matrix W, the spatial lag parameters (ρ,λ), and spatially lagged covariates (WX). After training the teacher on data (X,y,W), we extract either:
Regression: predicted continuous labels:
Classification: predicted logits: typically before softmax.
The student model, which is a lightweight feedforward neural network (e.g., MLP), takes only the original covariates X as input and is trained to minimize task loss (e.g., MSE or cross-entropy with ground truth) and mimic the teacher’s outputs via soft-label supervision using Kullback–Leibler (KL) divergence or L2 loss. show the inputs, outputs, and role of parameters of our models.
Knowledge Distillation Framework
Our knowledge distillation framework transfers spatial insights from complex teacher models to a lightweight student model, tailored for geospatial regression. It comprises three components:
1.Teacher Models: We employ four spatial regression models to encapsulate geo-spatial dependencies:
(1) Spatial Durbin Error Model (SDEM): Models spatial dependence in errors and exogenous effects, defined as
W is the spatial weights matrix, y is the dependent variable, X is the regressor matrix, and ε is the error term (LeSage and Pace Citation2009).
(2) Spatial Lag Model (Lagsarlm): Incorporates spatial autocorrelation in the de-pendent variable:
(3) Spatial Durbin Model (SDM): Extends Lagsarlm with spatially lagged regressors:
(4) Spatial Error Model (SEM): Focuses on spatial error correlation: y = Xβ+μ, u=λWμ+ϵ. These models were trained on geospatial datasets to predict housing prices (Boston) and election outcomes (Rustbelt), capturing spatial spillovers and heterogeneity.
Student Model: A simplified multilayer perceptron (MLP), reduces complexity by omitting explicit spatial terms (e.g., WXγ) while approximating teacher outputs.
Distillation Process: Teachers generate soft labels – probabilistic predictions – using a softmax function with temperature T = 2 to smooth outputs. The student is trained with a custom loss function:
(6) (6)
where α and β control the balance between task fidelity and teacher mimicry, T > 1 is a temperature to soften the teacher’s logits, σ is the softmax function, KL is the Kullback–Leibler divergence, and α = 0.7, β = 0.3 balance regression and distillation losses. For regression settings, we replace the KL term with an L2 loss:
The Adam optimizer (learning rate = 0.001) minimizes this loss, ensuring efficient knowledge transfer.
4. Tuning of α and β in the Composite Loss: Our student models were trained using a composite loss function combining the loss on the ground truth with the loss on the teacher’s output:
We determined the optimal weighting parameters α = 0.7 and β = 0.3 based on a grid search over the values [0.0,0.25,0.5,0.75,1.0], evaluating validation MSE in each case. These values achieved the best trade-off between imitating the teacher and matching the ground truth, while avoiding overfitting to soft targets alone.
Full results from this tuning process are provided in , showing that overly high β values degraded accuracy, while pure ground-truth training underperformed compared to mixed supervision.
AI-Optimized knowledge distillation
To further improve model performance, we integrate several AI-driven techniques as complementary optimizations to the core distillation process:
Fruit Fly Optimization Algorithm (FOA): Optimizes student model hyperparameters such as learning rate and dropout.
Bayesian Optimization: Refines learning rate and batch size using a Gaussian process surrogate.
Surrogate Modeling: Approximates the objective function for efficient hyperparameter search.
Distributed Computing: Parallelizes knowledge distillation using Dask to accelerate training across cores.
These techniques were applied selectively to refine the student training process, with evaluations detailed in AI-Inspired Knowledge Distillation Models for Geospatial Data Analysis.
Student model for knowledge Distillation
For the regression tasks, we designed a compact multilayer perceptron (MLP) architecture for the student model to balance predictive performance and computational efficiency. The architecture consists of:
1. Input layer: matching the dimensionality of the covariates (e.g., 13 features for Boston).
2. Dense Layer 1: 128 neurons, ReLU activation
3. Dropout Layer 1: 20% dropout rate (to prevent overfitting)
4. Dense Layer 2: 32 neurons, ReLU activation
5. Dropout Layer 2: 20%
6. Output Layer: 1 neuron with linear activation (for regression).
We trained the student model using the Adam optimizer with a learning rate of 0.001, batch size of 32, and 100 training epochs.
The student model training incorporated a custom loss function as described in Knowledge Distillation Framework. The student model’s performance was evaluated on the test dataset, and the corresponding loss was reported.
Data and experimental setting
The rationale of our two datasets for the knowledge distillation model of geospatial regression is because the Boston data covers housing prices with known spatial heterogeneity; Rustbelt election reflects binary classification under spatial autocorrelation – common benchmarks in spatial econometrics.
At first, this study utilizes the “Housing Values in Suburbs of Boston” dataset, available in the MASS package of the R programming language (Venables and Ripley Citation2002). This dataset is particularly valuable for demonstrating the potential of our approach to revolutionize geospatial data analysis in urban economics and city planning. This dataset is particularly valuable for demonstrating the potential of our approach to revolutionize geospatial data analysis in urban economics and city planning. The dataset encompasses various socio-economic, environmental, and structural attributes of suburbs in Boston, Massachusetts, U.S (Boston data). These attributes provide a comprehensive view of factors influencing housing prices, which are crucial for understanding urban development patterns and making informed policy decisions.
Our analysis focuses on predicting the median value of owner-occupied homes (in $1000 s), represented by the “medv” variable. This serves as our dependent variable, offering insights into housing market dynamics across different Boston suburbs.
We consider the following independent variables, each contributing to a multifaceted analysis of urban characteristics:
Crime Rate (“crim”): Per capita crime rate by town. This factor is crucial in understanding the relationship between public safety and property values.
Residential Zoning (“zn”): Proportion of residential land zoned for lots over 25,000 sq.ft. This variable helps in analyzing the impact of urban planning decisions on housing prices.
Industrial Presence (“indus”): Proportion of non-retail business acres per town. This measure allows us to examine how the mix of residential and commercial areas affects housing values.
Proximity to Charles River (“chas”): A dummy variable (1 if the tract bounds the river, 0 otherwise). This geographic feature helps assess the impact of natural amenities on property values.
Air Quality (“nox”): Nitrogen oxides concentration (parts per 10 million). This environmental factor is key in understanding how pollution levels influence housing prices.
Housing Characteristics (“rm”): Average number of rooms per dwelling. This structural attribute provides insight into the relationship between house size and value.
Age of Housing (“age”): Proportion of owner-occupied units built prior to 1940. This variable allows us to analyze how historical housing stock affects current market values.
Accessibility (“dis”): Weighted mean of distances to five Boston employment centers. This measure helps in understanding the impact of job market proximity on housing prices.
Highway Access (“rad”): Index of accessibility to radial highways. This transportation factor is crucial for assessing the influence of infrastructure on property values.
Property Tax (“tax”): Full-value property tax rate per $10,000. This economic factor helps in analyzing how local tax policies affect housing markets.
Education (“ptratio”): Pupil-teacher ratio by town. This variable allows us to examine the relationship between education quality and housing prices.
Racial Composition (“black”): 1000(Bk − 0.63)2, where Bk is the proportion of Black residents by town. This demographic factor helps in analyzing potential impacts of racial segregation on housing markets.
Socioeconomic Status (“lstat”): Percentage of lower status population. This variable provides insights into how socioeconomic factors influence property values.
By analyzing these diverse variables, our study aims to demonstrate how advanced geospatial data analysis techniques can provide a nuanced understanding of urban housing markets. This approach has the potential to inform more effective urban planning strategies, guide policy decisions, and offer valuable insights for real estate professionals and homebuyers alike.
The study also used the Republican’s voting share (Y) in the U.S. presidential election in 2020 as one of the variables related to the U.S. presidential election (Rustbelt data). The Boston Housing dataset is available via the MASS package in R, and the Rustbelt election data can be downloaded from https://github.com/tonmcg/US_County_Level_Election_Results_08-20.” it collected the 2020 election results at the county level, which were scraped from the results published by Fox News, Politico, and the New York Times. The variables include one dependent variable (Y) and six independent variables (X1 to X6):
Republican’s share of votes in U.S. presidential election (Y): as the dependent variable.
The share of respondents who thought they wore face masks often (X1)
The number of housing units (X2)
The number of residents who were high-school graduates or above (X3)
Unemployment rate (X4)
Household income (X5)
Death toll of COVID-19 cases (X6)
Results of the knowledge distillation on geospatial regression models
We evaluated teacher and student performance using mean squared error (MSE) and area under the ROC curve (AUC) across Boston Housing and Rustbelt election datasets. To demonstrate the benefits of distillation, we compare distilled students to baseline students trained solely on ground-truth labels (without teacher supervision). Results are split into teacher efficacy, baseline student performance, and distillation improvements.
Teacher Model Performance: shows MSE after 100 training iterations. For Boston data, SDEM yielded the lowest teacher MSE (16.04), outperforming SEM (24.728) by 35%, likely due to its robust handling of spatial error and exogenous effects. lagsarlm (14.309) excelled on Rustbelt data, reflecting stronger spatial lag effects.
Baseline and Distilled Student Improvement: Without distillation, baseline students exhibit higher MSE due to their lightweight architecture lacking explicit spatial modeling (e.g., 180.06 for SDEM-Boston baseline). Knowledge distillation reduced student MSE by an average of 20% across models (e.g., from 180.06 to 144.05 for SDEM on Boston). highlights these comparisons, with paired t-tests confirming significance (p < .05) between distilled and baseline students.
Classification Performance: AUC assessed binary classification (threshold: median of dependent variable). For Boston (), SDM-student achieved the highest AUC (0.5702), surpassing its baseline (0.5202) by 0.05, due to distillation amplifying key spatial signals. On Rustbelt (), SDEM-teacher led with 0.6555, and distillation improved student AUC from 0.5118 to 0.5618. These improvements highlight dataset-specific distillation dynamics.
Overall, SDEM emerged as the most consistent teacher, while student gains depended on spatial structure alignment between teacher and data.
According to , we can observe that SDEM model has the lowest MSE for the teacher model, and the student models show consistent 20% improvements via distillation.
According to , we can also observe the SDEM model has the smallest MSE for the teacher model, while the lagsarlm and Spatial Durbin model have the smallest loss value for the student model, with ~20% reductions from baselines.
Feature importance and sensitivity analysis
To assess the contribution of individual features in the teacher models, we perform permutation importance analysis on the trained SDEM model. Specifically, we permute each feature column in X one at a time while keeping all other features fixed, and measure the change in MSE on the validation set. Features causing the largest performance drop are considered the most important. This method allows us to rank input variables by their impact on predictive accuracy while preserving the spatial dependence structure. The results were shown on .
According to , we perform a feature ablation study to evaluate how sensitive the distillation process is to the removal of specific input variables. Using the top-3 most important features (rm, lstat, dis), we retrain both the teacher and student models after dropping each feature individually and jointly. We also test dropping the bottom-3 least influential features as a control. The change in MSE is recorded and reported in . Results show that the removal of top features leads to a significant degradation in student performance (e.g. ΔMSE = 10.722), while removing bottom-ranked features has a negligible impact, confirming the robustness of the model to low-signal variables.
In order to compare with the discriminating power of the classification results, the study used the AUC (area under the ROC curve) as the metrics. The area under the receiver operating characteristic curve (AUC) provides a global metric of a diagnostic test’s ability to discriminate between the presence and absence of a specific condition (Hoo, Candlish, and Teare Citation2017). The larger the area above the reference line of the ROC curve, the better the classification of the model. The study used 20% of data as test data and the other 80% as train data. The study also used the median of the dependent values of test data as the threshold. If the predictions of the teacher model or student model are larger than the threshold, we will mark it as 1. On the contrary, if they are lower than 1, we will mark it as 0. The AUC of the teacher models and student models of the two data were listed as the following tables.
According to , lagsarlm and Spatial Durbin model have the better performance of the knowledge distillation for the Boston data, and SDEM and SEM model has the better performance for the Rustbelt data.
The bar charts for MSE and AUC comparisons across the Boston and Rustbelt datasets are shown in . The key insights from MSE indicate that the student models retain some knowledge from teachers but still have a significantly higher error than teachers (expected due to capacity differences); however, distillation yields ~20% MSE reduction over baselines. SEM consistently underperforms and may not be a good teacher model for knowledge distillation. The key insights from AUC indicate that the gap between teacher and student AUC is smaller than in MSE, indicating that knowledge transfer helps classification tasks more effectively than regression tasks.
AI-Inspired knowledge distillation models for geospatial data analysis
This section extends our knowledge distillation framework to broader geospatial data analysis, leveraging AI optimization to enhance model performance. We applied the framework to the “naturalearth_lowres” dataset from GeoPandas, a low-resolution global map with continent-level geometries and population estimates, chosen for its simplicity and wide applicability in spatial classification tasks. The process involves preprocessing, teacher-student training, and AI-driven optimization, implemented in Python.
The analysis followed five steps:
(1) Data Preparation: We extracted centroid coordinates (latitude, longitude) from the “naturalearth_lowres” geometries as features, with population as the target variable. A binary classification task was defined: continents with populations > 10 million were labeled 1, otherwise 0, reflecting a threshold relevant to urban studies. The dataset (n = 177) was split into 80% training (141 samples) and 20% testing (36 samples). The geospatial file of “naturalearth_lowres” is shown on .
(2) Teacher Model Training: Using SDEM as the teacher (consistent with core experiments), trained via maximum likelihood in PySAL
(3) Soft Label Generation: The teacher produced soft labels (probabilistic outputs) with a temperature T = 2 to enrich student training.
(4) Student Model Training: A lighter network (128 and 64 neurons, ReLU activation) was trained using a custom loss combining binary cross-entropy on true labels, cross-entropy on soft labels (α = 0.3, β = 0.7). The architecture consists of:
1. Input layer: matching the dimensionality of the teacher’s inputs (e.g., 14 features for SDEM with spatial lag)
2. Dense Layer 1: 128 neurons, ReLU activation
3. Dropout Layer 1: 20% dropout rate (to prevent overfitting)
4. Dense Layer 2: 32 neurons, ReLU activation
5. Dropout Layer 2: 20%
6. Output Layer: 1 neuron with linear activation (for regression), or softmax for classification.
We trained the student model using the Adam optimizer with a learning rate of 0.001, batch size of 32, and 100 training epochs.
(5) Evaluation: Student performance was assessed on test data using accuracy and AUC.
To accelerate experiments involving spatial model training, knowledge distillation, and hyperparameter tuning, we employed a multi-core CPU parallelization setup. Experiments were conducted on a 64-bit Ubuntu 22.04 workstation with:
CPU: AMD Ryzen 95950X, 16 cores/32 threads
RAM: 128 GB DDR4
GPU: NVIDIA RTX 3090 (used only for deep learning components)
Storage: 2 TB NVMe SSD
For parallel training and hyperparameter optimization, we used the Google Colab as the platform. Deep learning experiments using Keras were GPU-accelerated via the TensorFlow backend. While no multi-node distributed cluster was used, all tuning and model evaluation steps were parallelized at the task level (e.g., one core per α–β or temperature configuration), enabling rapid experimentation with minimal overhead.
To accelerate and refine distillation, we integrated AI optimization techniques:
(1) Fruit Fly algorithm with Hyperparameter Optimization using Bayesian Optimization: The study also applied the Fruit Fly algorithm (FOA)to integrate into the knowledge distillation model in Python (Pan Citation2012). In the beginning, the study also refines the neural network with the same design for binary classification. We selected the FOA for hyperparameter tuning due to its low computational overhead, fast convergence behavior, and simplicity of implementation, which are particularly suitable for iterative student model training during knowledge distillation. Unlike more complex evolutionary algorithms such as Genetic Algorithms (GA) or Particle Swarm Optimization (PSO), FOA operates with a small number of parameters and requires fewer fitness evaluations to locate optimal regions of the search space. Empirical studies (Gu et al. Citation2021; Pan Citation2012) have shown that FOA performs competitively on regression and classification benchmarks with a significantly smaller search population. In our case, we optimize a small set of hyperparameters (learning rate, dropout, and temperature). FOA consistently produced better or equivalent results with fewer iterations compared to random search and grid search baselines. These characteristics made FOA a pragmatic and lightweight optimization framework, aligned with the goal of producing efficient student models under constrained computing budgets. We used FOA optimize the hyperparameters of student model and used them to train the Student Model with optimized hyperparameters incorporating dropout and L2 regularization. Bayesian optimization (Victoria and Maragatham Citation2021) was also applied to optimize hyperparameters more efficiently by modeling the objective function and using this model to select the most promising hyperparameters to evaluate. optimization to efficiently explore the search space, following the framework proposed by Snoek et al. (Citation2012) (Snoek, Larochelle, and Adams Citation2012). This method balances exploration and exploitation using a surrogate Gaussian process and acquisition function, enabling optimal selection of learning rate, temperature T, and dropout rate with fewer evaluations than grid or random search.
We conducted a two-stage grid search over learning‐rate (η) in{10−4, 10−3, 10−2}, batch‐size (BS) in {16,32,64}, and distillation temperature T in {1,2,5} on a held out 20% validation split of the naturalearth_lowres”geospatial file training data. For each combination we trained the student MLP for 100 epochs and recorded validation AUC. As shown in , the highest validation AUC was achieved at η = 10−3 and BS = 64, while a temperature of T = 2 yielded the best trade off between smooth soft labels and gradient signal. We therefore fixed these hyperparameters for all subsequent experiments.
(2) Surrogate Modeling: Surrogate modeling (Ping and Zhou Citation2020) was applied to approximate the objective function to reduce the number of expensive evaluations needed. The model constructs a neural network with L2 regularization and dropout is optimized using scikit-optimize’s Bayesian method with surrogate model over learning rate, batch size, and alpha (balancing true and soft labels). A teacher model generates soft labels, and the student model is trained with a custom loss combining true labels, soft labels, and regularization.
(3) Distributed Computing: The technique distributed the computational load across multiple machines or used cloud services for parallel processing. We designed the codes to parallelize the optimization of a machine learning model using dask (Shafi et al. Citation2021). This approach leverages dask to efficiently parallelize the hyperparameter optimization process, potentially speeding up the search for the best model configuration. The student model is trained with a custom loss combining true and soft labels, plus regularization, using an exponentially decaying learning rate.
reflects the updated values after incorporating class balancing and full hyperparameter tuning, resolving the discrepancy with lower baseline values. The metrics are for the binary classification of election outcomes.:
Discussion
The results of our study highlight the effectiveness of knowledge distillation in geospatial regression modeling, demonstrating improvements in student model performance through various distillation techniques. This section discusses the key findings, their implications, and potential areas for further research, with explicit reference to our stated contributions.
Key Findings and Interpretation of Knowledge Distillation Models: Geospatial Regression
The study utilized four spatial regression models – Spatial Durbin Error Model (SDEM), Spatial Simultaneous Autoregressive Lag Model (lagsarlm), Spatial Durbin Model (SDM), and Spatial Error Model (SEM) – as teacher models. The results revealed that the SDEM model consistently outperformed the other models in terms of minimizing the Mean Squared Error (MSE) across both the Boston and Rustbelt datasets (Contribution i: pioneering KD with classical teachers). This suggests that incorporating spatial dependence in both error terms and exogenous effects contributes to improved predictive performance.
For the student model, which had a simpler structure, knowledge distillation successfully transferred information from the teacher models, leading to a ~ 20% reduction in MSE compared to baselines. However, the extent of improvement varied by dataset. In the Boston dataset, the SDEM teacher model produced the lowest student MSE (144.05), whereas in the Rustbelt dataset, lagsarlm and SDM had the lowest student MSE (0.086). These results indicate that the effectiveness of knowledge distillation depends on the spatial structure and characteristics of the dataset.
To assess the ability of our models to capture spatial dependence, we computed Moran’s I on the residuals of both teacher and student models. The teacher model (SDEM) yielded a Moran’s I of 0.1397 with a p-value of 0.0026, indicating a modest but statistically significant spatial autocorrelation remaining in the residuals. This suggests the spatial lag model reduced, but did not fully eliminate, spatial structure in the data.
In contrast, the student model (a distilled MLP) exhibited a higher residual Moran’s I of 0.3052 (p < 1e-6), revealing substantially more spatial autocorrelation in its residuals. This indicates that the distilled student, while effective at approximating the teacher’s predictions, is less capable of internalizing the spatial dependency structure, since it lacks an explicit spatial mechanism (e.g., no spatial weights matrix W).
These findings are further supported by spatial visualizations (), where residuals from the student model display clearer clustering patterns compared to the teacher model. The contrast in residual autocorrelation validates that spatial structure is partially lost during distillation, highlighting a limitation of spatially unaware student architectures.
Although the student model exhibits a higher residual spatial autocorrelation (Moran’s I = 0.3052) than the teacher model (Moran’s I = 0.1397), this does not significantly degrade predictive accuracy. The mean squared error (MSE) on the test set remained within an acceptable margin, confirming that the student effectively mimics the teacher’s predictions at a global level. However, the elevated Moran’s I suggests the student fails to fully capture local spatial nuances – a trade-off typical in knowledge distillation when the student lacks spatially structured inductive biases (e.g., spatial lags or graph structure).
The classification performance was measured using the AUC metric. In the Boston dataset, the lagsarlm and SDM models demonstrated superior knowledge distillation performance, while in the Rustbelt dataset, SDEM and SEM performed best. This suggests that the teacher model’s ability to transfer spatial information varies depending on the geospatial properties of the data. The results imply that minor architectural modifications enable regression-based methods to surpass sophisticated state-of-the-art techniques in knowledge distillation from self-supervised models (Navaneet et al. Citation2022).
The performance variations among spatial models (SDEM, Lagsarlm, SDM, SEM) arise primarily from their differing assumptions about spatial dependence and how it enters the regression process. SDEM accounts for spatial autocorrelation in both the covariates (via spatially lagged WX) and the error term (via λWu), making it well-suited for capturing unobserved contextual effects or omitted variables that exhibit spatial patterns. Lagsarlm explicitly models spatial dependence in the dependent variable (Wy), reflecting feedback processes (e.g., neighborhood influence), which may be more relevant in social systems like housing markets. SDM generalizes both Lagsarlm and SDEM by including both WX and Wy often improving performance when both endogenous interaction and contextual spillover are present. SEM, by contrast, assumes that spatial dependence is confined to the error structure and not to the outcome itself, which may be too restrictive when spatial spillovers are direct.
Among the four AI-driven optimization techniques explored for hyperparameter tuning during knowledge distillation, Bayesian optimization consistently achieved the highest performance across classification tasks on the Rustbelt dataset, yielding the highest accuracy (0.81 ± 0.01) and AUC (0.84 ± 0.01), as shown in . Fruit Fly Optimization (FOA) ranked second with strong results (accuracy 0.78 ± 0.02, AUC 0.82 ± 0.02), followed by surrogate modeling, while distributed computing produced the lowest classification performance (accuracy 0.72 ± 0.03, AUC 0.76 ± 0.03). These findings indicate that sequential model-based optimization methods (Bayesian optimization and FOA) are particularly effective for tuning knowledge distillation in geospatial contexts, outperforming both brute-force distributed search and surrogate-assisted approaches in this study.
compares teachers (average across SDEM, SDM, Lagsarlm, SEM), baseline neural student (without distillation), distilled student, and the new ablation variants. Metrics include regression (MSE on Boston Housing) and classification (AUC on Rustbelt). The 15× FLOPs reduction (93% fewer) is relative to the baseline neural student. Moran’s I is for residual spatial autocorrelation (lower is better). Combined with a 15× reduction in FLOPs, this confirms successful knowledge transfer in terms of both accuracy and efficiency. However, the student’s higher residual Moran’s I (0.3052 vs. 0.1397) suggests spatial dependencies are under-modeled, revealing an avenue for future work in spatially aware student designs (Contribution iii: analysis of complexity-performance trade-offs).
In our experiments, models that explicitly include spatially lagged predictors (SDM, SDEM) tend to perform better on datasets like Boston Housing, where covariates such as LSTAT and DIS exhibit strong spatial gradients. On the Rustbelt election dataset, where spatial interdependence of outcomes (e.g., turnout) is stronger, Lagsarlm and SDM outperform SEM, highlighting the importance of modeling feedback and diffusion effects. These differences underscore the importance of aligning model structure with the specific type of spatial interaction present in the data.
Implications for knowledge distillation model: geospatial data analysis
The application of knowledge distillation to geospatial regression models provides a novel approach to improving prediction accuracy while reducing model complexity. Our findings have several implications, tying back to our contributions:
(1) Model Selection for Knowledge Distillation (Contribution i): The performance of teacher models varies depending on the dataset, suggesting that researchers must carefully evaluate the characteristics of their data before selecting an appropriate teacher model for knowledge distillation.
(2) Trade-off Between Complexity and Performance (Contribution iii): The student models benefited from knowledge distillation by achieving lower MSE values despite having a simpler structure than the teacher models. This highlights the potential of knowledge distillation to enhance computational efficiency while maintaining predictive accuracy.
(3) AI-Driven Optimization for Knowledge Distillation (Contribution ii): The integration of AI-inspired techniques such as Fruit Fly Optimization Algorithm (FOA), Bayesian optimization, surrogate modeling, and distributed computing provided modest enhancements to the distillation process. Among these techniques, FOA and Bayesian Optimization yielded the highest classification accuracy and AUC values in exploratory tests. This suggests that leveraging AI-driven optimization methods can support the performance of knowledge distillation in geospatial data analysis, though further validation on balanced datasets is warranted. Bayesian optimization emerged as the most effective AI-driven technique for optimizing the knowledge distillation process, delivering the highest classification accuracy (0.81) and AUC (0.84) on the Rustbelt election dataset (). This reinforces Contribution ii by demonstrating that sophisticated yet computationally efficient optimization strategies – rather than massively parallel or population-based methods – are best suited for tuning lightweight geospatial models under resource constraints.
These results demonstrate that AI-driven optimizations outperform grid search by achieving higher AUC (up to 0.84 vs. 0.75) with 30–50% fewer iterations, making them particularly valuable for resource-constrained geospatial tasks (Contribution ii). The key benefit is not just final performance, but the ability to reach it faster and with less compute – a critical factor in real-time or edge-based geospatial intelligence. To mitigate the higher residual Moran’s I in students (0.3052 vs. 0.1397 for teachers), we explored remedies such as incorporating WX features or graph-aware layers, which reduced Moran’s I by 15–20% in ablations. This enhances Contribution iii by providing actionable trade-offs for retaining spatial structure in distilled models.
In classification tasks, we used predicted probabilities from the teacher and student models to compute class predictions based on a threshold t∈[0.5,0.9]. The default threshold of 0.5 was used to report main results, but we also evaluated robustness to other thresholds (e.g., 0.7, 0.9) to account for class imbalance and application-specific precision-recall tradeoffs. However, we can observe the accuracy and F-1 Score achieve the best results when the optimal threshold equals 0.5 according to .
Ablation on AI Optimizations Vs. Grid Search
To evaluate the benefits of AI-driven optimizations (FOA, Bayesian optimization, surrogate modeling, distributed computing) over simpler methods, we compared them against a well-tuned grid search on the Rustbelt dataset. summarizes the results, showing that AI methods achieve higher AUC (up to 0.84 vs. 0.75) with 30–50% fewer iterations. This core advantage – superior sample efficiency in hyperparameter tuning – makes these techniques particularly valuable in resource-constrained geospatial settings where computational budget is limited (Contribution ii).
Among these, Fruit Fly Optimization (FOA) stands out for its biological inspiration and extremely low per-iteration cost, making it ideal for edge deployment scenarios. Unlike population-heavy methods, FOA explores the search space with minimal memory and parallelization overhead, achieving strong results (AUC 0.82 ± 0.02 in 80 iterations) while maintaining simplicity and speed. summarizes the results, showing AI methods achieve higher AUC with fewer iterations.
Limitations and future Work
Despite the promising results, the study has several limitations that should be addressed in future research:
(1)Data Generalizability: The study was conducted on two specific datasets (Boston and Rustbelt data), which may limit the generalizability of the findings. Future research may consider to test the proposed knowledge distillation models on a broader range of geospatial datasets with different spatial structures and levels of heterogeneity.
(2)While AI-driven optimization techniques improved the knowledge distillation process, further research is needed to refine hyperparameter tuning strategies and improve model interpretability, particularly in complex geospatial settings (Contribution ii).
(3)Comparative Studies with Other Machine Learning Models: Future studies can compare knowledge distillation with other state-of-the-art machine learning models, such as ensemble learning and deep learning-based geospatial regression models, to determine its relative advantages and limitations.
(4)Integration with Real-Time Geospatial Applications: The study primarily focused on static datasets, but the application of knowledge distillation to real-time geospatial analysis, such as disaster response or urban planning, remains an open area for exploration.
While our proposed KD framework successfully distills spatial econometric models into lightweight neural students for both regression and classification, it has important limitations that affect its generalizability. First, our experiments are limited to structured tabular datasets (Boston Housing and Rustbelt Election), both of which exhibit moderate spatial autocorrelation. The framework may require adaptation to handle spatiotemporal data (e.g., traffic forecasting) or irregular spatial domains (e.g., satellite imagery). Second, we assume that the teacher model – typically an interpretable spatial regression (e.g., SDEM or SDM) – provides sufficiently accurate and smooth predictions that can be effectively mimicked by a student network. In real-world applications with noisy or high-dimensional inputs, teacher predictions may be unstable or suboptimal, degrading distillation quality. Third, our student models do not explicitly encode spatial structure (e.g., via graph convolutions or spatial embeddings), which may limit their performance on tasks where spatial topology is crucial.
Despite these constraints, the framework is broadly generalizable to other regression/classification tasks with spatial dependence. The KD approach is model-agnostic: any spatial model capable of producing soft labels can serve as a teacher, and any neural architecture (e.g., MLP, CNN, GCN) can act as a student. Future work could extend this framework to spatiotemporal distillation, dynamic teacher-student co-training, or hybrid models where students learn spatially aware representations (e.g., via spatial embeddings or attention). Furthermore, integrating uncertainty quantification into the distillation process could improve robustness under spatial heterogeneity.
Conclusions
We introduced a pioneering knowledge distillation framework that leverages classical spatial econometric models as teachers to produce compact, efficient students for geospatial regression and classification tasks (Contribution i).
Our key findings are as follows: (1) Distilled student models exhibit a 20% average decrease in MSE on continuous value prediction and a 0.05 AUC uplift in binary spatial classification (Contribution iii). (2) Student models require 60% fewer FLOPs and 50% less memory footprint compared to untuned neural baselines, enabling edge deployment. It indicates our approach streamlines spatial analytics by allowing municipalities and NGOs to deploy accurate geospatial models on commodity hardware, enhancing capabilities in areas like flood risk assessment and land-use monitoring. AI optimizations provided supplementary benefits in hyperparameter tuning (Contribution ii).
While our framework addresses spatial-only dependencies, real-world phenomena often entail spatio-temporal dynamics. Future studies can extend this work by: (1) Distilling space–time autoregressive teacher models into recurrent or Transformer-based student models. (2) Incorporating graph neural network teachers to capture irregular spatial networks. (3) Evaluating resilience under noisy or incomplete spatial inputs. Future extensions of our framework (Contribution i) could integrate these remedies natively, such as graph neural students, to better preserve spatial dependencies.
We anticipate that our hybrid knowledge distillation paradigm will serve as a blueprint for integrating rigorous statistical models with modern AI techniques, paving the way for widespread, resource-efficient geospatial intelligence.
Abbreviations
| The following abbreviations are used in this manuscript: | = | |
| SDM | = | Spatial Durbin Model |
| SDEM | = | Spatial Durbin Error Model |
| lagsarlm | = | Spatial simultaneous autoregressive lag model |
| SEM | = | Spatial Error Model |
Acknowledgements
The author is responsible for the formulation of the whole paper.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Data availability statement
The data that support the findings of this study are openly available in Zenodo at https://doi.org/10.5281/zenodo 16,921,102 [REF 34], under a CC-BY 4.0 license.
The repository includes:
- The raw datasets: (1) Boston Housing dataset [REF 25], (2) U.S. County-Level Presidential Election Results 2008–2020 [REF 26], and (3) NaturalEarth low-resolution shapefile [REF 28].
- The processed data used to generate the regression/classification results.
- The numerical values underlying all reported tables and figures, including MSE, AUC, FLOPs, and feature importance analyses.
− The extracted points and intermediate values used for figure generation.
These datasets and supplementary materials allow full replication of all analyses reported in this manuscript.
Additional information
Funding
References
- Ahmadi, S. A., A. Mohammadzadeh, N. Yokoya, and A. Ghorbanian. 2023. “BD-SKUNet: Selective-Kernel UNets for Building Damage Assessment in High-Resolution Satellite Images.” Remote Sensing 16 (1): 182. https://doi.org/10.3390/rs16010182
- Anselin, L. 2021. “Spatial Models in Econometric Research.”
- Anselin, L., and S. J. Rey. 2014. Modern Spatial Econometrics in Practice: A Guide to GeoDa, GeoDaSpace and PySAL. GeoDa Press LLC.
- Cai, L., K. Janowicz, R. Zhu, G. Mai, B. Yan, and Z. Wang. 2023. “Hyperquaternione: A Hyperbolic Embedding Model for Qualitative Spatial and Temporal Reasoning.” Geoinformatica 27 (2): 159–20. https://doi.org/10.1007/s10707-022-00469-y
- Cheng, H., L. Yang, and Z. Liu. 2021. “Relation-Based Knowledge Distillation for Anomaly Detection.” Proceedings of the Pattern Recognition and Computer Vision, edited by H. Ma, L. Wang, C. Zhang, F. Wu, T. Tan, Y. Wang, J. Lai, and Y. Zhao. Cham: 105–116. Springer International Publishing.
- Cheng, Q., D. Gan, P. Fu, H. Huang, and Y. Zhou. 2021. “A Novel Ensemble Architecture of Residual Attention-Based Deep Metric Learning for Remote Sensing Image Retrieval.” Remote Sensing 13 (17): 3445. https://doi.org/10.3390/rs13173445
- Gao, S., Y. Hu, W. Li, and L. Zou. 2023. “Special Issue on Geospatial Artificial Intelligence.” Geoinformatica 27 (2): 133–136. https://doi.org/10.1007/s10707-023-00493-6
- Gou, J., B. Yu, S. J. Maybank, and D. Tao. 2021. “Knowledge Distillation: A Survey.” International Journal of Computer Vision 129 (6): 1789–1819. https://doi.org/10.1007/s11263-021-01453-z
- Gu, Q., Y. Chang, X. Li, Z. Chang, and Z. Feng. 2021. “A Novel F-SVM Based on FOA for Improving SVM Performance.” Xpert Systems with Applications 165:113713. https://doi.org/10.1016/j.eswa.2020.113713
- Hasan, M. J., F. Rahman, and N. Mohammed. 2023. “Distilling the Knowledge of Clinical Outcome Predictions in Large Language Models for Resource Constrained Healthcare Systems.” Available SSRN 4591013.
- Hinton, G., O. Vinyals, and J. Dean. 2015. “Distilling the Knowledge in a Neural Network.” ArXiv Prepr. arXiv1503.02531.
- Hoo, Z. H., J. Candlish, and D. Teare. 2017. “What Is an ROC Curve?” Emergency Medicine Journal 34 (6): 357–359. https://doi.org/10.1136/emermed-2017-206735
- Janowicz, K., S. Gao, G. McKenzie, Y. Hu, and B. Bhaduri. 2020. “GeoAI: Spatially Explicit Artificial Intelligence Techniques for Geographic Knowledge Discovery and Beyond.” International Journal of Geographical Information Science 34 (4): 625–636. https://doi.org/10.1080/13658816.2019.1684500
- Ji, M., B. Heo, and S. Park. 2021. “Show, Attend and Distill: Knowledge Distillation via Attention-Based Feature Matching.” The 35th AAAI Conference on Artificial Intelligence AAAI 2021 9B (9): 7945–7952. https://doi.org/10.1609/aaai.v35i9.16969
- Kiwelekar, A. W., G. S. Mahamunkar, L. D. Netak, and V. B. Nikam. 2020. “Deep Learning Techniques for Geospatial Data Analysis.” In Machine Learning Paradigms. Learning and Analytics in Intelligent Systems, edited by G. Tsihrintzis and L. Jain, Vol. 18. Cham: Springer. https://doi.org/10.1007/978-3-030-49724-8_3
- LeSage, J. P., and R. K. Pace. 2009. “Spatial Econometric Models.” In Handbook of Applied Spatial Analysis: Software Tools, Methods and Applications, edited by Manfred M. Fischer and Arthur Getis, 355–376. Springer.
- Li, Z., X. Li, L. Yang, B. Zhao, R. Song, L. Luo, J. Li, and J. Yang. 2023. “Curriculum Temperature for Knowledge Distillation.” Proceedings of the Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, Washington D.C. USA, 1504–1512.
- Ma, K., M. Tian, Y. Tan, Q. Qiu, Z. Xie, and R. Huang. 2023. “Ontology-Based BERT Model for Automated Information Extraction from Geological Hazard Reports.” Journal of Earth System Science 34 (5): 1390–1405. https://doi.org/10.1007/s12583-022-1724-z
- Navaneet, K. L., S. A. Koohpayegani, A. Tejankar, and H. Pirsiavash. 2022. “Simreg: Regression as a Simple Yet Effective Tool for Self-Supervised Knowledge Distillation.”
- Nejadshamsi, S., J. Bentahar, U. Eicker, C. Wang, and F. Jamshidi. 2025. “A Geographic-Semantic Context-Aware Urban Commuting Flow Prediction Model Using Graph Neural Network.” Xpert Systems with Applications 261:125534. https://doi.org/10.1016/j.eswa.2024.125534
- Pan, W. T. 2012. “A New Fruit Fly Optimization Algorithm: Taking the Financial Distress Model as an Example.” Knowledge-Based Systems 26:69–74. https://doi.org/10.1016/j.knosys.2011.07.001
- Ping, J., and Q. Zhou. 2020. “And S.X.Surrogate-Model-Based Design and Optimization.” In Surrogate Model-Based Engineering Design and Optimization, edited by Ping Jiang, Qi Zhou, and Xinyu Shao, 135–236. Singapore: Springer Singapore ISBN 978-981-15-0731-1.
- Rong, Y., W. Huang, T. Xu, and J. Huang. 2020. “DropEdge: Towards Deep Graph Convolutional Networks on Node Classification.” The 8th International Conference on Learning Representations ICLR 2020: 1–18. https://doi.org/10.48550/arXiv.1907.10903
- Shafi, A., J. M. Hashmi, H. Subramoni, and D. K. D. K. Panda. 2021. “Efficient MPI-Based Communication for GPU-Accelerated Dask Applications.” Proceedings of the 2021 IEEE/ACM 21st International Symposium on Cluster, Cloud and Internet Computing (CCGrid), Melbourne, Australia, 277–286.
- Snoek, J., H. Larochelle, and R. P. Adams. 2012. “Practical Bayesian Optimization of Machine Learning Algorithms.” Advances in Neural Information Processing Systems: 25.
- Song, J., H. Zhang, X. Wang, M. Xue, Y. Chen, L. Sun, D. Tao, and M. Song. 2021. “Tree-like Decision Distillation.” 13483–13492.
- Song, L., X. Gong, H. Zhou, J. Chen, Q. Zhang, D. Doermann, and J. Yuan. 2023. “Exploring the Knowledge Transferred by Response-Based Teacher-Student Distillation.” Proceedings of the Proceedings of the 31st ACM International Conference on Multimedia; Association for Computing Machinery, New York, NY, USA: 2704–2713.
- Tao, W. 2013. “Interdisciplinary Urban GIS for Smart Cities: Advancements and Opportunities.” Geo-Spatial Information Science 16 (1): 25–34. https://doi.org/10.1080/10095020.2013.774108
- Venables, W. N., and B. D. Ripley. 2002. Modern Applied Statistics with S; Fourth.; New York: Springer.
- Victoria, A. H., and G. Maragatham. 2021. “Automatic Tuning of Hyperparameters Using Bayesian Optimization.” Evolutionary Systematics 12 (1): 217–223. https://doi.org/10.1007/s12530-020-09345-2
- Wang, Y., Z. Yu, J. Wu, C. Wang, Q. Zhou, and J. Hu. 2024. “Adaptive Knowledge Distillation Based Lightweight Intelligent Fault Diagnosis Framework in IoT Edge Computing.” IEEE Internet of Things Journal 11 (13): 23156–23169. https://doi.org/10.1109/JIOT.2024.3387328
- Zhang, J., C. Xie, H. Cai, W. Shen, and R. Yang. 2024. “Knowledge Distillation-Based Spatio-Temporal MLP Model for Real-Time Traffic Flow Prediction.” IEEE Transactions on Intelligent Transportation Systems 25 (11): 18122–18135. https://doi.org/10.1109/TITS.2024.3424808
- Zhang, Q., X. Gao, H. Wang, S. M. Yiu, and H. Yin. 2025. “Efficient Traffic Prediction Through Spatio-Temporal Distillation.” Proceedings of the AAAI Conference on Artificial Intelligence 39 (1): 1093–1101. https://doi.org/10.1609/aaai.v39i1.32096