Electrical substation detection

energy
computer vision
data quality
Finding out why a substation segmentation model that scored well in testing failed on new images, then rebuilding its training data from OpenStreetMap: 2,728 substations in 10 countries, with a review tool to filter out label errors.

Finding out why a substation segmentation model that scored well in testing failed on new images, then rebuilding its training data from OpenStreetMap: 2,728 substations in 10 countries, with a review tool to filter out label errors.

Employer: GL Spatial (client project)
When: 2023
Data: High-resolution commercial satellite imagery, OpenStreetMap power infrastructure polygons, an open machine learning competition dataset of labelled substations
Methods: Deep learning semantic segmentation (DeepLabV3+ with an ImageNet-pretrained EfficientNet-B3 encoder), combined Dice and cross-entropy loss, independent validation, training data creation, manual label review
Stack: Python, PyTorch, segmentation_models_pytorch, GeoPandas, QGIS (PyQGIS), Jupyter, Paperspace Gradient

Background

Our client develops large data centres, and the power supply available near a candidate site is a key factor in where it builds. OpenStreetMap maps many electrical substations, but its accuracy varies from place to place and it may not be up to date. The project aimed to detect substations directly in satellite imagery, so maps of them could be kept current without ground surveys.

The work started under a previous consultant, who had adapted a deep learning model and dataset from an open machine learning competition and reached a high accuracy score. I took the project over in early 2023, working on it part time, about 10 hours a week, alongside my full-time role at Sparkgeo.

My contribution

  • Retraining: the model was a DeepLabV3+ segmentation network with an EfficientNet-B3 encoder pretrained on ImageNet. The previous consultant had been limited by computing resources and could only train for a few epochs. I adapted the training code to run on GPUs in Paperspace Gradient and trained for longer, which raised the validation mean IoU (intersection over union, the overlap between predicted and labelled substations) only slightly, from 0.93 to 0.94.
  • Independent validation: I tested the model on 42 substation images from Germany that it had never seen. It performed very poorly, both missing real substations and detecting substations where there were none.
  • Diagnosis: the training set held 2,400 images, but only 100 were real examples. The other 2,300 were augmented copies of those 100. That gave the model too few real substations to learn what one looks like. It also inflated the score: the validation set was a random 20% of all 2,400 images, so it held altered copies of the same substations the model had trained on. I recommended building a new training set with far more real examples.
  • Candidate substations: I filtered a global OpenStreetMap layer of more than a million power infrastructure features down to substations, assigned each to a country, and calculated its area. For each of 10 countries chosen with the client, I kept the 1,000 largest substations (by area in an equal-area projection) and dropped any too small to fill an image chip at a usable resolution, giving 2,728 substations.
  • Image chips: a PyQGIS script extracted a 1,024 × 1,024 pixel image chip from high-resolution commercial imagery for each substation, sized to its bounding box plus a 25% margin, together with a matching label chip drawn from the OpenStreetMap outline.
  • Label review: OpenStreetMap outlines weren’t always right. Some were drawn incorrectly, some marked substations that weren’t in the imagery, and some marked substations inside buildings, which can’t be identified from above. I built a review tool in a Jupyter notebook that shows each image chip beside its label and records one of three decisions: accept, accept but flag the outline for correction, or reject. It saves progress after every image, so the review can stop and resume.

Outcome

  • Delivered:
    • the retrained model (built from the challenge dataset)
    • a diagnosis of why the original model failed on new images
    • a pipeline that builds training data from OpenStreetMap along with a review tool for evaluating the training chips
    • a set of almost 3000 image chips along with labels
    • a report describing each workflow and the next steps
  • The client ended the project early as its priorities changed, so the model wasn’t retrained on the new training dataset, and we weren’t able to see if more real examples could fix the model’s failure to generalize.

Cover image: Power lines crossing over green fields. Photo by Bernd Dittrich on Unsplash.