BPKD : Boundary Privileged Knowledge Distillation For Semantic Segmentation

Authors: Liyang Liu, Zihan Wang, Minh Hieu Phan, Bowen Zhang, Jinchao Ge, Yifan Liu*.

*Corresponding author

[Paper] [Paper Fast Mirror] [Project] [Github] [Docker] [Pretrained models] [Visualization]

---

Abstract: Current approaches for knowledge distillation in semantic segmentation tend to adopt a holistic approach that treats all spatial locations equally. However, for dense prediction tasks, it is crucial to consider the knowledge representation for different spatial locations in a different manner. Furthermore, edge regions between adjacent categories are highly uncertain due to context information leakage, which is particularly pronounced for compact networks. To address this challenge, this paper proposes a novel approach called boundary-privileged knowledge distillation (BPKD). BPKD distills the knowledge of the teacher model's body and edges separately from the compact student model. Specifically, we employ two distinct loss functions: 1) Edge Loss, which aims to distinguish between ambiguous classes at the pixel level in edge regions. 2) Body Loss, which utilizes shape constraints and selectively attends to the inner-semantic regions. Our experiments demonstrate that the proposed BPKD method provides extensive refinements and aggregation for edge and body regions. Additionally, the method achieves state-of-the-art distillation performance for semantic segmentation on three popular benchmark datasets, highlighting its effectiveness and generalization ability. BPKD shows consistent improvements over various lightweight semantic segmentation structures.

What's New

• 25 Oct 2023 : Our Paper Accept by WACV 2024, we will refactor and release the training code after CVPR DDL, stay tuned.

• 18 Jul 2023 : Release evaluation code and weights for Pascal Context dataset.

• 17 Jul 2023 : Release evaluation code and weights for ADE20K dataset.

• 16 Jul 2023 : Release evaluation code and weights for Cityscape dataset.

Environment

mmrazor@8b57a07b5e6033dbd0052aeaf0f72668bdaecd00

mmseg==0.26.0

mmcv-full==1.6.0

Checkout requirements.txt for full requirements

Docker Image :

sudo docker run --gpus all -v ~/data:/data \
-e GITHUB_TOKEN="xxxx" \
-e WANDB_TOKEN="xxxx" \
-it --shm-size=64gb ghcr.io/uaws/pytorch-sshd:ngc-pytorch-1.13-mmcv-1.6.0-mmseg-0.26.0-ubuntu-20.04 /bin/bash

Preparing Dataset

1. Cityscapes
2. PASCAL Context
3. ADE20K

According to MMseg: https://github.com/open-mmlab/mmsegmentation/blob/master/docs/en/dataset_prepare.md

Evaluation

python tools/test.py {config.py} {checkpoint.pth} --eval mIoU

Results and models

We conducted all experiments using 4 NVIDIA A100 GPUs on The University of Adelaide High Performance Computing Cluster (HPC).

Cityscapes 512x1024 80K

Table: Performance on Cityscapes Dataset

Methods	FLOPs(G)	Parameters(M)	mIoU(%)	mAcc(%)	Config	ckpt
T: PSPNet-R101	256.89	68.07	79.74	86.56	Config	Model
S: PSPNet-R18	54.53	12.82	74.23	81.45	Config	Model
SKDS	54.53	12.82	76.13	82.58	Config	Model
IFVD	54.53	12.82	76.24	82.81	Config	Model
CIRKD	54.53	12.82	76.03	82.56	Config	Model
CWD	54.53	12.82	76.26	83.04	Config	Model
BPKD(Ours)	54.53	12.82	77.57	84.47	Config	Model
T: HRNetV2P-W48	95.64	65.95	80.65	87.39	Config	Model
S: HRNetV2P-W18S	10.49	3.97	75.31	83.71	Config	Model
SKDS	10.49	3.97	77.27	84.77	Config	Model
IFVD	10.49	3.97	77.18	84.74	Config	Model
CIRKD	10.49	3.97	77.36	84.97
CWD	10.49	3.97	77.87	84.98	Config	Model
BPKD(Ours)	10.49	3.97	78.58	85.78	Config	Model
T: DeeplabV3P-R101	255.67	62.68	80.98	88.7	Config	Model
S: DeeplabV3P+MV2	69.60	15.35	75.29	83.11	Config	Model
SKDS	69.60	15.35	76.05	84.14	Config	Model
IFVD	69.60	15.35	76.97	84.85	Config	Model
CIRKD	69.60	15.35	77.71	85.33	Config	Model
CWD	69.60	15.35	77.97	86.68	Config	Model
BPKD(Ours)	69.60	15.35	78.59	86.45	Config	Model
T: ISANet-R101	228.21	56.80	80.61	88.29	Config	Model
S: ISANet-R18	54.33	12.46	73.62	80.36	Config	Model
SKDS	54.33	12.46	74.99	82.61	Config	Model
IFVD	54.33	12.46	75.35	82.86	Config	Model
CIRKD	54.33	12.46	75.41	82.92	Config	Model
CWD	54.33	12.46	75.43	82.64	Config	Model
BPKD(Ours)	54.33	12.46	75.72	83.65	Config	Model

ADE20K 512x512 80K

Methods	FLOPs(G)	Parameters(M)	mIoU(%)	mAcc(%)	Config	ckpt
T: PSPNet-R101	256.89	68.07	44.39	54.75	Config	Model
S: PSPnet-R18	54.53	12.82	33.30	42.58	Config	Model
SKDS	54.53	12.82	34.49	44.28	Config	Model
IFVD	54.53	12.82	34.54	44.26	Config	Model
CIRKD	54.53	12.82	35.07	45.38	Config	Model
CWD	54.53	12.82	37.02	46.33	Config	Model
BPKD(Ours)	54.53	12.82	38.51	47.70	Config	Model
T: HRNetV2P-W48	95.64	65.95	42.02	53.52	Config	Model
S: HRNetV2P-W18S	10.49	3.97	31.38	41.39	Config	Model
SKDS	10.49	3.97	32.57	43.22	Config	Model
IFVD	10.49	3.97	32.66	43.23	Config	Model
CIRKD	10.49	3.97	33.06	44.30	Config	Model
CWD	10.49	3.97	34.00	42.76	Config	Model
BPKD(Ours)	10.49	3.97	35.31	46.11	Config	Model
T:DeeplabV3P-R101	255.67	62.68	45.47	56.41	Config	Model
S:DeeplabV3P+MV2	69.60	15.35	31.56	45.14	Config	Model
SKDS	69.60	15.35	32.49	46.47	Config	Model
IFVD	69.60	15.35	32.11	46.07	Config	Model
CIRKD	69.60	15.35	32.24	46.09	Config	Model
CWD	69.60	15.35	35.12	49.76	Config	Model
BPKD(Ours)	69.60	15.35	35.49	53.84	Config	Model
T: ISANet-R101	228.21	56.80	43.80	54.39	Config	Model
S: ISANet-R18	54.33	12.46	31.15	41.21	Config	Model
SKDS	54.33	12.46	32.16	41.80	Config	Model
IFVD	54.33	12.46	32.78	42.61	Config	Model
CIRKD	54.33	12.46	32.82	42.71	Config	Model
CWD	54.33	12.46	37.56	45.79	Config	Model
BPKD(Ours)	54.33	12.46	38.73	47.92	Config	Model

Pascal Context 59 480*480 80k

Methods	FLOPs(G)	Parameters(M)	mIoU(%)	mAcc(%)	Config	ckpt
T: PSPNet-R101	256.89	68.07	52.47	63.15	Config	Model
S:PSPnet-R18	54.53	12.82	43.79	54.46	Config	Model
SKDS	54.53	12.82	45.08	55.56	Config	Model
IFVD	54.53	12.82	45.97	56.6	Config	Model
CIRKD	54.53	12.82	45.62	56.15	Config	Model
CWD	54.53	12.82	45.99	55.56	Config	Model
BPKD(Ours)	54.53	12.82	46.82	56.29	Config	Model
T: HRNetV2P-W48	95.64	65.95	51.12	61.39	Config	Model
S:HRNetV2P-W18S	10.49	3.97	40.62	51.43
SKDS	10.49	3.97	41.54	52.18	Config	Model
IFVD	10.49	3.97	41.55	52.24	Config	Model
CIRKD	10.49	3.97	42.02	52.88	Config	Model
CWD	10.49	3.97	42.89	53.37	Config	Model
BPKD(Ours)	10.49	3.97	43.96	54.51	Config	Model
T:DeeplabV3P-R101	255.67	62.68	53.20	64.04	Config	Model
S:DeeplabV3P+MV2	69.60	15.35	41.01	52.92	Config	Model
SKDS	69.60	15.35	42.07	55.06	Config	Model
IFVD	69.60	15.35	41.73	54.34	Config	Model
CIRKD	69.60	15.35	42.25	55.12	Config	Model
CWD	69.60	15.35	43.74	56.37	Config	Model
BPKD(Ours)	69.60	15.35	46.23	58.12	Config	Model
T:ISANet-R101	228.21	56.80	53.41	64.04	Config	Model
S: ISANet-R18	54.33	12.46	44.05	54.67	Config	Model
SKDS	54.33	12.46	45.69	56.27	Config	Model
IFVD	54.33	12.46	46.75	56.4	Config	Model
CIRKD	54.33	12.46	45.83	56.11	Config	Model
CWD	54.33	12.46	46.76	56.48	Config	Model
BPKD(Ours)	54.33	12.46	47.25	56.81	Config	Model

License

For academic use, this project is licensed under the 2-clause BSD License - see the LICENSE file for details. For commercial use, please contact the corresponding author.

Citation

@article{liu2023bpkd,
  title={BPKD: Boundary Privileged Knowledge Distillation For Semantic Segmentation},
  author={Liu, Liyang and Wang, Zihan and Phan, Minh Hieu and Zhang, Bowen and Liu, Yifan},
  journal={IEEE/CVF Winter Conference on Applications of Computer Vision 2024},
  year={2023}
}

Code

Training Code Coming Soon ...