Adaptive Convolution and Feature Aggregation for Single-Image Shadow Removal
Keywords:
Shadow Removal, Adaptive Convolution, Feature Aggregation, Deep Learning, Illumination ConsistencyAbstract
Shadows degrade image quality and hinder computer vision tasks. Existing deep learning methods suffer from fine-detail loss due to downsampling and indiscriminate processing of shadow/non-shadow regions, causing unwanted alterations. To address these issues, this thesis proposes two complementary approaches. First, the Adaptive Alignment and Illumination-Aware Convolution (AAIC) framework uses a Feature Alignment Module (FAM) to recover lost details and an Illumination-Aware Weighting Module (IWM) for spatially varying convolution, achieving high quantitative performance on ISTD+ and SRD datasets. Second, the Feature Aggregation Shadow Removal Network (FASR-Net) reconstructs shadow-free images via learned fusion rather than end-to-end transformation, employing a Detail Feature Extractor (DFE), Dual-Branch Aggregation Weight Generator (DAWG), and Feature Enhancement (FE) modules to preserve non-shadow regions and ensure global consistency. Both methods are extensively evaluated using RMSE and SSIM, outperforming state-of-the-art techniques. Ablation studies validate each component, and the methods improve foreground segmentation in videos and generalize to remote sensing images without fine-tuning.
References
S. Wang and H. Zhang, "Clustering-based shadow edge detection in a single color image," in Proc. MEC, 2013, pp. 1038–1041.
J. M. Wang, Y. L. Li, and X. Zhang, "Shadow-based 3D reconstruction from single images," IEEE TPAMI, vol. 41, no. 5, pp. 1123–1136, 2019.
S. H. Lee and K. M. Lee, "Light source localization using shadow cues," in CVPR, 2018, pp. 2345–2353.
A. Prati, I. Mikic, M. M. Trivedi, and R. Cucchiara, "Detecting moving shadows: algorithms and evaluation," IEEE TPAMI, vol. 25, no. 7, pp. 918–923, 2003.
H. Zhang, K. Sun, and W. Li, "Shadow detection and removal in remote sensing images: A survey," ISPRS JPRS, vol. 168, pp. 1–16, 2020.
F. S. H. Almutairi and M. S. Sarfraz, "Shadows in autonomous driving: A survey," IEEE TITS, vol. 23, no. 8, pp. 10234–10248, 2022.
G. D. Finlayson, S. D. Hordley, and M. S. Drew, "Removing shadows from images," in ECCV, 2002, pp. 823–836.
G. D. Finlayson, S. D. Hordley, C. Lu, et al., "On the removal of shadows from images," IEEE TPAMI, vol. 28, no. 1, pp. 59–68, 2005.
Y. Shor and D. Lischinski, "The shadow meets the mask: Pyramid-based shadow removal," in Computer Graphics Forum, vol. 27, no. 2, 2008, pp. 577–586.
R. Guo, Q. Dai, and D. Hoiem, "Single-image shadow detection and removal using paired regions," in CVPR, 2011, pp. 2033–2040.
C. Xiao, D. Xiao, L. Zhang, et al., "Efficient shadow removal using subregion matching illumination transfer," Computer Graphics Forum, vol. 32, no. 7, pp. 421–430, 2013.
L. Zhang, Q. Zhang, and C. Xiao, "Shadow remover: Image shadow removal based on illumination recovering optimization," IEEE TIP, vol. 24, no. 11, pp. 4623–4636, 2015.
M. Gryka, M. Terry, and G. J. Brostow, "Learning to remove soft shadows," ACM TOG, vol. 34, no. 5, pp. 1–15, 2015.
N. Su, Y. Zhang, S. Tian, et al., "Shadow detection and removal for occluded object information recovery in urban high-resolution panchromatic satellite images," IEEE JSTARS, vol. 9, no. 6, pp. 2568–2582, 2016.
K. He, R. Zhen, J. Yan, et al., "Single-image shadow removal using 3D intensity surface modeling," IEEE TIP, vol. 26, no. 12, pp. 6046–6060, 2017.
O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," in MICCAI, 2015, pp. 234–241.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, et al., "Generative adversarial nets," in NeurIPS, 2014, vol. 27.
J.-Y. Zhu, T. Park, P. Isola, et al., "Unpaired image-to-image translation using cycle-consistent adversarial networks," in ICCV, 2017, pp. 2223–2232.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, et al., "An image is worth 16x16 words: Transformers for image recognition at scale," in ICLR, 2021.
H. Le and D. Samaras, "Shadow removal via shadow image decomposition," in ICCV, 2019, pp. 8577–8586.
L. Fu, C. Zhou, Q. Guo, et al., "Auto-exposure fusion for single-image shadow removal," in CVPR, 2021, pp. 10571–10580.
Y. Zhu, Z. Xiao, Y. Fang, et al., "Efficient model-driven network for shadow removal," in AAAI, vol. 36, no. 3, 2022, pp. 3635–3643.
X. Cun, C.-M. Pun, and C. Shi, "Towards ghost-free shadow removal via dual hierarchical aggregation network and shadow matting GAN," in AAAI, vol. 34, no. 7, 2020, pp. 10680–10687.
Y. Jin, A. Sharma, and R. T. Tan, "DC-ShadowNet: Single-image hard and soft shadow removal using unsupervised domain-classifier guided network," in ICCV, 2021, pp. 5007–5016.
Y. Xu, M. Lin, H. Yang, et al., "Shadow-aware dynamic convolution for shadow removal," Pattern Recognition, vol. 146, p. 109969, 2024.
J. Wan, H. Yin, Z. Wu, et al., "Style-guided shadow removal," in ECCV, 2022, pp. 361–378.
J. Liu, Q. Wang, H. Fan, et al., "A decoupled multi-task network for shadow removal," IEEE TMM, vol. 25, pp. 9449–9463, 2023.
J. Liu, Q. Wang, H. Fan, et al., "A shadow imaging bilinear model and three-branch residual network for shadow removal," IEEE TNNLS, pp. 1–15, 2023.
X. Hu, Y. Jiang, C.-W. Fu, et al., "Mask-ShadowGAN: Learning to remove shadows from unpaired data," in ICCV, 2019, pp. 2472–2481.
F. A. Vasluianu, A. Romero, and L. V. Gool, "Self-supervised shadow removal," arXiv:2010.11619, 2020.
C. Tan and X. Feng, "Unsupervised shadow removal using target-consistency generative adversarial network," arXiv:2010.01291, 2020.
J. Wang, X. Li, and J. Yang, "Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal," in CVPR, 2018, pp. 1788–1797.
Z. Chen, C. Long, L. Zhang, et al., "CANet: A context-aware network for shadow removal," in ICCV, 2021, pp. 4743–4752.
R. Abiko and M. Ikehara, "Channel attention GAN trained with enhanced dataset for single-image shadow removal," IEEE Access, vol. 10, pp. 12322–12333, 2022.
J. Wan, H. Yin, Z. Wu, et al., "CRFormer: A cross-region transformer for shadow removal," arXiv:2207.01600, 2022.
L. Guo, C. Wang, W. Yang, et al., "ShadowDiffusion: When degradation prior meets diffusion model for shadow removal," in CVPR, 2023, pp. 14049–14058.
M. Sen, S. P. Chermala, N. N. Nagori, et al., "SHARDS: Efficient shadow removal using dual stage network for high-resolution images," in WACV, 2023, pp. 1809–1817.
K. Niu, Y. Liu, E. Wu, et al., "A boundary-aware network for shadow removal," IEEE TMM, vol. 25, pp. 6782–6793, 2023.
L. Qu, J. Tian, S. He, et al., "DeshadowNet: A multi-context embedding deep network for shadow removal," in CVPR, 2017, pp. 2308–2316.
L. Zhang, C. Long, X. Zhang, et al., "RIS-GAN: Explore residual and illumination with generative adversarial networks for shadow removal," in AAAI, vol. 34, no. 7, 2020, pp. 12829–12836.
Y. Zhu, J. Huang, X. Fu, et al., "Bijective mapping network for shadow removal," in CVPR, 2022, pp. 5617–5626.
Z. Liu, H. Yin, Y. Mi, et al., "Shadow removal by a lightness-guided network with training on unpaired data," IEEE TIP, vol. 30, 2021.
Q. Yu, N. Zheng, J. Huang, et al., "CNSNet: A cleanness-navigated-shadow network for shadow removal," in ECCV Workshops, 2023, pp. 221–238.
S. Dasgupta, A. Das, S. Yogamani, et al., "UnShadowNet: Illumination critic guided contrastive learning for shadow removal," IEEE Access, vol. 11, pp. 87760–87774, 2023.
Hu, L. Shen, S. Albanie, et al., "Squeeze-and-excitation networks," IEEE TPAMI, vol. 42, no. 8, pp. 2011–2023, 2020.
X. Wang, R. Girshick, A. Gupta, et al., "Non-local neural networks," in CVPR, 2018, pp. 7794–7803.
L. Guo, S. Huang, D. Liu, et al., "ShadowFormer: Global context helps image shadow removal," in AAAI, vol. 37, no. 1, 2023, pp. 710–718.
Y. Jin, W. Yang, W. Ye, et al., "DeS3: Adaptive attention-driven self and soft shadow removal using ViT similarity," arXiv:2211.08089, 2022.
L. Guo, C. Wang, W. Yang, et al., "ShadowDiffusion: When degradation prior meets diffusion model for shadow removal," in CVPR, 2023, pp. 14049–14058.
J. Long, E. Shelhamer, and T. Darrell, "Fully convolutional networks for semantic segmentation," in CVPR, 2015, pp. 3431–3440.
V. Badrinarayanan, A. Kendall, and R. Cipolla, "SegNet: A deep convolutional encoder-decoder architecture for image segmentation," IEEE TPAMI, vol. 39, no. 12, pp. 2481–2495, 2017.
L.-C. Chen, G. Papandreou, I. Kokkinos, et al., "Semantic image segmentation with deep convolutional nets and fully connected CRFs," arXiv:1412.7062, 2014.
L.-C. Chen, G. Papandreou, I. Kokkinos, et al., "DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs," IEEE TPAMI, vol. 40, no. 4, pp. 834–848, 2017.
J. Wang, K. Sun, T. Cheng, et al., "Deep high-resolution representation learning for visual recognition," IEEE TPAMI, vol. 43, no. 10, pp. 3349–3364, 2021.
J. Fu, J. Liu, H. Tian, et al., "Dual attention network for scene segmentation," in CVPR, 2019, pp. 3146–3154.
Y. Cao, J. Xu, S. Lin, et al., "GCNet: Non-local networks meet squeeze-excitation networks and beyond," in ICCVW, 2019, pp. 1971–1980.
Z. Huang, X. Wang, L. Huang, et al., "CCNet: Criss-cross attention for semantic segmentation," in ICCV, 2019, pp. 603–612.
J. Dai, H. Qi, Y. Xiong, et al., "Deformable convolutional networks," in ICCV, 2017, pp. 764–773.
X. Zhu, H. Hu, S. Lin, et al., "Deformable ConvNets v2: More deformable, better results," in CVPR, 2019, pp. 9308–9316.
H. Su, V. Jampani, D. Sun, et al., "Pixel-adaptive convolutional neural networks," in CVPR, 2019, pp. 11158–11167.
J. Chen, X. Wang, Z. Guo, et al., "Dynamic region-aware convolution," in CVPR, 2021, pp. 8060–8069.
Z. Wang, A. C. Bovik, H. R. Sheikh, et al., "Image quality assessment: from error visibility to structural similarity," IEEE TIP, vol. 13, no. 4, pp. 600–612, 2004.
Z. Peng, W. Huang, S. Gu, et al., "Conformer: Local features coupling global representations for visual recognition," in ICCV, 2021, pp. 367–376.
S. Chatterjee and A. S. Hadi, "Influential observations, high leverage points, and outliers in linear regression," Statistical Science, pp. 379–393, 1986.
R. Rombach, A. Blattmann, D. Lorenz, et al., "High-resolution image synthesis with latent diffusion models," in CVPR, 2022, pp. 10684–10695.
