Zihan Li
LViT: Language meets Vision Transformer in Medical Image Segmentation
Li, Zihan; Li, Yunxiang; Li, Qingde; Wang, Puyang; Guo, Dazhou; Lu, Le; Jin, Dakai; Zhang, You; Hong, Qingqi
Authors
Yunxiang Li
Dr Qingde Li Q.Li@hull.ac.uk
Lecturer
Puyang Wang
Dazhou Guo
Le Lu
Dakai Jin
You Zhang
Qingqi Hong
Abstract
Deep learning has been widely used in medical image segmentation and other aspects. However, the performance of existing medical image segmentation models has been limited by the challenge of obtaining sufficient high-quality labeled data due to the prohibitive data annotation cost. To alleviate this limitation, we propose a new text-augmented medical image segmentation model LViT (Language meets Vision Transformer). In our LViT model, medical text annotation is incorporated to compensate for the quality deficiency in image data. In addition, the text information can guide to generate pseudo labels of improved quality in the semi-supervised learning. We also propose an Exponential Pseudo label Iteration mechanism (EPI) to help the Pixel-Level Attention Module (PLAM) preserve local image features in semi-supervised LViT setting. In our model, LV (Language-Vision) loss is designed to supervise the training of unlabeled images using text information directly. For evaluation, we construct three multimodal medical segmentation datasets (image + text) containing X-rays and CT images. Experimental results show that our proposed LViT has superior segmentation performance in both fully-supervised and semi-supervised setting. The code and datasets are available at https://github.com/HUANGLIZI/LViT.
Citation
Li, Z., Li, Y., Li, Q., Wang, P., Guo, D., Lu, L., Jin, D., Zhang, Y., & Hong, Q. (2024). LViT: Language meets Vision Transformer in Medical Image Segmentation. IEEE Transactions on Medical Imaging, 43(1), 96-107. https://doi.org/10.1109/TMI.2023.3291719
Journal Article Type | Article |
---|---|
Acceptance Date | Jul 3, 2023 |
Online Publication Date | Jul 3, 2023 |
Publication Date | 2024-01 |
Deposit Date | Jan 19, 2024 |
Publicly Available Date | Feb 6, 2024 |
Journal | IEEE Transactions on Medical Imaging |
Print ISSN | 0278-0062 |
Electronic ISSN | 1558-254X |
Publisher | Institute of Electrical and Electronics Engineers |
Peer Reviewed | Peer Reviewed |
Volume | 43 |
Issue | 1 |
Pages | 96-107 |
DOI | https://doi.org/10.1109/TMI.2023.3291719 |
Keywords | Vision-language; Medical image segmentation; Semi-supervised learning |
Public URL | https://hull-repository.worktribe.com/output/4338534 |
Files
Accepted manuscript
(1.4 Mb)
PDF
Copyright Statement
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
You might also like
ScribFormer: Transformer Makes CNN Work Better for Scribble-based Medical Image Segmentation
(2024)
Journal Article
Using outlier elimination to assess learning-based correspondence matching methods
(2024)
Journal Article
Consensus Adversarial Defense Method Based on Augmented Examples
(2022)
Journal Article
Downloadable Citations
About Repository@Hull
Administrator e-mail: repository@hull.ac.uk
This application uses the following open-source libraries:
SheetJS Community Edition
Apache License Version 2.0 (http://www.apache.org/licenses/)
PDF.js
Apache License Version 2.0 (http://www.apache.org/licenses/)
Font Awesome
SIL OFL 1.1 (http://scripts.sil.org/OFL)
MIT License (http://opensource.org/licenses/mit-license.html)
CC BY 3.0 ( http://creativecommons.org/licenses/by/3.0/)
Powered by Worktribe © 2025
Advanced Search