Training a Large Language Model (LLM) for Tumor Segmentation using Concatenation

In this post, we’ll walk through how to train a Large Language Model (LLM) for tumor segmentation using concatenation. The model will accept four MRI images (from different modalities) and output bounding box coordinates that specify the tumor’s location. We’ll define how to process the images and train the model with the appropriate bounding box outputs.

Dataset Structure and Input Format

We will use a prompt-based input where the model is given four MRI images, and the task is to predict the bounding box coordinates of the tumor. The prompt will look like this:

Each image is represented by a placeholder (<ImageHere>) and the model will need to predict the coordinates of a bounding box around the tumor.

"<Img1><ImageHere></Img1><Img2><ImageHere></Img2><Img3><ImageHere></Img3><Img4><ImageHere></Img4> where is the tumor? <box>"

Data Preparation

The dataset will consist of MRI images from four modalities and a summary file (summary.jsonl) containing the ground truth bounding box coordinates.

Example structure of summary.jsonl:

{
  "t2f Sagittal_path": "BraTS-GLI-00000-000_2_0.jpg",
  "t2w Sagittal_path": "BraTS-GLI-00000-000_2_2.jpg",
  "t1n Sagittal_path": "BraTS-GLI-00000-000_2_3.jpg",
  "t1c Sagittal_path": "BraTS-GLI-00000-000_2_4.jpg",
  "box": [[96, 84, 68, 68]]
}

This JSON contains the paths to the four MRI modalities and the ground truth bounding box coordinates.

Code Implementation

Conclusion

This post walks you through the process of training an LLM to predict bounding boxes for tumor segmentation using concatenated MRI inputs. With the correct setup and dataset, you can fine-tune the model to perform well on your specific medical imaging tasks.