hi @qubvel is there any update on this?
I've just trained a model using this branch and it worked.

@Rusteam is the code merged into the main repo??i want to use this model to fine-tune my data?

It's not. Not sure if @qubvel has had a chance to look into this PR. You could use my fork in the meanwhile. And do let me know how your fine-Tuning goes because I haven't had much success so far.

@Rusteam how to train a model ,can u give some outlines?as author is not responding pls help me to train a model.. I have sent u an mail pls give a look

make sure you install this package from my fork pip instal git+https://github.com/Rusteam/segmentation_models.pytorch.git@sam and then initialize your model as usual create_model("SAM", "sam-vit_b", encoder_weights=None, **kwargs) and run your training. You could pass weights="sa-1b" in kwargs if you want to fine-tune from pre-trained weights.

So far I have been able to train the model, but I can't say it's learning. I'm still struggling there. Also I cannot fit more than 1 sample per batch on a 32gb gpu with a 512 input size.

@Rusteam how about this: https://github.com/tianrun-chen/SAM-Adapter-PyTorch

thanks for sharing, I'll try it if my current approach does not work. I've able to get some learning with this transformers notebook

Hi @Rusteam, thanks a lot for your contribution and sorry for the delay, I am going to review the request and will let you know

Hey hey hey. While this solution worked I can't say the model was able to learn on my data. We might need to use the version before my ddp adjustments or make the model handle points and boxes as inputs, or use Sam image encoder with unet or other architectures.

Yes, I was actually thinking about just pre-trained encoder integration, did you test it?

can we use this model to train on custom data??

@qubvel It didn't work with Unet yet, but I can make it work. Which models would be essential to integrate?

@Rusteam @qubvel can we use this model to train on custom data??

that was my intention as well, but I was unable to make it learn without passing box/point prompts. However, when passing a prompt along with input image, it does learn. We might need to integrate multiple inputs to forward() call for it to work, or just use sam's image encoder with other arches like Unet

The following worked for me:
-git clone the sam branch,
-modify the sam.py file like below to get rid of the errors:
-change def forward(self, x: torch.Tensor) -> list[torch.Tensor]: to def forward(self, x: torch.Tensor):

• import segmentation_models_pytorch as smp (python file in same folder as git clone branch)
• smp.create_model("Unet", "sam-vit_b", encoder_weights="sa-1b", encoder_depth=4, decoder_channels=[256, 128, 64, 32])
• Try training
  What did not work -
• For me, I tried fine tuning with 2 RTX A6000 GPU with batch size of 2 on the ACDC data (https://www.creatis.insa-lyon.fr/Challenge/acdc/databases.html) but my Dice loss did not improve after 700 epochs. (Maybe some other setting works, but I did not have time to recreate it)

@qubvel hey any updates?

Please add this, this library hasn't have new features for a long time

This PR is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 15 days.

Is this PR ready?

It's ready.

The current PR seems to work with image with the size of 1024x1024 only.

Yes, as the original Sam model

This PR is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 15 days.

Any progress on this?

Why is the library dying? no new updates in a long time

This PR is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 15 days.

@qubvel can you merge this? It did work

@Rusteam Consider contributing this to TorchSeg

@qubvel should we revive this PR? @giswqs may be interested in this.

A relevant PR: huggingface/transformers#32317

I think you can already do this because timm supports the SAM ViT weights like:

Unet("tu-samvit_base_patch16.sa1b")

But I'm not sure how well SAM works with U-Net instead of their own custom decoder.

Agreed, it's likely highly dependent on the prompt embeddings as well.

I don't think SAM works out of the box like this.