Using TableTransformer in Standalone Mode Without Hugging Face Hub Access

Question

I need help with using the Hugging Face transformers library, specifically with the TableTransformer model. Due to a network firewall, I cannot directly download models from the Hugging Face Hub (all requests are blocked), so I need to use it in standalone mode.

Environment Details:

• Python Version: 3.12.3
• Pip Version: 23.2.1
• OS: Windows 11 (Version 23H2)
• Transformers Version: 4.49.0

What I Tried

1. Downloaded the model manually from Hugging Face and placed the table-transformer-detection model in a local models/ directory.

2. Loaded the model using from_pretrained:

   from transformers import TableTransformerModel
   model = TableTransformerModel.from_pretrained("./models/table-transformer-detection")
   

3. Encountered an error: The model still attempts to download dependencies from the Hub:

   urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443):
   Max retries exceeded with url: /timm/resnet18.a1_in1k/resolve/main/pytorch_model.bin
   

   Issue: The TableTransformer uses a ResNet backbone that it tries to fetch online.

Attempted Fixes

• Manually downloaded the resnet-50 model and tried specifying it as the backbone:

  from transformers import TableTransformerConfig, TableTransformerModel, ResNetConfig
  resnet_50_path = "./models/resnet-50"
  table_detection_path = "./models/table-transformer-detection"
  backbone_config = ResNetConfig.from_pretrained(resnet_50_path)
  config = TableTransformerConfig(backbone_config=backbone_config)
  model = TableTransformerModel(config).from_pretrained(table_detection_path)
  

  Error:

  ValueError: You can't specify both `backbone` and `backbone_config`.
  

• Tried setting backbone=None to resolve the conflict:

  config = TableTransformerConfig(backbone_config=backbone_config, backbone=None)
  

  Error:

  ValueError: You can't specify both `backbone_config` and `use_timm_backbone`.
  

• Adjusted settings to disable pretrained backbones:

  import os
  os.environ["HF_HUB_OFFLINE"] = "1"
  os.environ["TRANSFORMERS_OFFLINE"] = "1"
  from transformers import ResNetConfig, TableTransformerConfig, TableTransformerForObjectDetection, AutoImageProcessor
  backbone_resnet50_path = "./models/resnet-50"
  backbone_resnet50_config = ResNetConfig.from_pretrained(backbone_resnet50_path)
  backbone_resnet50_config.name_or_path=backbone_resnet50_path
  print("backbone resnet50 config:", backbone_resnet50_config)
  table_config = TableTransformerConfig(
   backbone_config=backbone_resnet50_config,
   backbone=None,
   use_timm_backbone=False,
   use_pretrained_backbone=False
  )
  table_model_table = "./models/table-transformer-detection"
  processor = AutoImageProcessor.from_pretrained(table_model_table)
  table_model = TableTransformerForObjectDetection(table_config).from_pretrained(
   pretrained_model_name_or_path=table_model_table,
   config=table_config,
   ignore_mismatched_sizes=True
  )
  

  Final Error: The model initializes, but I receive a warning about mismatched shapes in the checkpoint and uninitialized layers:

  You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
  Some weights of TableTransformerForObjectDetection were not initialized from the model checkpoint at ./models/table-transformer-detection and are newly initialized because the shapes did not match:
  - model.input_projection.weight: found shape torch.Size([256, 512, 1, 1]) in the checkpoint and torch.Size([256, 2048, 1, 1]) in the model instantiated
  - model.query_position_embeddings.weight: found shape torch.Size([15, 256]) in the checkpoint and torch.Size([100, 256]) in the model instantiated
  You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
  

Summary of My Findings After extensive testing and troubleshooting, I have managed to prevent the model from attempting to download files from the Hugging Face Hub. However, I suspect that my custom configuration has broken several things, as I now face shape mismatches and potentially other issues due to the manual setup.

My Question

• How does the TableTransformer backbone configuration actually work?
• How can I load the TableTransformer model entirely offline, ensuring all dependencies (like ResNet) are resolved locally?
• Is this even possible with the current implementation of transformers, or does the library inherently require online access?

Would really appreciate any guidance from someone familiar with Hugging Face's model loading mechanisms!

Thanks in advance!
Enzo