View on huggingface.co 

model = AutoAdapterModel.from_pretrained("distilroberta-base")
model.load_adapter("yoh/distilroberta-base-sept-adapter", source="hf")

Description

Adapter yoh/distilroberta-base-sept-adapter for distilroberta-base

An adapter for the distilroberta-base model that was trained on the AllNLI, Sentence compression and Stackexchange duplicate question datasets (see information here).

This adapter was created for usage with the adapter-transformers library. See this paper and repository for more information on the tasks.

Usage

First, install adapter-transformers and sentence-transformers:

pip install -U adapter-transformers sentence-transformers

Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. More

Now, the adapter can be loaded and activated like this:

from sentence_transformers import SentenceTransformer, models

# Load pre-trained model
word_embedding_model = models.Transformer("distilroberta-base")
# Load and activate adapter
word_embedding_model.auto_model.load_adapter("yoh/distilroberta-base-sept-adapter", source="hf", set_active=True)
# Create sentence transformer
pooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension(), pooling_mode='mean')
model = SentenceTransformer(modules=[word_embedding_model, pooling_model])

Architecture & Training

See this paper

Evaluation results

See this paper

Citation

@article{huang2023adasent,
 title={AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classification}, 
 author={Yongxin Huang and Kexin Wang and Sourav Dutta and Raj Nath Patel and Goran Glavaš and Iryna Gurevych},
 journal = {ArXiv preprint},
 url = {https://arxiv.org/abs/2311.00408},
 volume = {abs/2311.00408},
 year={2023},
}

Properties

Pre-trained model
distilroberta-base
Adapter type
Task
Prediction Head
  No
Task
Natural Language Inference
Dataset
MultiNLI

Architecture

{
  "adapter_residual_before_ln": false,
  "cross_adapter": false,
  "factorized_phm_W": true,
  "factorized_phm_rule": false,
  "hypercomplex_nonlinearity": "glorot-uniform",
  "init_weights": "mam_adapter",
  "inv_adapter": null,
  "inv_adapter_reduction_factor": null,
  "is_parallel": true,
  "learn_phm": true,
  "leave_out": [],
  "ln_after": false,
  "ln_before": false,
  "mh_adapter": false,
  "non_linearity": "relu",
  "original_ln_after": true,
  "original_ln_before": false,
  "output_adapter": true,
  "phm_bias": true,
  "phm_c_init": "normal",
  "phm_dim": 4,
  "phm_init_range": 0.0001,
  "phm_layer": false,
  "phm_rank": 1,
  "reduction_factor": 2,
  "residual_before_ln": true,
  "scaling": 4.0,
  "shared_W_phm": false,
  "shared_phm_rule": true,
  "use_gating": false
}

Citations

Task
@misc{williams2017broadcoverage,
  title={A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference},
  author={Adina Williams and Nikita Nangia and Samuel R. Bowman},
  year={2017},
  eprint={1704.05426},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}