View on huggingface.co

model = AutoAdapterModel.from_pretrained("roberta-base")
model.load_adapter("UKP-SQuARE/SearchQA_Adapter_RoBERTa", source="hf")

Description

Description

This is the single-dataset adapter for the TriviaQA partition of the MRQA 2019 Shared Task Dataset. The adapter was created by Friedman et al. (2021) and should be used with the roberta-base encoder.

The UKP-SQuARE team created this model repository to simplify the deployment of this model on the UKP-SQuARE platform. The GitHub repository of the original authors is https://github.com/princeton-nlp/MADE

Usage

This model contains the same weights as https://huggingface.co/princeton-nlp/MADE/resolve/main/single_dataset_adapters/SearchQA/model.pt. The only difference is that our repository follows the standard format of AdapterHub. Therefore, you could load this model as follows:

from transformers import RobertaForQuestionAnswering, RobertaTokenizerFast

model = RobertaForQuestionAnswering.from_pretrained("roberta-base")
model.load_adapter("UKP-SQuARE/SearchQA_Adapter_RoBERTa",  source="hf")
model.set_active_adapters("SearchQA")

tokenizer = RobertaTokenizerFast.from_pretrained('roberta-base')

pipe = pipeline("question-answering", model=model, tokenizer=tokenizer)
pipe({"question": "What is the capital of Germany?",  "context": "The capital of Germany is Berlin."})

Note you need the adapter-transformers library https://adapterhub.ml

Evaluation

Friedman et al. report an F1 score of 85.1 on SearchQA.

Please refer to the original publication for more information.

Citation

Single-dataset Experts for Multi-dataset Question Answering (Friedman et al., EMNLP 2021)

Properties

Pre-trained model
roberta-base
Adapter type
Prediction Head
  Yes
Task
Question Answering
Dataset

Architecture

{
  "adapter_residual_before_ln": false,
  "cross_adapter": false,
  "factorized_phm_W": true,
  "factorized_phm_rule": false,
  "hypercomplex_nonlinearity": "glorot-uniform",
  "init_weights": "bert",
  "inv_adapter": null,
  "inv_adapter_reduction_factor": null,
  "is_parallel": false,
  "learn_phm": true,
  "leave_out": [],
  "ln_after": false,
  "ln_before": false,
  "mh_adapter": true,
  "non_linearity": "swish",
  "original_ln_after": true,
  "original_ln_before": false,
  "output_adapter": true,
  "phm_bias": true,
  "phm_c_init": "normal",
  "phm_dim": 4,
  "phm_init_range": 0.0001,
  "phm_layer": false,
  "phm_rank": 1,
  "reduction_factor": 16,
  "residual_before_ln": true,
  "scaling": 1.0,
  "shared_W_phm": false,
  "shared_phm_rule": true,
  "use_gating": false
}

Citations