View on huggingface.co 

model = AutoAdapterModel.from_pretrained("bert-base-uncased")
model.load_adapter("AdapterHub/bert-base-uncased-pf-cq", source="hf")

Description

Adapter AdapterHub/bert-base-uncased-pf-cq for bert-base-uncased

An adapter for the bert-base-uncased model that was trained on the qa/cq dataset and includes a prediction head for question answering.

This adapter was created for usage with the adapter-transformers library.

Usage

First, install adapter-transformers:

pip install -U adapter-transformers

Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. More

Now, the adapter can be loaded and activated like this:

from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("bert-base-uncased")
adapter_name = model.load_adapter("AdapterHub/bert-base-uncased-pf-cq", source="hf")
model.active_adapters = adapter_name

Architecture & Training

The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer. In particular, training configurations for all tasks can be found here.

Evaluation results

Refer to the paper for more information on results.

Citation

If you use this adapter, please cite our paper "What to Pre-Train on? Efficient Intermediate Task Selection":

@inproceedings{poth-etal-2021-pre,
    title = "{W}hat to Pre-Train on? {E}fficient Intermediate Task Selection",
    author = {Poth, Clifton  and
      Pfeiffer, Jonas  and
      R{"u}ckl{'e}, Andreas  and
      Gurevych, Iryna},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.827",
    pages = "10585--10605",
}

Properties

Pre-trained model
bert-base-uncased
Adapter type
Task
Prediction Head
  Yes
Task
Question Answering
Dataset
ComplexQuestions

Architecture

{
  "adapter_residual_before_ln": false,
  "cross_adapter": false,
  "inv_adapter": null,
  "inv_adapter_reduction_factor": null,
  "leave_out": [],
  "ln_after": false,
  "ln_before": false,
  "mh_adapter": false,
  "non_linearity": "relu",
  "original_ln_after": true,
  "original_ln_before": true,
  "output_adapter": true,
  "reduction_factor": 16,
  "residual_before_ln": true
}

Citations

Task
@inproceedings{baoConstraintbasedQuestionAnswering2016,
  title = {Constraint-Based Question Answering with Knowledge Graph},
  booktitle = {{{COLING}} 2016, 26th International Conference on Computational Linguistics, Proceedings of the Conference: {{Technical}} Papers, December 11-16, 2016, Osaka, Japan},
  author = {Bao, Junwei and Duan, Nan and Yan, Zhao and Zhou, Ming and Zhao, Tiejun},
  editor = {Calzolari, Nicoletta and Matsumoto, Yuji and Prasad, Rashmi},
  year = {2016},
  pages = {2503--2514},
  publisher = {{Association for Computational Linguistics}},
  bibsource = {dblp computer science bibliography, https://dblp.org},
  biburl = {https://dblp.org/rec/conf/coling/BaoDYZZ16.bib},
  timestamp = {Wed, 06 May 2020 08:06:19 +0200}
}