View on

model = AutoAdapterModel.from_pretrained("bert-base-uncased")
model.load_adapter("AdapterHub/bert-base-uncased-pf-copa", source="hf")


Adapter AdapterHub/bert-base-uncased-pf-copa for bert-base-uncased

An adapter for the bert-base-uncased model that was trained on the comsense/copa dataset and includes a prediction head for multiple choice.

This adapter was created for usage with the adapter-transformers library.


First, install adapter-transformers:

pip install -U adapter-transformers

Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. More

Now, the adapter can be loaded and activated like this:

from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("bert-base-uncased")
adapter_name = model.load_adapter("AdapterHub/bert-base-uncased-pf-copa", source="hf")
model.active_adapters = adapter_name

Architecture & Training

The training code for this adapter is available at In particular, training configurations for all tasks can be found here.

Evaluation results

Refer to the paper for more information on results.


If you use this adapter, please cite our paper "What to Pre-Train on? Efficient Intermediate Task Selection":

    title={What to Pre-Train on? Efficient Intermediate Task Selection},
    author={Clifton Poth and Jonas Pfeiffer and Andreas Rücklé and Iryna Gurevych},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
    month = nov,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "",
    pages = "to appear",


Pre-trained model
Adapter type
Prediction Head
Common Sense Reasoning


  "adapter_residual_before_ln": false,
  "cross_adapter": false,
  "inv_adapter": null,
  "inv_adapter_reduction_factor": null,
  "leave_out": [],
  "ln_after": false,
  "ln_before": false,
  "mh_adapter": false,
  "non_linearity": "relu",
  "original_ln_after": true,
  "original_ln_before": true,
  "output_adapter": true,
  "reduction_factor": 16,
  "residual_before_ln": true


  title = {{{SemEval}}-2012 Task 7: {{Choice}} of Plausible Alternatives: {{An}} Evaluation of Commonsense Causal Reasoning},
  booktitle = {Proceedings of the 6th International Workshop on Semantic Evaluation, {{SemEval}}@{{NAACL}}-{{HLT}} 2012, Montr\'eal, Canada, June 7-8, 2012},
  author = {Gordon, Andrew S. and Kozareva, Zornitsa and Roemmele, Melissa},
  editor = {Agirre, Eneko and Bos, Johan and Diab, Mona T.},
  year = {2012},
  pages = {394--398},
  publisher = {{The Association for Computer Linguistics}},
  bibsource = {dblp computer science bibliography,},
  biburl = {},
  timestamp = {Mon, 16 Sep 2019 17:08:53 +0200}