View on huggingface.co 

model = AutoAdapterModel.from_pretrained("roberta-base")
model.load_adapter("AdapterHub/roberta-base-pf-commonsense_qa", source="hf")

Description

Adapter AdapterHub/roberta-base-pf-commonsense_qa for roberta-base

An adapter for the roberta-base model that was trained on the comsense/csqa dataset and includes a prediction head for multiple choice.

This adapter was created for usage with the adapter-transformers library.

Usage

First, install adapter-transformers:

pip install -U adapter-transformers

Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. More

Now, the adapter can be loaded and activated like this:

from transformers import AutoModelWithHeads

model = AutoModelWithHeads.from_pretrained("roberta-base")
adapter_name = model.load_adapter("AdapterHub/roberta-base-pf-commonsense_qa", source="hf")
model.active_adapters = adapter_name

Architecture & Training

The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer. In particular, training configurations for all tasks can be found here.

Evaluation results

Refer to the paper for more information on results.

Citation

If you use this adapter, please cite our paper "What to Pre-Train on? Efficient Intermediate Task Selection":

@inproceedings{poth-etal-2021-what-to-pre-train-on,
    title={What to Pre-Train on? Efficient Intermediate Task Selection},
    author={Clifton Poth and Jonas Pfeiffer and Andreas Rücklé and Iryna Gurevych},
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
    month = nov,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/2104.08247",
    pages = "to appear",
}

Properties

Pre-trained model
roberta-base
Adapter type
Task
Prediction Head
  Yes
Task
Common Sense Reasoning
Dataset
Commonsense QA

Architecture

{
  "adapter_residual_before_ln": false,
  "cross_adapter": false,
  "inv_adapter": null,
  "inv_adapter_reduction_factor": null,
  "leave_out": [],
  "ln_after": false,
  "ln_before": false,
  "mh_adapter": false,
  "non_linearity": "relu",
  "original_ln_after": true,
  "original_ln_before": true,
  "output_adapter": true,
  "reduction_factor": 16,
  "residual_before_ln": true
}

Citations

Task
@inproceedings{talmor2019commonsenseqa,
  title={CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge},
  author={Talmor, Alon and Herzig, Jonathan and Lourie, Nicholas and Berant, Jonathan},
  booktitle={Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational
  Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  pages={4149--4158},
  year={2019}
}