model = AutoAdapterModel.from_pretrained("roberta-base") model.load_adapter("AdapterHub/roberta-base-pf-hellaswag", source="hf")
AdapterHub/roberta-base-pf-hellaswag
for roberta-baseAn adapter for the roberta-base
model that was trained on the comsense/hellaswag dataset and includes a prediction head for multiple choice.
This adapter was created for usage with the adapter-transformers library.
First, install adapter-transformers
:
pip install -U adapter-transformers
Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. More
Now, the adapter can be loaded and activated like this:
from transformers import AutoModelWithHeads
model = AutoModelWithHeads.from_pretrained("roberta-base")
adapter_name = model.load_adapter("AdapterHub/roberta-base-pf-hellaswag", source="hf")
model.active_adapters = adapter_name
The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer. In particular, training configurations for all tasks can be found here.
Refer to the paper for more information on results.
If you use this adapter, please cite our paper "What to Pre-Train on? Efficient Intermediate Task Selection":
@inproceedings{poth-etal-2021-what-to-pre-train-on,
title={What to Pre-Train on? Efficient Intermediate Task Selection},
author={Clifton Poth and Jonas Pfeiffer and Andreas Rücklé and Iryna Gurevych},
booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
month = nov,
year = "2021",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/2104.08247",
pages = "to appear",
}
{ "adapter_residual_before_ln": false, "cross_adapter": false, "inv_adapter": null, "inv_adapter_reduction_factor": null, "leave_out": [], "ln_after": false, "ln_before": false, "mh_adapter": false, "non_linearity": "relu", "original_ln_after": true, "original_ln_before": true, "output_adapter": true, "reduction_factor": 16, "residual_before_ln": true }
@inproceedings{zellers2019hellaswag, title={HellaSwag: Can a Machine Really Finish Your Sentence?}, author={Zellers, Rowan and Holtzman, Ari and Bisk, Yonatan and Farhadi, Ali and Choi, Yejin}, booktitle={Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics}, pages={4791--4800}, year={2019} }