model = AutoAdapterModel.from_pretrained("roberta-base") model.load_adapter("AdapterHub/roberta-base-pf-mnli", source="hf")
AdapterHub/roberta-base-pf-mnli
for roberta-baseAn adapter for the roberta-base
model that was trained on the nli/multinli dataset and includes a prediction head for classification.
This adapter was created for usage with the adapter-transformers library.
First, install adapter-transformers
:
pip install -U adapter-transformers
Note: adapter-transformers is a fork of transformers that acts as a drop-in replacement with adapter support. More
Now, the adapter can be loaded and activated like this:
from transformers import AutoModelWithHeads
model = AutoModelWithHeads.from_pretrained("roberta-base")
adapter_name = model.load_adapter("AdapterHub/roberta-base-pf-mnli", source="hf")
model.active_adapters = adapter_name
The training code for this adapter is available at https://github.com/adapter-hub/efficient-task-transfer. In particular, training configurations for all tasks can be found here.
Refer to the paper for more information on results.
If you use this adapter, please cite our paper "What to Pre-Train on? Efficient Intermediate Task Selection":
@inproceedings{poth-etal-2021-pre,
title = "{W}hat to Pre-Train on? {E}fficient Intermediate Task Selection",
author = {Poth, Clifton and
Pfeiffer, Jonas and
R{"u}ckl{'e}, Andreas and
Gurevych, Iryna},
booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
month = nov,
year = "2021",
address = "Online and Punta Cana, Dominican Republic",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.emnlp-main.827",
pages = "10585--10605",
}
{ "adapter_residual_before_ln": false, "cross_adapter": false, "inv_adapter": null, "inv_adapter_reduction_factor": null, "leave_out": [], "ln_after": false, "ln_before": false, "mh_adapter": false, "non_linearity": "relu", "original_ln_after": true, "original_ln_before": true, "output_adapter": true, "reduction_factor": 16, "residual_before_ln": true }
@misc{williams2017broadcoverage, title={A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference}, author={Adina Williams and Nikita Nangia and Samuel R. Bowman}, year={2017}, eprint={1704.05426}, archivePrefix={arXiv}, primaryClass={cs.CL} }