[Adapt] [Seminar] Pitfalls and Solutions for Vocabulary-dependent Models

[贾琪]

Hi Adapters,


Large-scale pre-trained language models have significantly advanced the whole NLP community.  Each of them is always tied with a tokenization tool made up of a vocabulary and a huge embedding matrix. Although subword tokenizers are the de facto standard in modern NLP, there are still pitfalls of these widely-accepted tokenizers, possibly resulting in pre-trained language models with poor generalization ability and low robustness. 
In this seminar, I'll introduce the limitations of vocabulary-dependent models/tokenizers and introduce a paper named "HashFormers: Towards Vocabulary-independent Pre-trained Transformers" solving these limitations to some extent while performing competitively compared to the vanilla model. 
Hope you can get some interesting inspirations from this topic!
Time: Wed 4:00 pm
Venue: SEIEE 3-414
Best Regards,
Angel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://cs.sjtu.edu.cn/pipermail/adapt/attachments/20221025/2d993784/attachment.htm>