pcunwa/BS-Roformer-Inst-EXP-Value-Residual
3
⚠Warning⚠ this is an experimental weight. It may not have practical performance.<br> Also, the model file must be manually rewritten or replaced to use this weight.<br>
The model file is available here.<br> https://github.com/lucidrains/BS-RoFormer
The BS-Roformer has been updated in terms of architecture for the first time in a while.<br> In the 0.5.x update, a mechanism called "Value Residual Learning" was introduced. (https://arxiv.org/abs/2410.17897)<br> The paper argues that this mechanism can reduce the over-focus of attention and further reduce the vanishing gradient problem.