train_llava记录

首先时build_model

【自定义多模态大模型LLaVA——LLaVA系列】 https://www.bilibili.com/video/BV1GS411P74b/?share_source=copy_web&vd_source=820e17ab3d5412c1c94e68f2e5f07ae7

作者用的4B的qwen1.5 但是这里发现3090是只能部署但是推理爆显存所以只能换成0.5B的

问题

首先是BF16的问题

RuntimeError: "triu_tril_cuda_template" not implemented for 'BFloat16'

首先很多人提到了改pytorch和transformer的版本号 Qwen2报错——RuntimeError: “triu_tril_cuda_template“ not implemented for ‘BFloat16‘_qwen2 pytorch2.0-CSDN博客

改了之后这个确实没问题了但是出现了

ValueError: Cannot instantiate this tokenizer from a slow version. If it's based on sentencepiece, make sure you have sentencepiece installed.
pip安装了之后还是发现问题
return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
TypeError: not a string

【大模型】return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg) TypeError: not a string_llamatokenizer typeerror: not a string-CSDN博客

又重新保存了模型

最后是直接改源码

解决RuntimeError: “triu_tril_cuda_template“ not implemented for ‘BFloat16‘-CSDN博客

还有在需要安装peft的时候直接用pip install peft

直接把torch和cuda都更新到最新版本了

下载的时候需要用pip install peft --no-dependencies

how to use peft at torch <=1.12.1 and CUDA <= 10.2 envrionment · Issue #207 · huggingface/peft (github.com)

训练环节

首先是4张卡训练要用78小时

但是在4张卡上运行的时候发信啊这个loss很诡异最开始的时候是5然后很快就变成2变成零点几

然后以为是通信原因放到一张卡上发现要用300多小时

  0%|          | 71/186050 [07:11<318:48:19,  6.17s/it]{'loss': 4.4566, 'grad_norm': 3.2468795776367188, 'learning_rate': 0.00039997850040311743, 'epoch': 0.0}
{'loss': 2.328, 'grad_norm': 2.6973390579223633, 'learning_rate': 0.0003999570008062349, 'epoch': 0.0}
{'loss': 1.444, 'grad_norm': 2.200422763824463, 'learning_rate': 0.0003999355012093523, 'epoch': 0.0}
{'loss': 0.814, 'grad_norm': 1.6881272792816162, 'learning_rate': 0.0003999140016124698, 'epoch': 0.0}
{'loss': 0.4423, 'grad_norm': 1.5033029317855835, 'learning_rate': 0.0003998925020155872, 'epoch': 0.0}
{'loss': 0.263, 'grad_norm': 0.9194843769073486, 'learning_rate': 0.00039987100241870466, 'epoch': 0.0}
{'loss': 0.1796, 'grad_norm': 1.131528615951538, 'learning_rate': 0.0003998495028218221, 'epoch': 0.0}

训练加速技巧