当前位置：首页 > news >正文

13.编码器的结构

news 2025/4/27 15:04:56

从入门AI到手写Transformer-13.编码器的结构

13.编码器的结构
代码

整理自视频老袁不说话。

13.编码器的结构

$T r an s f or m er E n co d er : 输入 [b, n]$

$E mb e dd in g : - > [b, n, d]$
$P os i t i o na lE n co d er : - > [b, n, d]$
$Dro p o u t : - > [b, n, d]$
$E n co d er Bl oc k : [b, n, d] - > [b, n, d]$ 重复N次
- $M u lt ih e a d A tt e n t i o n : 3 * [b, n, d] - > [b, n, d]$
- $Dro p o u t : [b, n, d] - > [b, n, d]$
- $A dd N or m : 2 * [b, n, d] (Dro u p o u t 输出， M u lt ih e a d A tt e n t i o n 输入) - > [b, n, d]$
- $FFN : [b, n, d] - > [b, n, d]$
- $Dro p o u t : [b, n, d] - > [b, n, d]$
- $A dd N or m : 2 * [b, n, d] (Dro u p o u t 输出， FFN 输入) - > [b, n, d]$
  
  编码器结构
  
  多处执行Dropout

代码

import torch.nn as nnclass Embedding(nn.Module):def __init__(self,*args,**kwargs)->None:super().__init__(*args,**kwargs)def forward(self):print(self.__class__.__name__)
class PositionalEncoding(nn.Module):def __init__(self,*args,**kwargs)->None:super().__init__(*args,**kwargs)def forward(self):print(self.__class__.__name__)
class MultiheadAttention(nn.Module):def __init__(self,*args,**kwargs)->None:super().__init__(*args,**kwargs)def forward(self):print(self.__class__.__name__)
class Dropout(nn.Module):def __init__(self,*args,**kwargs)->None:super().__init__(*args,**kwargs)def forward(self):print(self.__class__.__name__)
class AddNorm(nn.Module):def __init__(self,*args,**kwargs)->None:super().__init__(*args,**kwargs)def forward(self):print(self.__class__.__name__)
class FFN(nn.Module):def __init__(self,*args,**kwargs)->None:super().__init__(*args,**kwargs)def forward(self):print(self.__class__.__name__)class EncoderBlock(nn.Module):def __init__(self,*args, **kwargs)->None:super().__init__(*args,**kwargs)self.mha = MultiheadAttention()self.dropout1=Dropout()self.addnorm1=AddNorm()self.ffn=FFN()self.dropout2=Dropout()self.addnorm2 = AddNorm()def forward(self):self.mha()self.dropout1()self.addnorm1()self.ffn()self.dropout2()self.addnorm2()class TransformerEncoder(nn.Module):def __init__(self,*args,**kwargs)->None:super().__init__(*args,**kwargs)self.embedding=Embedding() # 把序号转变为有语义信息的编码self.posenc=PositionalEncoding()self.dropout=Dropout()self.encblocks=nn.Sequential()for i in range(3):self.encblocks.add_module(str(i),EncoderBlock())def forward(self):self.embedding()self.posenc()self.dropout()for i,blk in enumerate(self.encblocks):print(i)blk()te=TransformerEncoder()
te()

输出结果

Embedding
PositionalEncoding
Dropout
0
MultiheadAttention
Dropout
AddNorm
FFN
Dropout
AddNorm
1
MultiheadAttention
Dropout
AddNorm
FFN
Dropout
AddNorm
2
MultiheadAttention
Dropout
AddNorm
FFN
Dropout
AddNorm

查看全文

http://www.xdnf.cn/news/5725.html