NLP基础（三）：Seq2Seq模型Encoder-Decoder和Attention

1.Encoder-Decoder

基础的seq2seq = Encoder + Decoder + 语义编码c （连接两者的中间状态向量）。Encoder–Decoder 结构，输入是一个序列，输出也是一个序列。Encoder 中将可变长度的信号序列变为固定长度的向量表达，Decoder 将这个固定长度的向量变成可变长度的目标信号序列Encoder，Decoder可以是CNN，RNN，Transformer三种结构，Encoder和Decoder可以是相同的结构，也可以是不同的结构。

class Encoder(tf.keras.layers.Layer):
    def __init__(self, vocab_size, embedding_dim, enc_units, batch_sz, embedding_matrix):
        super(Encoder, self).__init__()
        self.batch_sz = batch_sz
        # self.enc_units = enc_units，使用双向GRU则enc_units除以2
        self.enc_units = enc_units // 2
        self.embedding = tf.keras.layers.Embedding(vocab_size,
                                                   embedding_dim,
                                                   weights=[embedding_matrix],
                                                   trainable=False)
        # self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
        # tf.keras.layers.GRU自动匹配cpu、gpu
        self.gru = tf.keras.layers.GRU(self.enc_units,
                                       return_sequences=True,
                                       return_state=True,
                                       recurrent_initializer='glorot_uniform')

        self.bigru = tf.keras.layers.Bidirectional(self.gru, merge_mode='concat')

    def call(self, x, hidden):
        """
        @desc   batch_size是每次批量训练的样本量，每次训练传入训练样本内所有的词，这里hidden_size=embedding_dim
        @param  x: encoder input word id(batch_size, seq_len)
                hidden: hidden state(batch_size, hidden_size)
        @return output: output vector(batch_size, seq_len, embedding_dim)
                state: hidden state(batch_size, hidden_size)
        """
        x = self.embedding(x)
        hidden = tf.split(hidden, num_or_size_splits=2, axis=1)
        output, forward_state, backward_state = self.bigru(x, initial_state=hidden)
        state = tf.concat([forward_state, backward_state], axis=1)
        # output, state = self.gru(x, initial_state=hidden)
        return output, state

    def initialize_hidden_state(self):
        return tf.zeros((self.batch_sz, 2*self.enc_units))

class Decoder(tf.keras.layers.Layer):
    def __init__(self, vocab_size, embedding_dim, dec_units, batch_sz, embedding_matrix):
        super(Decoder, self).__init__()
        self.batch_sz = batch_sz
        self.dec_units = dec_units
        self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim,
                                                   weights=[embedding_matrix],
                                                   trainable=False)
        # self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
        self.gru = tf.keras.layers.GRU(self.dec_units,
                                       return_sequences=True,
                                       return_state=True,
                                       recurrent_initializer='glorot_uniform')
        # self.dropout = tf.keras.layers.Dropout(0.5)
        self.fc = tf.keras.layers.Dense(vocab_size, activation=tf.keras.activations.softmax)
        # self.fc = tf.keras.layers.Dense(vocab_size)

    def call(self, x, hidden, enc_output, context_vector):
        """
        @desc   按照输出词的数量迭代该函数，每次迭代进行一个词的预测
        @param  x: decoder input word id(batch_size, 1)
                hidden: decoder hidden state(batch_size, hidden_size)
                enc_output: encoder的输出(batch_size, seq_length, embedding_dim)
                context_vector: 注意力机制的输出(batch_size, hidden_size)
        @return x: output vector(batch_size, 1, embedding_dim + hidden_size)
                out: output vector(batch_size, vocab_size)
                state: hidden state(batch_size, hidden_size)
        """
        # enc_output shape == (batch_size, max_length, hidden_size)

        # x shape after passing through embedding == (batch_size, 1, embedding_dim)
        x = self.embedding(x)

        # x shape after concatenation == (batch_size, 1, embedding_dim + hidden_size)
        # 把attention机制得到的context_vector与输入词向量进行拼接
        x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)

        # passing the concatenated vector to the GRU
        output, state = self.gru(x)
        # output shape == (batch_size * 1, hidden_size)
        output = tf.reshape(output, (-1, output.shape[2]))

        # output = self.dropout(output)
        # out shape == (batch_size, vocab_size)
        out = self.fc(output)

        return x, out, state

2.Attention机制

Encoder和Decoder之间的唯一联系就是一个固定长度的语义向量C，即Encoder要将整个序列的信息压缩进一个固定长度的向量中去因为语义向量无法完全表示整个序列的信息，而且之前先输入的内容带有的信息会被后输入的信息稀释掉，输入序列越长，这个现象就越严重。这样Decoder时，没有获得输入序列足够的信息，因此Deocder的准确度自然会降低。因此提出基于Attention的encoder-decoder模型。

令encoder hidden为 $\overline h_{s}$ ，decoder hidden为 $h_{t}$ 。在decoder过程中，t时刻的输出需基于 $h_{t-1}$ 和所有的encoder hidden state计算相似度，得到attention权重 $\alpha_{ts}=softmax(score(\overline h_{s},h_{t-1})$