seamless_m4t

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t

MindSpore SeamlessM4T model.

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.HifiGanResidualBlock

Bases: Module

This class represents a High Fidelity Generative Adversarial Network (HifiGan) Residual Block. It is a subclass of nn.Module and is used in the forwardion of the HifiGan model.

ATTRIBUTE	DESCRIPTION
channels	The number of input and output channels for the convolutional layers. TYPE: int
kernel_size	The size of the convolutional kernel. TYPE: int
dilation	The dilation factors to be applied to the convolutional layers. TYPE: tuple
leaky_relu_slope	The slope of the negative region of the leaky ReLU activation function. TYPE: float

METHOD	DESCRIPTION
__init__	Initializes a new instance of the HifiGanResidualBlock class.
get_padding	Calculates the padding to be applied to the convolutional layers.
apply_weight_norm	Applies weight normalization to the convolutional layers.
remove_weight_norm	Removes weight normalization from the convolutional layers.
forward	Constructs the HifiGanResidualBlock by applying the convolutional layers and residual connections to the input hidden states.

Note

The HifiGanResidualBlock class inherits from nn.Module, which is a base class for all neural network modules in MindSpore. It provides basic functionalities for forwarding and managing neural networks.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

class HifiGanResidualBlock(nn.Module):

    """
    This class represents a High Fidelity Generative Adversarial Network (HifiGan) Residual Block.
    It is a subclass of nn.Module and is used in the forwardion of the HifiGan model.

    Attributes:
        channels (int): The number of input and output channels for the convolutional layers.
        kernel_size (int): The size of the convolutional kernel.
        dilation (tuple): The dilation factors to be applied to the convolutional layers.
        leaky_relu_slope (float): The slope of the negative region of the leaky ReLU activation function.

    Methods:
        __init__:
            Initializes a new instance of the HifiGanResidualBlock class.

        get_padding:
            Calculates the padding to be applied to the convolutional layers.

        apply_weight_norm:
            Applies weight normalization to the convolutional layers.

        remove_weight_norm:
            Removes weight normalization from the convolutional layers.

        forward:
            Constructs the HifiGanResidualBlock by applying the convolutional layers and residual connections to
            the input hidden states.

    Note:
        The HifiGanResidualBlock class inherits from nn.Module, which is a base class for all neural network modules
        in MindSpore. It provides basic functionalities for forwarding and managing neural networks.
    """
    def __init__(self, channels, kernel_size=3, dilation=(1, 3, 5), leaky_relu_slope=0.1):
        """
        __init__

        Initializes a new instance of the HifiGanResidualBlock class.

        Args:
            channels (int): The number of input and output channels for the convolutional layers.
            kernel_size (int, optional): The size of the convolutional kernel. Defaults to 3.
            dilation (tuple of int, optional): The dilation rates for the convolutional layers. Defaults to (1, 3, 5).
            leaky_relu_slope (float, optional): The slope for the Leaky ReLU activation function. Defaults to 0.1.

        Returns:
            None.

        Raises:
            ValueError: If channels, kernel_size, or any element in the dilation tuple is less than or equal to 0.
            TypeError: If the provided values for channels, kernel_size, dilation, or leaky_relu_slope are not of
                the expected types.
        """
        super().__init__()
        self.leaky_relu_slope = leaky_relu_slope

        self.convs1 = nn.ModuleList(
            [
                nn.Conv1d(
                    channels,
                    channels,
                    kernel_size,
                    stride=1,
                    dilation=dilation[i],
                    pad_mode='pad',
                    padding=self.get_padding(kernel_size, dilation[i]),
                )
                for i in range(len(dilation))
            ]
        )
        self.convs2 = nn.ModuleList(
            [
                nn.Conv1d(
                    channels,
                    channels,
                    kernel_size,
                    stride=1,
                    dilation=1,
                    pad_mode='pad',
                    padding=self.get_padding(kernel_size, 1),
                )
                for _ in range(len(dilation))
            ]
        )

    def get_padding(self, kernel_size, dilation=1):
        """
        Returns the required padding size for a given kernel size and dilation factor.

        Args:
            self (HifiGanResidualBlock): An instance of the HifiGanResidualBlock class.
            kernel_size (int): The size of the kernel.
            dilation (int, optional): The dilation factor (default is 1).

        Returns:
            int: The calculated padding size.

        Raises:
            None.

        This method calculates the required padding size based on the given kernel size and dilation factor.
        The padding size is determined by the formula: (kernel_size * dilation - dilation) // 2. The method
        then returns the calculated padding size as an integer value.
        """
        return (kernel_size * dilation - dilation) // 2

    def apply_weight_norm(self):
        """
        Apply weight normalization to the convolutional layers in the HifiGanResidualBlock.

        Args:
            self: The instance of the HifiGanResidualBlock class.

        Returns:
            None.

        Raises:
            None.
        """
        for layer in self.convs1:
            nn.utils.weight_norm(layer)
        for layer in self.convs2:
            nn.utils.weight_norm(layer)

    def remove_weight_norm(self):
        """
        Removes weight normalization from the convolutional layers within the HifiGanResidualBlock.

        Args:
            self: An instance of the HifiGanResidualBlock class.

        Returns:
            None.

        Raises:
            None.
        """
        for layer in self.convs1:
            nn.utils.remove_weight_norm(layer)
        for layer in self.convs2:
            nn.utils.remove_weight_norm(layer)

    def forward(self, hidden_states):
        """
        Constructs a single residual block in the HifiGan model.

        Args:
            self (HifiGanResidualBlock): The instance of the HifiGanResidualBlock class.
            hidden_states (Tensor): The input hidden states for the residual block.
                Expected shape is [batch_size, channels, sequence_length].

        Returns:
            None

        Raises:
            None
        """
        for conv1, conv2 in zip(self.convs1, self.convs2):
            residual = hidden_states
            hidden_states = ops.leaky_relu(hidden_states, self.leaky_relu_slope)
            hidden_states = conv1(hidden_states)
            hidden_states = ops.leaky_relu(hidden_states, self.leaky_relu_slope)
            hidden_states = conv2(hidden_states)
            hidden_states = hidden_states + residual
        return hidden_states

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.HifiGanResidualBlock.__init__(channels, kernel_size=3, dilation=(1, 3, 5), leaky_relu_slope=0.1)

init

Initializes a new instance of the HifiGanResidualBlock class.

PARAMETER	DESCRIPTION
channels	The number of input and output channels for the convolutional layers. TYPE: int
kernel_size	The size of the convolutional kernel. Defaults to 3. TYPE: int DEFAULT: 3
dilation	The dilation rates for the convolutional layers. Defaults to (1, 3, 5). TYPE: tuple of int DEFAULT: (1, 3, 5)
leaky_relu_slope	The slope for the Leaky ReLU activation function. Defaults to 0.1. TYPE: float DEFAULT: 0.1

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If channels, kernel_size, or any element in the dilation tuple is less than or equal to 0.
TypeError	If the provided values for channels, kernel_size, dilation, or leaky_relu_slope are not of the expected types.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def __init__(self, channels, kernel_size=3, dilation=(1, 3, 5), leaky_relu_slope=0.1):
    """
    __init__

    Initializes a new instance of the HifiGanResidualBlock class.

    Args:
        channels (int): The number of input and output channels for the convolutional layers.
        kernel_size (int, optional): The size of the convolutional kernel. Defaults to 3.
        dilation (tuple of int, optional): The dilation rates for the convolutional layers. Defaults to (1, 3, 5).
        leaky_relu_slope (float, optional): The slope for the Leaky ReLU activation function. Defaults to 0.1.

    Returns:
        None.

    Raises:
        ValueError: If channels, kernel_size, or any element in the dilation tuple is less than or equal to 0.
        TypeError: If the provided values for channels, kernel_size, dilation, or leaky_relu_slope are not of
            the expected types.
    """
    super().__init__()
    self.leaky_relu_slope = leaky_relu_slope

    self.convs1 = nn.ModuleList(
        [
            nn.Conv1d(
                channels,
                channels,
                kernel_size,
                stride=1,
                dilation=dilation[i],
                pad_mode='pad',
                padding=self.get_padding(kernel_size, dilation[i]),
            )
            for i in range(len(dilation))
        ]
    )
    self.convs2 = nn.ModuleList(
        [
            nn.Conv1d(
                channels,
                channels,
                kernel_size,
                stride=1,
                dilation=1,
                pad_mode='pad',
                padding=self.get_padding(kernel_size, 1),
            )
            for _ in range(len(dilation))
        ]
    )

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.HifiGanResidualBlock.apply_weight_norm()

Apply weight normalization to the convolutional layers in the HifiGanResidualBlock.

PARAMETER	DESCRIPTION
self	The instance of the HifiGanResidualBlock class.

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def apply_weight_norm(self):
    """
    Apply weight normalization to the convolutional layers in the HifiGanResidualBlock.

    Args:
        self: The instance of the HifiGanResidualBlock class.

    Returns:
        None.

    Raises:
        None.
    """
    for layer in self.convs1:
        nn.utils.weight_norm(layer)
    for layer in self.convs2:
        nn.utils.weight_norm(layer)

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.HifiGanResidualBlock.forward(hidden_states)

Constructs a single residual block in the HifiGan model.

PARAMETER	DESCRIPTION
self	The instance of the HifiGanResidualBlock class. TYPE: HifiGanResidualBlock
hidden_states	The input hidden states for the residual block. Expected shape is [batch_size, channels, sequence_length]. TYPE: Tensor

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def forward(self, hidden_states):
    """
    Constructs a single residual block in the HifiGan model.

    Args:
        self (HifiGanResidualBlock): The instance of the HifiGanResidualBlock class.
        hidden_states (Tensor): The input hidden states for the residual block.
            Expected shape is [batch_size, channels, sequence_length].

    Returns:
        None

    Raises:
        None
    """
    for conv1, conv2 in zip(self.convs1, self.convs2):
        residual = hidden_states
        hidden_states = ops.leaky_relu(hidden_states, self.leaky_relu_slope)
        hidden_states = conv1(hidden_states)
        hidden_states = ops.leaky_relu(hidden_states, self.leaky_relu_slope)
        hidden_states = conv2(hidden_states)
        hidden_states = hidden_states + residual
    return hidden_states

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.HifiGanResidualBlock.get_padding(kernel_size, dilation=1)

Returns the required padding size for a given kernel size and dilation factor.

PARAMETER	DESCRIPTION
self	An instance of the HifiGanResidualBlock class. TYPE: HifiGanResidualBlock
kernel_size	The size of the kernel. TYPE: int
dilation	The dilation factor (default is 1). TYPE: int DEFAULT: 1

RETURNS	DESCRIPTION
int	The calculated padding size.

This method calculates the required padding size based on the given kernel size and dilation factor. The padding size is determined by the formula: (kernel_size * dilation - dilation) // 2. The method then returns the calculated padding size as an integer value.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def get_padding(self, kernel_size, dilation=1):
    """
    Returns the required padding size for a given kernel size and dilation factor.

    Args:
        self (HifiGanResidualBlock): An instance of the HifiGanResidualBlock class.
        kernel_size (int): The size of the kernel.
        dilation (int, optional): The dilation factor (default is 1).

    Returns:
        int: The calculated padding size.

    Raises:
        None.

    This method calculates the required padding size based on the given kernel size and dilation factor.
    The padding size is determined by the formula: (kernel_size * dilation - dilation) // 2. The method
    then returns the calculated padding size as an integer value.
    """
    return (kernel_size * dilation - dilation) // 2

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.HifiGanResidualBlock.remove_weight_norm()

Removes weight normalization from the convolutional layers within the HifiGanResidualBlock.

PARAMETER	DESCRIPTION
self	An instance of the HifiGanResidualBlock class.

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def remove_weight_norm(self):
    """
    Removes weight normalization from the convolutional layers within the HifiGanResidualBlock.

    Args:
        self: An instance of the HifiGanResidualBlock class.

    Returns:
        None.

    Raises:
        None.
    """
    for layer in self.convs1:
        nn.utils.remove_weight_norm(layer)
    for layer in self.convs2:
        nn.utils.remove_weight_norm(layer)

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TAttention

Bases: Module

Multi-headed attention from 'Attention Is All You Need' paper

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

class SeamlessM4TAttention(nn.Module):
    """Multi-headed attention from 'Attention Is All You Need' paper"""
    # Copied from transformers.models.bart.modeling_bart.BartAttention.__init__ with Bart->SeamlessM4T
    def __init__(
        self,
        embed_dim: int,
        num_heads: int,
        dropout: float = 0.0,
        is_decoder: bool = False,
        bias: bool = True,
        is_causal: bool = False,
        config: Optional[SeamlessM4TConfig] = None,
    ):
        """
        Initialize the SeamlessM4TAttention class.

        Args:
            embed_dim (int): The dimension of the input embeddings.
            num_heads (int): The number of attention heads.
            dropout (float, optional): The dropout probability. Defaults to 0.0.
            is_decoder (bool, optional): Flag indicating if the attention is used in a decoder context.
                Defaults to False.
            bias (bool): Flag indicating whether to include bias in linear transformations.
            is_causal (bool): Flag indicating if the attention is causal.
            config (Optional[SeamlessM4TConfig]): An optional configuration object for the attention mechanism.

        Returns:
            None.

        Raises:
            ValueError: If embed_dim is not divisible by num_heads.
        """
        super().__init__()
        self.embed_dim = embed_dim
        self.num_heads = num_heads
        self.dropout = dropout
        self.head_dim = embed_dim // num_heads
        self.config = config

        if (self.head_dim * num_heads) != self.embed_dim:
            raise ValueError(
                f"embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim}"
                f" and `num_heads`: {num_heads})."
            )
        self.scaling = self.head_dim**-0.5
        self.is_decoder = is_decoder
        self.is_causal = is_causal

        self.k_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.v_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.q_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.out_proj = nn.Linear(embed_dim, embed_dim, bias=bias)

    def _shape(self, tensor: mindspore.Tensor, seq_len: int, bsz: int):
        """
        This method '_shape' is defined within the class 'SeamlessM4TAttention' and is used to reshape the input tensor
        based on the provided sequence length and batch size.

        Args:
            self: An instance of the 'SeamlessM4TAttention' class.
            tensor (mindspore.Tensor): The input tensor to be reshaped.
            seq_len (int): The length of the sequence.
            bsz (int): The batch size.

        Returns:
            None: This method does not return any value. It modifies the input tensor in place to reshape it as per the
                specified sequence length and batch size.

        Raises:
            None.
        """
        return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).swapaxes(1, 2)

    def forward(
        self,
        hidden_states: mindspore.Tensor,
        encoder_hidden_states: Optional[mindspore.Tensor] = None,
        past_key_value: Optional[Tuple[mindspore.Tensor]] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        output_attentions: bool = False,
    ) -> Tuple[mindspore.Tensor, Optional[mindspore.Tensor], Optional[Tuple[mindspore.Tensor]]]:
        """Input shape: Batch x Time x Channel"""
        # if encoder_hidden_states are provided this layer is used as a cross-attention layer
        # for the decoder
        is_cross_attention = encoder_hidden_states is not None

        bsz, tgt_len, _ = hidden_states.shape

        # get query proj
        query_states = self.q_proj(hidden_states) * self.scaling
        # get key, value proj
        # `past_key_value[0].shape[2] == encoder_hidden_states.shape[1]`
        # is checking that the `sequence_length` of the `past_key_value` is the same as
        # the provided `encoder_hidden_states` to support prefix tuning
        if (
            is_cross_attention
            and past_key_value is not None
            and past_key_value[0].shape[2] == encoder_hidden_states.shape[1]
        ):
            # reuse k,v, cross_attentions
            key_states = past_key_value[0]
            value_states = past_key_value[1]
        elif is_cross_attention:
            # cross_attentions
            key_states = self._shape(self.k_proj(encoder_hidden_states), -1, bsz)
            value_states = self._shape(self.v_proj(encoder_hidden_states), -1, bsz)
        elif past_key_value is not None:
            # reuse k, v, self_attention
            key_states = self._shape(self.k_proj(hidden_states), -1, bsz)
            value_states = self._shape(self.v_proj(hidden_states), -1, bsz)
            key_states = ops.cat([past_key_value[0], key_states], axis=2)
            value_states = ops.cat([past_key_value[1], value_states], axis=2)
        else:
            # self_attention
            key_states = self._shape(self.k_proj(hidden_states), -1, bsz)
            value_states = self._shape(self.v_proj(hidden_states), -1, bsz)

        if self.is_decoder:
            # if cross_attention save Tuple(mindspore.Tensor, mindspore.Tensor) of all cross attention key/value_states.
            # Further calls to cross_attention layer can then reuse all cross-attention
            # key/value_states (first "if" case)
            # if uni-directional self-attention (decoder) save Tuple(mindspore.Tensor, mindspore.Tensor) of
            # all previous decoder key/value_states. Further calls to uni-directional self-attention
            # can concat previous decoder key/value_states to current projected key/value_states (third "elif" case)
            # if encoder bi-directional self-attention `past_key_value` is always `None`
            past_key_value = (key_states, value_states)

        proj_shape = (bsz * self.num_heads, -1, self.head_dim)
        query_states = self._shape(query_states, tgt_len, bsz).view(*proj_shape)
        key_states = key_states.reshape(*proj_shape)
        value_states = value_states.reshape(*proj_shape)

        src_len = key_states.shape[1]
        attn_weights = ops.bmm(query_states, key_states.swapaxes(1, 2))

        if attn_weights.shape != (bsz * self.num_heads, tgt_len, src_len):
            raise ValueError(
                f"Attention weights should be of size {(bsz * self.num_heads, tgt_len, src_len)}, but is"
                f" {attn_weights.shape}"
            )

        if attention_mask is not None:
            if attention_mask.shape != (bsz, 1, tgt_len, src_len):
                raise ValueError(
                    f"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is {attention_mask.shape}"
                )
            attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attention_mask
            attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)

        attn_weights = ops.softmax(attn_weights, axis=-1)

        if output_attentions:
            # this operation is a bit awkward, but it's required to
            # make sure that attn_weights keeps its gradient.
            # In order to do so, attn_weights have to be reshaped
            # twice and have to be reused in the following
            attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
            attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len)
        else:
            attn_weights_reshaped = None

        attn_probs = ops.dropout(attn_weights, p=self.dropout, training=self.training)

        attn_output = ops.bmm(attn_probs, value_states)

        if attn_output.shape != (bsz * self.num_heads, tgt_len, self.head_dim):
            raise ValueError(
                f"`attn_output` should be of size {(bsz * self.num_heads, tgt_len, self.head_dim)}, but is"
                f" {attn_output.shape}"
            )

        attn_output = attn_output.view(bsz, self.num_heads, tgt_len, self.head_dim)
        attn_output = attn_output.swapaxes(1, 2)

        # Use the `embed_dim` from the config (stored in the class) rather than `hidden_state` because `attn_output` can be
        # partitioned across GPUs when using tensor-parallelism.
        attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim)

        attn_output = self.out_proj(attn_output)

        return attn_output, attn_weights_reshaped, past_key_value

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TAttention.__init__(embed_dim, num_heads, dropout=0.0, is_decoder=False, bias=True, is_causal=False, config=None)

Initialize the SeamlessM4TAttention class.

PARAMETER	DESCRIPTION
embed_dim	The dimension of the input embeddings. TYPE: int
num_heads	The number of attention heads. TYPE: int
dropout	The dropout probability. Defaults to 0.0. TYPE: float DEFAULT: 0.0
is_decoder	Flag indicating if the attention is used in a decoder context. Defaults to False. TYPE: bool DEFAULT: False
bias	Flag indicating whether to include bias in linear transformations. TYPE: bool DEFAULT: True
is_causal	Flag indicating if the attention is causal. TYPE: bool DEFAULT: False
config	An optional configuration object for the attention mechanism. TYPE: Optional[SeamlessM4TConfig] DEFAULT: None

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If embed_dim is not divisible by num_heads.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def __init__(
    self,
    embed_dim: int,
    num_heads: int,
    dropout: float = 0.0,
    is_decoder: bool = False,
    bias: bool = True,
    is_causal: bool = False,
    config: Optional[SeamlessM4TConfig] = None,
):
    """
    Initialize the SeamlessM4TAttention class.

    Args:
        embed_dim (int): The dimension of the input embeddings.
        num_heads (int): The number of attention heads.
        dropout (float, optional): The dropout probability. Defaults to 0.0.
        is_decoder (bool, optional): Flag indicating if the attention is used in a decoder context.
            Defaults to False.
        bias (bool): Flag indicating whether to include bias in linear transformations.
        is_causal (bool): Flag indicating if the attention is causal.
        config (Optional[SeamlessM4TConfig]): An optional configuration object for the attention mechanism.

    Returns:
        None.

    Raises:
        ValueError: If embed_dim is not divisible by num_heads.
    """
    super().__init__()
    self.embed_dim = embed_dim
    self.num_heads = num_heads
    self.dropout = dropout
    self.head_dim = embed_dim // num_heads
    self.config = config

    if (self.head_dim * num_heads) != self.embed_dim:
        raise ValueError(
            f"embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim}"
            f" and `num_heads`: {num_heads})."
        )
    self.scaling = self.head_dim**-0.5
    self.is_decoder = is_decoder
    self.is_causal = is_causal

    self.k_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
    self.v_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
    self.q_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
    self.out_proj = nn.Linear(embed_dim, embed_dim, bias=bias)

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TAttention.forward(hidden_states, encoder_hidden_states=None, past_key_value=None, attention_mask=None, output_attentions=False)

Input shape: Batch x Time x Channel

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def forward(
    self,
    hidden_states: mindspore.Tensor,
    encoder_hidden_states: Optional[mindspore.Tensor] = None,
    past_key_value: Optional[Tuple[mindspore.Tensor]] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    output_attentions: bool = False,
) -> Tuple[mindspore.Tensor, Optional[mindspore.Tensor], Optional[Tuple[mindspore.Tensor]]]:
    """Input shape: Batch x Time x Channel"""
    # if encoder_hidden_states are provided this layer is used as a cross-attention layer
    # for the decoder
    is_cross_attention = encoder_hidden_states is not None

    bsz, tgt_len, _ = hidden_states.shape

    # get query proj
    query_states = self.q_proj(hidden_states) * self.scaling
    # get key, value proj
    # `past_key_value[0].shape[2] == encoder_hidden_states.shape[1]`
    # is checking that the `sequence_length` of the `past_key_value` is the same as
    # the provided `encoder_hidden_states` to support prefix tuning
    if (
        is_cross_attention
        and past_key_value is not None
        and past_key_value[0].shape[2] == encoder_hidden_states.shape[1]
    ):
        # reuse k,v, cross_attentions
        key_states = past_key_value[0]
        value_states = past_key_value[1]
    elif is_cross_attention:
        # cross_attentions
        key_states = self._shape(self.k_proj(encoder_hidden_states), -1, bsz)
        value_states = self._shape(self.v_proj(encoder_hidden_states), -1, bsz)
    elif past_key_value is not None:
        # reuse k, v, self_attention
        key_states = self._shape(self.k_proj(hidden_states), -1, bsz)
        value_states = self._shape(self.v_proj(hidden_states), -1, bsz)
        key_states = ops.cat([past_key_value[0], key_states], axis=2)
        value_states = ops.cat([past_key_value[1], value_states], axis=2)
    else:
        # self_attention
        key_states = self._shape(self.k_proj(hidden_states), -1, bsz)
        value_states = self._shape(self.v_proj(hidden_states), -1, bsz)

    if self.is_decoder:
        # if cross_attention save Tuple(mindspore.Tensor, mindspore.Tensor) of all cross attention key/value_states.
        # Further calls to cross_attention layer can then reuse all cross-attention
        # key/value_states (first "if" case)
        # if uni-directional self-attention (decoder) save Tuple(mindspore.Tensor, mindspore.Tensor) of
        # all previous decoder key/value_states. Further calls to uni-directional self-attention
        # can concat previous decoder key/value_states to current projected key/value_states (third "elif" case)
        # if encoder bi-directional self-attention `past_key_value` is always `None`
        past_key_value = (key_states, value_states)

    proj_shape = (bsz * self.num_heads, -1, self.head_dim)
    query_states = self._shape(query_states, tgt_len, bsz).view(*proj_shape)
    key_states = key_states.reshape(*proj_shape)
    value_states = value_states.reshape(*proj_shape)

    src_len = key_states.shape[1]
    attn_weights = ops.bmm(query_states, key_states.swapaxes(1, 2))

    if attn_weights.shape != (bsz * self.num_heads, tgt_len, src_len):
        raise ValueError(
            f"Attention weights should be of size {(bsz * self.num_heads, tgt_len, src_len)}, but is"
            f" {attn_weights.shape}"
        )

    if attention_mask is not None:
        if attention_mask.shape != (bsz, 1, tgt_len, src_len):
            raise ValueError(
                f"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is {attention_mask.shape}"
            )
        attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attention_mask
        attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)

    attn_weights = ops.softmax(attn_weights, axis=-1)

    if output_attentions:
        # this operation is a bit awkward, but it's required to
        # make sure that attn_weights keeps its gradient.
        # In order to do so, attn_weights have to be reshaped
        # twice and have to be reused in the following
        attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
        attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len)
    else:
        attn_weights_reshaped = None

    attn_probs = ops.dropout(attn_weights, p=self.dropout, training=self.training)

    attn_output = ops.bmm(attn_probs, value_states)

    if attn_output.shape != (bsz * self.num_heads, tgt_len, self.head_dim):
        raise ValueError(
            f"`attn_output` should be of size {(bsz * self.num_heads, tgt_len, self.head_dim)}, but is"
            f" {attn_output.shape}"
        )

    attn_output = attn_output.view(bsz, self.num_heads, tgt_len, self.head_dim)
    attn_output = attn_output.swapaxes(1, 2)

    # Use the `embed_dim` from the config (stored in the class) rather than `hidden_state` because `attn_output` can be
    # partitioned across GPUs when using tensor-parallelism.
    attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim)

    attn_output = self.out_proj(attn_output)

    return attn_output, attn_weights_reshaped, past_key_value

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TCodeHifiGan

Bases: PreTrainedModel

This class represents a high fidelity generative adversarial network (HiFi-GAN) model for seamless text-to-speech synthesis in the SeamlessM4T framework. The model includes components for duration prediction, unit embeddings, speaker embeddings, language embeddings, and the HiFi-GAN architecture.

The class includes methods for computing output lengths after the duration layer and the HiFi-GAN convolutional layers. It also provides functionality for forwarding the model using input sequences, speaker IDs, and language IDs, and initializing and applying weight normalization to the model's components.

The class inherits from PreTrainedModel and contains methods for weight initialization, applying weight normalization, and removing weight normalization from the HiFi-GAN components. Additionally, it includes utility functions for weight normalization operations.

For detailed information on each method and its parameters, please refer to the method docstrings within the class definition.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

class SeamlessM4TCodeHifiGan(PreTrainedModel):

    """
    This class represents a high fidelity generative adversarial network (HiFi-GAN) model for seamless text-to-speech
    synthesis in the SeamlessM4T framework. The model includes components for duration prediction, unit embeddings,
    speaker embeddings, language embeddings, and the HiFi-GAN architecture.

    The class includes methods for computing output lengths after the duration layer and the HiFi-GAN convolutional
    layers. It also provides functionality for forwarding the model using input sequences, speaker IDs, and language
    IDs, and initializing and applying weight normalization to the model's components.

    The class inherits from PreTrainedModel and contains methods for weight initialization, applying weight
    normalization, and removing weight normalization from the HiFi-GAN components. Additionally, it includes utility
    functions for weight normalization operations.

    For detailed information on each method and its parameters, please refer to the method docstrings within the
    class definition.
    """
    config_class = SeamlessM4TConfig
    main_input_name = "input_embeds"
    _no_split_modules = []

    def __init__(self, config):
        """
        Initializes the SeamlessM4TCodeHifiGan class.

        Args:
            self: The instance of the class.
            config: A configuration object that contains various settings and parameters for the HifiGan model.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(config)

        self.pad_token_id = config.t2u_pad_token_id
        self.dur_predictor = SeamlessM4TVariancePredictor(config)

        self.unit_embedding = nn.Embedding(config.unit_hifi_gan_vocab_size, config.unit_embed_dim)
        self.speaker_embedding = nn.Embedding(config.vocoder_num_spkrs, config.spkr_embed_dim)
        self.language_embedding = nn.Embedding(config.vocoder_num_langs, config.lang_embed_dim)

        self.hifi_gan = SeamlessM4THifiGan(config)

        # Initialize weights and apply final processing
        self.post_init()

    def _get_dur_output_lengths(self, input_ids, dur_out):
        """
        Computes the output length after the duration layer.
        """
        unit_lengths = (input_ids != self.pad_token_id).sum(1)

        # take care of edge cases where no padding or too many padding
        unit_lengths = ops.clamp(unit_lengths, 0, dur_out.shape[1] - 1)

        cumulative_dur_out = ops.cumsum(dur_out, axis=1)
        unit_lengths = cumulative_dur_out.gather_elements(dim=1, index=unit_lengths.unsqueeze(1)).squeeze()

        return unit_lengths

    def _get_output_hifigan_lengths(self, input_lengths: Union[mindspore.Tensor, int]):
        """
        Computes the output length of the hifigan convolutional layers
        """
        def _conv_out_length(input_length, kernel_size, stride, pad, dilation=1):
            # 1D convolutional layer output length formula taken
            # from https://pyops.org/docs/stable/generated/ops.nn.Conv1d.html
            return (
                ops.div(input_length + 2 * pad - dilation * (kernel_size - 1) - 1, stride, rounding_mode="floor") + 1
            )

        def _swapaxes_conv_out_length(input_length, kernel_size, stride, pad, dilation=1):
            return (input_length - 1) * stride - 2 * pad + dilation * (kernel_size - 1) + 1

        # conv_pre
        input_lengths = _conv_out_length(input_lengths, 7, 1, 3)

        # upsampler
        for _, (upsample_rate, kernel_size) in enumerate(
            zip(self.config.upsample_rates, self.config.upsample_kernel_sizes)
        ):
            input_lengths = _swapaxes_conv_out_length(
                input_lengths, kernel_size, upsample_rate, (kernel_size - upsample_rate) // 2
            )

        # resblock
        for _ in range(len(self.config.upsample_rates)):
            for kernel_size, dilation in zip(self.config.resblock_kernel_sizes, self.config.resblock_dilation_sizes):
                for dil in dilation:
                    input_lengths = _conv_out_length(
                        input_lengths, kernel_size, 1, (kernel_size - 1) * dil // 2, dilation=dil
                    )

                for dil in dilation:
                    input_lengths = _conv_out_length(input_lengths, kernel_size, 1, (kernel_size - 1) // 2, dilation=1)

        # conv_post
        input_lengths = _conv_out_length(input_lengths, 7, 1, 3)

        return input_lengths

    def forward(
        self, input_ids: mindspore.Tensor, spkr_id: mindspore.Tensor, lang_id: mindspore.Tensor
    ) -> Tuple[mindspore.Tensor]:
        """
        Args:
            input_ids (`mindspore.Tensor` of shape `(batch_size, sequence_length)`):
                Indices of input sequence tokens in the vocabulary.

                Indices can be obtained using [`SeamlessM4TTextToUnitForConditionalGeneration`]. [What are input
                IDs?](../glossary#input-ids)
            spkr_id (`int`, *optional*):
                The id of the speaker used for speech synthesis. Must be lower than `config.vocoder_num_spkrs`.
            tgt_lang (`str`, *optional*):
                The language id to use as target language for translation.
        """
        hidden_states = self.unit_embedding(input_ids).swapaxes(1, 2)
        spkr = self.speaker_embedding(spkr_id).swapaxes(1, 2)
        lang = self.language_embedding(lang_id).swapaxes(1, 2)

        log_dur_pred = self.dur_predictor(hidden_states.swapaxes(1, 2))
        dur_out = ops.clamp(ops.round((ops.exp(log_dur_pred) - 1)).long(), min=1)
        # B x C x T
        if hidden_states.shape[0] == 1:
            hidden_states = ops.repeat_interleave(hidden_states, dur_out.view(-1), axis=2)
        else:
            # if batched sample, need to interleave per sample, and pad -> loss of parallelism
            if hidden_states.shape[0] > 1 and self.training:
                logger.warning(
                    """`self.training=True` and you use batching. You lose parallelism during the hifigan
                               forward pass because the samples are interleaved."""
                )
            hidden_states = [
                ops.repeat_interleave(hidden_state, duration, axis=-1).swapaxes(0, 1)
                for (hidden_state, duration) in zip(hidden_states, dur_out)
            ]

            # hidden_states = nn.utils.rnn.pad_sequence(hidden_states, batch_first=True).swapaxes(1, 2)
            hidden_states = ops.stack(hidden_states).swapaxes(1, 2)

        spkr = spkr.repeat(1, 1, hidden_states.shape[-1])
        lang = lang.repeat(1, 1, hidden_states.shape[-1])
        hidden_states = ops.cat([lang, hidden_states, spkr], axis=1)

        hidden_states = self.hifi_gan(hidden_states)

        unit_lengths = self._get_dur_output_lengths(input_ids, dur_out)
        lengths = self._get_output_hifigan_lengths(unit_lengths)

        return hidden_states, lengths

    def _init_weights(self, cell):
        """Initialize the weights."""
        if isinstance(cell, (nn.Linear, nn.Conv1d, nn.Conv1dTranspose)):
            cell.weight.set_data(initializer(Normal(self.config.initializer_range),
                                                    cell.weight.shape, cell.weight.dtype))
            if cell.bias is not None:
                cell.bias.set_data(initializer('zeros', cell.bias.shape, cell.bias.dtype))
        elif isinstance(cell, nn.Embedding):
            weight = initializer(Normal(self.config.initializer_range),
                                                 cell.weight.shape,
                                                 cell.weight.dtype)
            if cell.padding_idx is not None:
                weight[cell.padding_idx] = 0
            cell.weight.set_data(weight)

    def apply_weight_norm(self):
        """
        Applies weight normalization to the layers of the SeamlessM4TCodeHifiGan model.

        Args:
            self: An instance of the SeamlessM4TCodeHifiGan class.

        Returns:
            None: This method modifies the model's layers in-place.

        Raises:
            None.

        This method applies weight normalization to the layers of the HifiGan model within the SeamlessM4TCodeHifiGan
        class. It iterates through each layer and applies weight normalization using the nn.utils.weight_norm() function.

        The layers that are subjected to weight normalization are:

        - self.hifi_gan.conv_pre: Convolutional layer before upsampling.
        - self.hifi_gan.upsampler: List of upsampling layers.
        - self.hifi_gan.resblocks: List of residual blocks.
        - self.hifi_gan.conv_post: Convolutional layer after upsampling.

        The weight normalization technique normalizes the weights of each layer, making the training process more
        stable and accelerating the convergence. It helps to reduce the internal covariate shift and improves the
        generalization performance of the model.

        Note:
            The method modifies the original model's layers and does not return any value.
        """
        nn.utils.weight_norm(self.hifi_gan.conv_pre)
        for layer in self.hifi_gan.upsampler:
            nn.utils.weight_norm(layer)
        for layer in self.hifi_gan.resblocks:
            layer.apply_weight_norm()
        nn.utils.weight_norm(self.hifi_gan.conv_post)

    def remove_weight_norm(self):
        """
        Removes weight normalization from the specified layers in the SeamlessM4TCodeHifiGan class.

        Args:
            self: An instance of the SeamlessM4TCodeHifiGan class.

        Returns:
            None.

        Raises:
            None.

        Description:
            This method removes weight normalization from the layers in the HifiGan model.
            The following layers are affected:

            - self.hifi_gan.conv_pre: This is the convolutional layer before the upsampling layers.
            - self.hifi_gan.upsampler: These are the upsampling layers in the HifiGan model.
            - self.hifi_gan.resblocks: These are the residual blocks in the HifiGan model.
            - self.hifi_gan.conv_post: This is the convolutional layer after the upsampling layers.

        Note:
            Weight normalization is a technique used in deep learning to normalize the weights of a neural network layer.
            Removing weight normalization can improve the performance or stability of the model in certain scenarios.
        """
        nn.utils.remove_weight_norm(self.hifi_gan.conv_pre)
        for layer in self.hifi_gan.upsampler:
            nn.utils.remove_weight_norm(layer)
        for layer in self.hifi_gan.resblocks:
            layer.remove_weight_norm()
        nn.utils.remove_weight_norm(self.hifi_gan.conv_post)

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TCodeHifiGan.__init__(config)

Initializes the SeamlessM4TCodeHifiGan class.

PARAMETER	DESCRIPTION
self	The instance of the class.
config	A configuration object that contains various settings and parameters for the HifiGan model.

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def __init__(self, config):
    """
    Initializes the SeamlessM4TCodeHifiGan class.

    Args:
        self: The instance of the class.
        config: A configuration object that contains various settings and parameters for the HifiGan model.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(config)

    self.pad_token_id = config.t2u_pad_token_id
    self.dur_predictor = SeamlessM4TVariancePredictor(config)

    self.unit_embedding = nn.Embedding(config.unit_hifi_gan_vocab_size, config.unit_embed_dim)
    self.speaker_embedding = nn.Embedding(config.vocoder_num_spkrs, config.spkr_embed_dim)
    self.language_embedding = nn.Embedding(config.vocoder_num_langs, config.lang_embed_dim)

    self.hifi_gan = SeamlessM4THifiGan(config)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TCodeHifiGan.apply_weight_norm()

Applies weight normalization to the layers of the SeamlessM4TCodeHifiGan model.

PARAMETER	DESCRIPTION
self	An instance of the SeamlessM4TCodeHifiGan class.

RETURNS	DESCRIPTION
None	This method modifies the model's layers in-place.

This method applies weight normalization to the layers of the HifiGan model within the SeamlessM4TCodeHifiGan class. It iterates through each layer and applies weight normalization using the nn.utils.weight_norm() function.

The layers that are subjected to weight normalization are:

• self.hifi_gan.conv_pre: Convolutional layer before upsampling.
• self.hifi_gan.upsampler: List of upsampling layers.
• self.hifi_gan.resblocks: List of residual blocks.
• self.hifi_gan.conv_post: Convolutional layer after upsampling.

The weight normalization technique normalizes the weights of each layer, making the training process more stable and accelerating the convergence. It helps to reduce the internal covariate shift and improves the generalization performance of the model.

Note

The method modifies the original model's layers and does not return any value.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def apply_weight_norm(self):
    """
    Applies weight normalization to the layers of the SeamlessM4TCodeHifiGan model.

    Args:
        self: An instance of the SeamlessM4TCodeHifiGan class.

    Returns:
        None: This method modifies the model's layers in-place.

    Raises:
        None.

    This method applies weight normalization to the layers of the HifiGan model within the SeamlessM4TCodeHifiGan
    class. It iterates through each layer and applies weight normalization using the nn.utils.weight_norm() function.

    The layers that are subjected to weight normalization are:

    - self.hifi_gan.conv_pre: Convolutional layer before upsampling.
    - self.hifi_gan.upsampler: List of upsampling layers.
    - self.hifi_gan.resblocks: List of residual blocks.
    - self.hifi_gan.conv_post: Convolutional layer after upsampling.

    The weight normalization technique normalizes the weights of each layer, making the training process more
    stable and accelerating the convergence. It helps to reduce the internal covariate shift and improves the
    generalization performance of the model.

    Note:
        The method modifies the original model's layers and does not return any value.
    """
    nn.utils.weight_norm(self.hifi_gan.conv_pre)
    for layer in self.hifi_gan.upsampler:
        nn.utils.weight_norm(layer)
    for layer in self.hifi_gan.resblocks:
        layer.apply_weight_norm()
    nn.utils.weight_norm(self.hifi_gan.conv_post)

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TCodeHifiGan.forward(input_ids, spkr_id, lang_id)

PARAMETER	DESCRIPTION
input_ids	Indices of input sequence tokens in the vocabulary. Indices can be obtained using [SeamlessM4TTextToUnitForConditionalGeneration]. What are input IDs? TYPE: `mindspore.Tensor` of shape `(batch_size, sequence_length)`
spkr_id	The id of the speaker used for speech synthesis. Must be lower than config.vocoder_num_spkrs. TYPE: `int`, *optional*
tgt_lang	The language id to use as target language for translation. TYPE: `str`, *optional*

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def forward(
    self, input_ids: mindspore.Tensor, spkr_id: mindspore.Tensor, lang_id: mindspore.Tensor
) -> Tuple[mindspore.Tensor]:
    """
    Args:
        input_ids (`mindspore.Tensor` of shape `(batch_size, sequence_length)`):
            Indices of input sequence tokens in the vocabulary.

            Indices can be obtained using [`SeamlessM4TTextToUnitForConditionalGeneration`]. [What are input
            IDs?](../glossary#input-ids)
        spkr_id (`int`, *optional*):
            The id of the speaker used for speech synthesis. Must be lower than `config.vocoder_num_spkrs`.
        tgt_lang (`str`, *optional*):
            The language id to use as target language for translation.
    """
    hidden_states = self.unit_embedding(input_ids).swapaxes(1, 2)
    spkr = self.speaker_embedding(spkr_id).swapaxes(1, 2)
    lang = self.language_embedding(lang_id).swapaxes(1, 2)

    log_dur_pred = self.dur_predictor(hidden_states.swapaxes(1, 2))
    dur_out = ops.clamp(ops.round((ops.exp(log_dur_pred) - 1)).long(), min=1)
    # B x C x T
    if hidden_states.shape[0] == 1:
        hidden_states = ops.repeat_interleave(hidden_states, dur_out.view(-1), axis=2)
    else:
        # if batched sample, need to interleave per sample, and pad -> loss of parallelism
        if hidden_states.shape[0] > 1 and self.training:
            logger.warning(
                """`self.training=True` and you use batching. You lose parallelism during the hifigan
                           forward pass because the samples are interleaved."""
            )
        hidden_states = [
            ops.repeat_interleave(hidden_state, duration, axis=-1).swapaxes(0, 1)
            for (hidden_state, duration) in zip(hidden_states, dur_out)
        ]

        # hidden_states = nn.utils.rnn.pad_sequence(hidden_states, batch_first=True).swapaxes(1, 2)
        hidden_states = ops.stack(hidden_states).swapaxes(1, 2)

    spkr = spkr.repeat(1, 1, hidden_states.shape[-1])
    lang = lang.repeat(1, 1, hidden_states.shape[-1])
    hidden_states = ops.cat([lang, hidden_states, spkr], axis=1)

    hidden_states = self.hifi_gan(hidden_states)

    unit_lengths = self._get_dur_output_lengths(input_ids, dur_out)
    lengths = self._get_output_hifigan_lengths(unit_lengths)

    return hidden_states, lengths

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TCodeHifiGan.remove_weight_norm()

Removes weight normalization from the specified layers in the SeamlessM4TCodeHifiGan class.

PARAMETER	DESCRIPTION
self	An instance of the SeamlessM4TCodeHifiGan class.

RETURNS	DESCRIPTION
	None.

Description

This method removes weight normalization from the layers in the HifiGan model. The following layers are affected:

• self.hifi_gan.conv_pre: This is the convolutional layer before the upsampling layers.
• self.hifi_gan.upsampler: These are the upsampling layers in the HifiGan model.
• self.hifi_gan.resblocks: These are the residual blocks in the HifiGan model.
• self.hifi_gan.conv_post: This is the convolutional layer after the upsampling layers.

Note

Weight normalization is a technique used in deep learning to normalize the weights of a neural network layer. Removing weight normalization can improve the performance or stability of the model in certain scenarios.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def remove_weight_norm(self):
    """
    Removes weight normalization from the specified layers in the SeamlessM4TCodeHifiGan class.

    Args:
        self: An instance of the SeamlessM4TCodeHifiGan class.

    Returns:
        None.

    Raises:
        None.

    Description:
        This method removes weight normalization from the layers in the HifiGan model.
        The following layers are affected:

        - self.hifi_gan.conv_pre: This is the convolutional layer before the upsampling layers.
        - self.hifi_gan.upsampler: These are the upsampling layers in the HifiGan model.
        - self.hifi_gan.resblocks: These are the residual blocks in the HifiGan model.
        - self.hifi_gan.conv_post: This is the convolutional layer after the upsampling layers.

    Note:
        Weight normalization is a technique used in deep learning to normalize the weights of a neural network layer.
        Removing weight normalization can improve the performance or stability of the model in certain scenarios.
    """
    nn.utils.remove_weight_norm(self.hifi_gan.conv_pre)
    for layer in self.hifi_gan.upsampler:
        nn.utils.remove_weight_norm(layer)
    for layer in self.hifi_gan.resblocks:
        layer.remove_weight_norm()
    nn.utils.remove_weight_norm(self.hifi_gan.conv_post)

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerAdapter

Bases: Module

This class represents a seamless multi-task (M4T) Conformer adapter, designed for adapting transformer-based models for multi-task learning. The adapter consists of multiple adapter layers that can be stacked on top of each other to adapt the model's hidden states for different tasks.

ATTRIBUTE	DESCRIPTION
layers	A list of SeamlessM4TConformerAdapterLayer instances, each representing an adapter layer in the adapter stack. TYPE: ModuleList

METHOD	DESCRIPTION
__init__	Initializes the SeamlessM4TConformerAdapter instance with the specified configuration. Args: config (dict): A dictionary containing configuration parameters for the adapter.
forward	Constructs the adapter by applying each adapter layer in the stack to the input hidden states. Args: hidden_states (Tensor): The input hidden states to be adapted by the adapter. attention_mask (Tensor): The attention mask to be applied during adaptation. Returns: Tensor: The adapted hidden states after passing through all adapter layers.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

class SeamlessM4TConformerAdapter(nn.Module):

    """
    This class represents a seamless multi-task (M4T) Conformer adapter, designed for adapting transformer-based models
    for multi-task learning. The adapter consists of multiple adapter layers that can be stacked on top of each other
    to adapt the model's hidden states for different tasks.

    Attributes:
        layers (nn.ModuleList): A list of SeamlessM4TConformerAdapterLayer instances, each representing an adapter layer
            in the adapter stack.

    Methods:
        __init__:
            Initializes the SeamlessM4TConformerAdapter instance with the specified configuration.

            Args:

            - config (dict): A dictionary containing configuration parameters for the adapter.

        forward:
            Constructs the adapter by applying each adapter layer in the stack to the input hidden states.

            Args:

            - hidden_states (Tensor): The input hidden states to be adapted by the adapter.
            - attention_mask (Tensor): The attention mask to be applied during adaptation.

            Returns:

            - Tensor: The adapted hidden states after passing through all adapter layers.
    """
    def __init__(self, config):
        """
        Initializes an instance of the SeamlessM4TConformerAdapter class.

        Args:
            self (SeamlessM4TConformerAdapter): The instance of the class itself.
            config:
                A configuration object containing the necessary parameters for initializing the adapter.

                - num_adapter_layers (int): The number of adapter layers to create.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()

        self.layers = nn.ModuleList([SeamlessM4TConformerAdapterLayer(config) for _ in range(config.num_adapter_layers)])

    def forward(self, hidden_states, attention_mask):
        """
        Constructs the SeamlessM4TConformerAdapter by applying the layers to the input hidden states.

        Args:
            self (SeamlessM4TConformerAdapter): An instance of the SeamlessM4TConformerAdapter class.
            hidden_states (Tensor): The input hidden states. It should have a shape of
                [batch_size, sequence_length, hidden_size].
            attention_mask (Tensor): The attention mask tensor. It should have a shape of [batch_size, sequence_length]
                and is used to mask certain positions in the input sequence.

        Returns:
            None.

        Raises:
            None.
        """
        # down project hidden_states if necessary

        for layer in self.layers:
            hidden_states = layer(hidden_states, attention_mask)

        return hidden_states

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerAdapter.__init__(config)

Initializes an instance of the SeamlessM4TConformerAdapter class.

PARAMETER	DESCRIPTION
self	The instance of the class itself. TYPE: SeamlessM4TConformerAdapter
config	A configuration object containing the necessary parameters for initializing the adapter. num_adapter_layers (int): The number of adapter layers to create.

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def __init__(self, config):
    """
    Initializes an instance of the SeamlessM4TConformerAdapter class.

    Args:
        self (SeamlessM4TConformerAdapter): The instance of the class itself.
        config:
            A configuration object containing the necessary parameters for initializing the adapter.

            - num_adapter_layers (int): The number of adapter layers to create.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()

    self.layers = nn.ModuleList([SeamlessM4TConformerAdapterLayer(config) for _ in range(config.num_adapter_layers)])

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerAdapter.forward(hidden_states, attention_mask)

Constructs the SeamlessM4TConformerAdapter by applying the layers to the input hidden states.

PARAMETER	DESCRIPTION
self	An instance of the SeamlessM4TConformerAdapter class. TYPE: SeamlessM4TConformerAdapter
hidden_states	The input hidden states. It should have a shape of [batch_size, sequence_length, hidden_size]. TYPE: Tensor
attention_mask	The attention mask tensor. It should have a shape of [batch_size, sequence_length] and is used to mask certain positions in the input sequence. TYPE: Tensor

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def forward(self, hidden_states, attention_mask):
    """
    Constructs the SeamlessM4TConformerAdapter by applying the layers to the input hidden states.

    Args:
        self (SeamlessM4TConformerAdapter): An instance of the SeamlessM4TConformerAdapter class.
        hidden_states (Tensor): The input hidden states. It should have a shape of
            [batch_size, sequence_length, hidden_size].
        attention_mask (Tensor): The attention mask tensor. It should have a shape of [batch_size, sequence_length]
            and is used to mask certain positions in the input sequence.

    Returns:
        None.

    Raises:
        None.
    """
    # down project hidden_states if necessary

    for layer in self.layers:
        hidden_states = layer(hidden_states, attention_mask)

    return hidden_states

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerAdapterLayer

Bases: Module

The SeamlessM4TConformerAdapterLayer class is a Python class that represents a layer in the SeamlessM4TConformer adapter model. This layer is used to adapt the input hidden states using self-attention and feed-forward networks.

This class inherits from the nn.Module class.

ATTRIBUTE	DESCRIPTION
`kernel_size`	The size of the kernel used in the convolutional layers. TYPE: int
`stride`	The stride used in the convolutional layers. TYPE: int
`residual_layer_norm`	A layer normalization module applied to the residual hidden states. TYPE: LayerNorm
`residual_conv`	A 1D convolutional layer used to transform the residual hidden states. TYPE: Conv1d
`activation`	The activation function applied to the transformed residual hidden states. TYPE: GLU
`self_attn_layer_norm`	A layer normalization module applied to the self-attention hidden states. TYPE: LayerNorm
`self_attn_conv`	A 1D convolutional layer used to transform the self-attention hidden states. TYPE: Conv1d
`self_attn`	The self-attention module used to compute attention weights. TYPE: SeamlessM4TConformerSelfAttention
`self_attn_dropout`	A dropout layer applied to the self-attention hidden states. TYPE: Dropout
`ffn_layer_norm`	A layer normalization module applied to the feed-forward hidden states. TYPE: LayerNorm
`ffn`	The feed-forward module used to transform the feed-forward hidden states. TYPE: SeamlessM4TConformerFeedForward

METHOD	DESCRIPTION
`_compute_sub_sample_lengths_from_attention_mask`	Computes the sub-sampled lengths of the hidden states based on the attention mask.
`forward`	Constructs the output hidden states by applying the adapter layer transformations to the input hidden states.

Note

This class assumes the existence of the following helper functions: _compute_new_attention_mask, _prepare_4d_attention_mask.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

class SeamlessM4TConformerAdapterLayer(nn.Module):

    """
    The `SeamlessM4TConformerAdapterLayer` class is a Python class that represents a layer in the SeamlessM4TConformer
    adapter model. This layer is used to adapt the input hidden states using self-attention and feed-forward networks.

    This class inherits from the `nn.Module` class.

    Attributes:
        `kernel_size` (int): The size of the kernel used in the convolutional layers.
        `stride` (int): The stride used in the convolutional layers.
        `residual_layer_norm` (nn.LayerNorm): A layer normalization module applied to the residual hidden states.
        `residual_conv` (nn.Conv1d): A 1D convolutional layer used to transform the residual hidden states.
        `activation` (nn.GLU): The activation function applied to the transformed residual hidden states.
        `self_attn_layer_norm` (nn.LayerNorm): A layer normalization module applied to the self-attention hidden states.
        `self_attn_conv` (nn.Conv1d): A 1D convolutional layer used to transform the self-attention hidden states.
        `self_attn` (SeamlessM4TConformerSelfAttention): The self-attention module used to compute attention weights.
        `self_attn_dropout` (nn.Dropout): A dropout layer applied to the self-attention hidden states.
        `ffn_layer_norm` (nn.LayerNorm): A layer normalization module applied to the feed-forward hidden states.
        `ffn` (SeamlessM4TConformerFeedForward): The feed-forward module used to transform the feed-forward hidden states.

    Methods:
        `_compute_sub_sample_lengths_from_attention_mask`: Computes the sub-sampled lengths of the hidden states
            based on the attention mask.
        `forward`: Constructs the output hidden states by applying the adapter layer transformations to the
            input hidden states.

    Note:
        This class assumes the existence of the following helper functions: `_compute_new_attention_mask`,
        `_prepare_4d_attention_mask`.

    """
    def __init__(self, config):
        """Initializes an instance of the SeamlessM4TConformerAdapterLayer class.

        Args:
            self: The instance of the class.
            config:
                An object of the configuration class containing the following attributes:

                - hidden_size: An integer representing the size of the hidden dimension.
                - adaptor_dropout: A float representing the dropout probability for adapter layers.
                - adaptor_kernel_size: An integer representing the kernel size for the convolutional layer.
                - adaptor_stride: An integer representing the stride for the convolutional layer.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        embed_dim = config.hidden_size
        dropout = config.adaptor_dropout

        self.kernel_size = config.adaptor_kernel_size
        self.stride = config.adaptor_stride

        # 1. residual convolution
        self.residual_layer_norm = nn.LayerNorm([embed_dim])
        self.residual_conv = nn.Conv1d(
            embed_dim,
            2 * embed_dim,
            self.kernel_size,
            stride=self.stride,
            pad_mode='pad',
            padding=self.stride // 2,
        )
        self.activation = nn.GLU(axis=1)

        # Self-Attention
        self.self_attn_layer_norm = nn.LayerNorm([embed_dim])
        self.self_attn_conv = nn.Conv1d(
            embed_dim,
            2 * embed_dim,
            self.kernel_size,
            stride=self.stride,
            pad_mode='pad',
            padding=self.stride // 2,
        )
        self.self_attn = SeamlessM4TConformerSelfAttention(config, use_position_embeddings=False)
        self.self_attn_dropout = nn.Dropout(p=dropout)

        # Feed-forward
        self.ffn_layer_norm = nn.LayerNorm([embed_dim])
        self.ffn = SeamlessM4TConformerFeedForward(config, act_fn="relu", dropout=dropout)

    def _compute_sub_sample_lengths_from_attention_mask(self, attention_mask):
        """
        Computes the lengths of sub-samples from the given attention mask.

        Args:
            self (SeamlessM4TConformerAdapterLayer): The instance of the SeamlessM4TConformerAdapterLayer class.
            attention_mask (mindspore.Tensor): The attention mask tensor of shape [batch_size, sequence_length].
                It masks the input sequence to exclude certain positions from being attended to.
                The values should be either 0 or 1, where 0 indicates that the position is masked and 1 indicates
                that the position is not masked.

        Returns:
            None.

        Raises:
            None.

        This method calculates the lengths of sub-samples based on the attention mask provided.
        It applies the following steps:

        - Calculate the padding value based on the kernel size.
        - Calculate the sequence lengths by subtracting the sum of all non-masked positions (indicated by 1 in the mask)
        from the total sequence length.
        - Adjust the sequence lengths by considering the padding and kernel size, and divide it by the stride length.
        - Add 1 to the adjusted sequence lengths.
        - Convert the sequence lengths to the float32 data type.
        - Round down the sequence lengths to the nearest integer.

        Note:
            - The padding value is determined by dividing the kernel size by 2 and taking the integer division.
            - The stride length is assumed to be a pre-defined value.
            - The method assumes that the attention mask is a binary tensor with values 0 and 1.

        Example:
            ```python
            >>> # Create an instance of SeamlessM4TConformerAdapterLayer
            >>> adapter_layer = SeamlessM4TConformerAdapterLayer()
            ...
            >>> # Create an attention mask tensor
            >>> attention_mask = mindspore.Tensor([[1, 1, 1, 0, 0], [1, 1, 0, 0, 0]])
            ...
            >>> # Compute the sub-sample lengths from the attention mask
            >>> adapter_layer._compute_sub_sample_lengths_from_attention_mask(attention_mask)
            ```
        """
        pad = self.kernel_size // 2
        seq_lens = attention_mask.shape[1] - (1 - attention_mask.int()).sum(1)

        seq_lens = ((seq_lens + 2 * pad - self.kernel_size) / self.stride) + 1

        return seq_lens.astype(mindspore.float32).floor()

    def forward(
        self,
        hidden_states,
        attention_mask: Optional[mindspore.Tensor] = None,
        output_attentions: bool = False,
    ):
        """
        Constructs a SeamlessM4TConformerAdapterLayer.

        This method applies the necessary transformations and computations to the input `hidden_states` to produce
        the final output `hidden_states`.

        Args:
            self (SeamlessM4TConformerAdapterLayer): The instance of the SeamlessM4TConformerAdapterLayer class.
            hidden_states (mindspore.Tensor): The input hidden states tensor. It should have a shape of
                (batch_size, sequence_length, hidden_size).
            attention_mask (Optional[mindspore.Tensor]): An optional tensor representing the attention mask.
                It should have a shape of (batch_size, sequence_length).
            output_attentions (bool): A flag indicating whether to output attentions. Defaults to False.

        Returns:
            mindspore.Tensor: The output hidden states tensor. It has the same shape as the input `hidden_states`.

        Raises:
            None.
        """
        residual = self.residual_layer_norm(hidden_states)

        # Apply pooling to the residual to match the sequence length of the
        # multi-head attention output.
        # (batch, seq_len, feature_dim) -> (batch, feature_dim, seq_len)
        residual = residual.swapaxes(1, 2)
        residual = self.residual_conv(residual)
        residual = self.activation(residual)
        # (batch, feature_dim, seq_len) -> (batch, seq_len, feature_dim)
        residual = residual.swapaxes(1, 2)

        hidden_states = self.self_attn_layer_norm(hidden_states)
        # Apply pooling before feeding to the multihead-attention layer.
        # (batch, seq_len, feature_dim) -> (batch, feature_dim, seq_len)
        hidden_states = hidden_states.swapaxes(1, 2)
        hidden_states = self.self_attn_conv(hidden_states)
        hidden_states = self.activation(hidden_states)
        # (batch, feature_dim, seq_len) -> (batch, seq_len, feature_dim)
        hidden_states = hidden_states.swapaxes(1, 2)

        if attention_mask is not None:
            sub_sampled_lengths = self._compute_sub_sample_lengths_from_attention_mask(attention_mask)
            attention_mask = _compute_new_attention_mask(hidden_states=hidden_states, seq_lens=sub_sampled_lengths)
            attention_mask = _prepare_4d_attention_mask(
                attention_mask,
                hidden_states.dtype,
            )

        # The rest of the computation is identical to a vanilla Transformer
        # encoder layer.
        hidden_states, _ = self.self_attn(
            hidden_states,
            attention_mask=attention_mask,
            output_attentions=output_attentions,
        )
        hidden_states = self.self_attn_dropout(hidden_states)
        hidden_states = hidden_states + residual

        residual = hidden_states

        hidden_states = self.ffn_layer_norm(hidden_states)
        hidden_states = self.ffn(hidden_states) + residual

        return hidden_states

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerAdapterLayer.__init__(config)

Initializes an instance of the SeamlessM4TConformerAdapterLayer class.

PARAMETER	DESCRIPTION
self	The instance of the class.
config	An object of the configuration class containing the following attributes: hidden_size: An integer representing the size of the hidden dimension. adaptor_dropout: A float representing the dropout probability for adapter layers. adaptor_kernel_size: An integer representing the kernel size for the convolutional layer. adaptor_stride: An integer representing the stride for the convolutional layer.

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def __init__(self, config):
    """Initializes an instance of the SeamlessM4TConformerAdapterLayer class.

    Args:
        self: The instance of the class.
        config:
            An object of the configuration class containing the following attributes:

            - hidden_size: An integer representing the size of the hidden dimension.
            - adaptor_dropout: A float representing the dropout probability for adapter layers.
            - adaptor_kernel_size: An integer representing the kernel size for the convolutional layer.
            - adaptor_stride: An integer representing the stride for the convolutional layer.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    embed_dim = config.hidden_size
    dropout = config.adaptor_dropout

    self.kernel_size = config.adaptor_kernel_size
    self.stride = config.adaptor_stride

    # 1. residual convolution
    self.residual_layer_norm = nn.LayerNorm([embed_dim])
    self.residual_conv = nn.Conv1d(
        embed_dim,
        2 * embed_dim,
        self.kernel_size,
        stride=self.stride,
        pad_mode='pad',
        padding=self.stride // 2,
    )
    self.activation = nn.GLU(axis=1)

    # Self-Attention
    self.self_attn_layer_norm = nn.LayerNorm([embed_dim])
    self.self_attn_conv = nn.Conv1d(
        embed_dim,
        2 * embed_dim,
        self.kernel_size,
        stride=self.stride,
        pad_mode='pad',
        padding=self.stride // 2,
    )
    self.self_attn = SeamlessM4TConformerSelfAttention(config, use_position_embeddings=False)
    self.self_attn_dropout = nn.Dropout(p=dropout)

    # Feed-forward
    self.ffn_layer_norm = nn.LayerNorm([embed_dim])
    self.ffn = SeamlessM4TConformerFeedForward(config, act_fn="relu", dropout=dropout)

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerAdapterLayer.forward(hidden_states, attention_mask=None, output_attentions=False)

Constructs a SeamlessM4TConformerAdapterLayer.

This method applies the necessary transformations and computations to the input hidden_states to produce the final output hidden_states.

PARAMETER	DESCRIPTION
self	The instance of the SeamlessM4TConformerAdapterLayer class. TYPE: SeamlessM4TConformerAdapterLayer
hidden_states	The input hidden states tensor. It should have a shape of (batch_size, sequence_length, hidden_size). TYPE: Tensor
attention_mask	An optional tensor representing the attention mask. It should have a shape of (batch_size, sequence_length). TYPE: Optional[Tensor] DEFAULT: None
output_attentions	A flag indicating whether to output attentions. Defaults to False. TYPE: bool DEFAULT: False

RETURNS	DESCRIPTION
	mindspore.Tensor: The output hidden states tensor. It has the same shape as the input hidden_states.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def forward(
    self,
    hidden_states,
    attention_mask: Optional[mindspore.Tensor] = None,
    output_attentions: bool = False,
):
    """
    Constructs a SeamlessM4TConformerAdapterLayer.

    This method applies the necessary transformations and computations to the input `hidden_states` to produce
    the final output `hidden_states`.

    Args:
        self (SeamlessM4TConformerAdapterLayer): The instance of the SeamlessM4TConformerAdapterLayer class.
        hidden_states (mindspore.Tensor): The input hidden states tensor. It should have a shape of
            (batch_size, sequence_length, hidden_size).
        attention_mask (Optional[mindspore.Tensor]): An optional tensor representing the attention mask.
            It should have a shape of (batch_size, sequence_length).
        output_attentions (bool): A flag indicating whether to output attentions. Defaults to False.

    Returns:
        mindspore.Tensor: The output hidden states tensor. It has the same shape as the input `hidden_states`.

    Raises:
        None.
    """
    residual = self.residual_layer_norm(hidden_states)

    # Apply pooling to the residual to match the sequence length of the
    # multi-head attention output.
    # (batch, seq_len, feature_dim) -> (batch, feature_dim, seq_len)
    residual = residual.swapaxes(1, 2)
    residual = self.residual_conv(residual)
    residual = self.activation(residual)
    # (batch, feature_dim, seq_len) -> (batch, seq_len, feature_dim)
    residual = residual.swapaxes(1, 2)

    hidden_states = self.self_attn_layer_norm(hidden_states)
    # Apply pooling before feeding to the multihead-attention layer.
    # (batch, seq_len, feature_dim) -> (batch, feature_dim, seq_len)
    hidden_states = hidden_states.swapaxes(1, 2)
    hidden_states = self.self_attn_conv(hidden_states)
    hidden_states = self.activation(hidden_states)
    # (batch, feature_dim, seq_len) -> (batch, seq_len, feature_dim)
    hidden_states = hidden_states.swapaxes(1, 2)

    if attention_mask is not None:
        sub_sampled_lengths = self._compute_sub_sample_lengths_from_attention_mask(attention_mask)
        attention_mask = _compute_new_attention_mask(hidden_states=hidden_states, seq_lens=sub_sampled_lengths)
        attention_mask = _prepare_4d_attention_mask(
            attention_mask,
            hidden_states.dtype,
        )

    # The rest of the computation is identical to a vanilla Transformer
    # encoder layer.
    hidden_states, _ = self.self_attn(
        hidden_states,
        attention_mask=attention_mask,
        output_attentions=output_attentions,
    )
    hidden_states = self.self_attn_dropout(hidden_states)
    hidden_states = hidden_states + residual

    residual = hidden_states

    hidden_states = self.ffn_layer_norm(hidden_states)
    hidden_states = self.ffn(hidden_states) + residual

    return hidden_states

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerConvolutionModule

Bases: Module

Convolution block used in the conformer block

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

class SeamlessM4TConformerConvolutionModule(nn.Module):
    """Convolution block used in the conformer block"""
    def __init__(self, config):
        """
        Initializes the SeamlessM4TConformerConvolutionModule.

        Args:
            self (object): The instance of the class.
            config (object):
                An object containing configuration parameters for the module.

                - conv_depthwise_kernel_size (int): The kernel size for depthwise convolution.
                - hidden_size (int): The size of the hidden layer.
                - speech_encoder_hidden_act (str): The activation function for the hidden layer.
                - speech_encoder_dropout (float): The dropout rate.

        Returns:
            None.

        Raises:
            ValueError: Raised if the 'config.conv_depthwise_kernel_size' is not an odd number, which is required
                for 'SAME' padding.
        """
        super().__init__()
        if (config.conv_depthwise_kernel_size - 1) % 2 == 1:
            raise ValueError("`config.conv_depthwise_kernel_size` should be a odd number for 'SAME' padding")
        self.layer_norm = nn.LayerNorm([config.hidden_size])
        self.pointwise_conv1 = nn.Conv1d(
            config.hidden_size,
            2 * config.hidden_size,
            kernel_size=1,
            stride=1,
            padding=0,
            bias=False,
        )
        self.glu = nn.GLU(axis=1)
        self.depthwise_conv = nn.Conv1d(
            config.hidden_size,
            config.hidden_size,
            config.conv_depthwise_kernel_size,
            stride=1,
            pad_mode="same",
            group=config.hidden_size,
            bias=False,
        )
        self.batch_norm = nn.BatchNorm1d(config.hidden_size)
        self.activation = ACT2FN[config.speech_encoder_hidden_act]
        self.pointwise_conv2 = nn.Conv1d(
            config.hidden_size,
            config.hidden_size,
            kernel_size=1,
            stride=1,
            padding=0,
            bias=False,
        )
        self.dropout = nn.Dropout(p=config.speech_encoder_dropout)

    def forward(self, hidden_states, attention_mask=None):
        """
        Constructs the SeamlessM4TConformerConvolutionModule.

        Args:
            self: The instance of the SeamlessM4TConformerConvolutionModule class.
            hidden_states (torch.Tensor): The input hidden states. It should have shape
                (batch_size, sequence_length, hidden_size).
            attention_mask (torch.Tensor, optional): An optional attention mask. It should have the same shape as
                hidden_states. Each element of the mask should be 0 or 1, indicating whether a token is valid or masked.
                If provided, the hidden states corresponding to the masked tokens will be set to 0.0. Default is None.

        Returns:
            torch.Tensor: The transformed hidden states after passing through the SeamlessM4TConformerConvolutionModule.
                It has the same shape as the input hidden states.

        Raises:
            None.
        """
        hidden_states = self.layer_norm(hidden_states)

        # Ensure that we do not leak padded positions in depthwise convolution.
        # Put 0 where necessary
        if attention_mask is not None:
            hidden_states = hidden_states.masked_fill(~attention_mask.bool().unsqueeze(-1), 0.0)

        # exchange the temporal dimension and the feature dimension
        hidden_states = hidden_states.swapaxes(1, 2)

        # GLU mechanism
        # => (batch, 2*channel, dim)
        hidden_states = self.pointwise_conv1(hidden_states)
        # => (batch, channel, dim)
        hidden_states = self.glu(hidden_states)

        # 1D Depthwise Conv
        hidden_states = self.depthwise_conv(hidden_states)
        hidden_states = self.batch_norm(hidden_states)
        hidden_states = self.activation(hidden_states)

        hidden_states = self.pointwise_conv2(hidden_states)
        hidden_states = self.dropout(hidden_states)
        hidden_states = hidden_states.swapaxes(1, 2)
        return hidden_states

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerConvolutionModule.__init__(config)

Initializes the SeamlessM4TConformerConvolutionModule.

PARAMETER	DESCRIPTION
self	The instance of the class. TYPE: object
config	An object containing configuration parameters for the module. conv_depthwise_kernel_size (int): The kernel size for depthwise convolution. hidden_size (int): The size of the hidden layer. speech_encoder_hidden_act (str): The activation function for the hidden layer. speech_encoder_dropout (float): The dropout rate. TYPE: object

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	Raised if the 'config.conv_depthwise_kernel_size' is not an odd number, which is required for 'SAME' padding.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def __init__(self, config):
    """
    Initializes the SeamlessM4TConformerConvolutionModule.

    Args:
        self (object): The instance of the class.
        config (object):
            An object containing configuration parameters for the module.

            - conv_depthwise_kernel_size (int): The kernel size for depthwise convolution.
            - hidden_size (int): The size of the hidden layer.
            - speech_encoder_hidden_act (str): The activation function for the hidden layer.
            - speech_encoder_dropout (float): The dropout rate.

    Returns:
        None.

    Raises:
        ValueError: Raised if the 'config.conv_depthwise_kernel_size' is not an odd number, which is required
            for 'SAME' padding.
    """
    super().__init__()
    if (config.conv_depthwise_kernel_size - 1) % 2 == 1:
        raise ValueError("`config.conv_depthwise_kernel_size` should be a odd number for 'SAME' padding")
    self.layer_norm = nn.LayerNorm([config.hidden_size])
    self.pointwise_conv1 = nn.Conv1d(
        config.hidden_size,
        2 * config.hidden_size,
        kernel_size=1,
        stride=1,
        padding=0,
        bias=False,
    )
    self.glu = nn.GLU(axis=1)
    self.depthwise_conv = nn.Conv1d(
        config.hidden_size,
        config.hidden_size,
        config.conv_depthwise_kernel_size,
        stride=1,
        pad_mode="same",
        group=config.hidden_size,
        bias=False,
    )
    self.batch_norm = nn.BatchNorm1d(config.hidden_size)
    self.activation = ACT2FN[config.speech_encoder_hidden_act]
    self.pointwise_conv2 = nn.Conv1d(
        config.hidden_size,
        config.hidden_size,
        kernel_size=1,
        stride=1,
        padding=0,
        bias=False,
    )
    self.dropout = nn.Dropout(p=config.speech_encoder_dropout)

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerConvolutionModule.forward(hidden_states, attention_mask=None)

Constructs the SeamlessM4TConformerConvolutionModule.

PARAMETER	DESCRIPTION
self	The instance of the SeamlessM4TConformerConvolutionModule class.
hidden_states	The input hidden states. It should have shape (batch_size, sequence_length, hidden_size). TYPE: Tensor
attention_mask	An optional attention mask. It should have the same shape as hidden_states. Each element of the mask should be 0 or 1, indicating whether a token is valid or masked. If provided, the hidden states corresponding to the masked tokens will be set to 0.0. Default is None. TYPE: Tensor DEFAULT: None

RETURNS	DESCRIPTION
	torch.Tensor: The transformed hidden states after passing through the SeamlessM4TConformerConvolutionModule. It has the same shape as the input hidden states.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def forward(self, hidden_states, attention_mask=None):
    """
    Constructs the SeamlessM4TConformerConvolutionModule.

    Args:
        self: The instance of the SeamlessM4TConformerConvolutionModule class.
        hidden_states (torch.Tensor): The input hidden states. It should have shape
            (batch_size, sequence_length, hidden_size).
        attention_mask (torch.Tensor, optional): An optional attention mask. It should have the same shape as
            hidden_states. Each element of the mask should be 0 or 1, indicating whether a token is valid or masked.
            If provided, the hidden states corresponding to the masked tokens will be set to 0.0. Default is None.

    Returns:
        torch.Tensor: The transformed hidden states after passing through the SeamlessM4TConformerConvolutionModule.
            It has the same shape as the input hidden states.

    Raises:
        None.
    """
    hidden_states = self.layer_norm(hidden_states)

    # Ensure that we do not leak padded positions in depthwise convolution.
    # Put 0 where necessary
    if attention_mask is not None:
        hidden_states = hidden_states.masked_fill(~attention_mask.bool().unsqueeze(-1), 0.0)

    # exchange the temporal dimension and the feature dimension
    hidden_states = hidden_states.swapaxes(1, 2)

    # GLU mechanism
    # => (batch, 2*channel, dim)
    hidden_states = self.pointwise_conv1(hidden_states)
    # => (batch, channel, dim)
    hidden_states = self.glu(hidden_states)

    # 1D Depthwise Conv
    hidden_states = self.depthwise_conv(hidden_states)
    hidden_states = self.batch_norm(hidden_states)
    hidden_states = self.activation(hidden_states)

    hidden_states = self.pointwise_conv2(hidden_states)
    hidden_states = self.dropout(hidden_states)
    hidden_states = hidden_states.swapaxes(1, 2)
    return hidden_states

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerEncoder

Bases: Module

This class represents a SeamlessM4TConformerEncoder which is responsible for encoding input sequences using a Conformer model architecture. The encoder consists of multiple ConformerEncoderLayer instances stacked on top of each other. It handles positional embeddings, dropout, layer normalization, and gradient checkpointing.

PARAMETER	DESCRIPTION
config	An object containing configuration parameters for the encoder.
Inherits	nn.Module TYPE: from

METHOD	DESCRIPTION
__init__	Initializes the SeamlessM4TConformerEncoder with the provided configuration. Sets up positional embeddings based on the specified type, dropout, encoder layers, layer normalization, and gradient checkpointing.
forward	Constructs the encoder by processing the input hidden states through each encoder layer. It applies dropout, handles attention masks, and computes relative position embeddings. Returns the encoded hidden states, hidden states history if enabled, and attention weights if requested.

ATTRIBUTE	DESCRIPTION
config	Configuration parameters for the encoder.
embed_positions	Positional embedding module based on the specified type ('relative' or 'rotary').
dropout	Dropout module for regularization.
layers	List of ConformerEncoderLayer instances representing the stacked encoder layers.
layer_norm	Layer normalization module to normalize the hidden states.
gradient_checkpointing	Flag indicating whether gradient checkpointing is enabled.

For detailed usage instructions and examples, refer to the official documentation.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

class SeamlessM4TConformerEncoder(nn.Module):

    """
    This class represents a SeamlessM4TConformerEncoder which is responsible for encoding input sequences using a
    Conformer model architecture.
    The encoder consists of multiple ConformerEncoderLayer instances stacked on top of each other.
    It handles positional embeddings, dropout, layer normalization, and gradient checkpointing.

    Parameters:
        config: An object containing configuration parameters for the encoder.
        Inherits from: nn.Module

    Methods:
        __init__: Initializes the SeamlessM4TConformerEncoder with the provided configuration. Sets up positional
            embeddings based on the specified type, dropout, encoder layers, layer normalization, and
            gradient checkpointing.
        forward: Constructs the encoder by processing the input hidden states through each encoder layer.
            It applies dropout, handles attention masks, and computes relative position embeddings.
            Returns the encoded hidden states, hidden states history if enabled, and attention weights if requested.

    Attributes:
        config: Configuration parameters for the encoder.
        embed_positions: Positional embedding module based on the specified type ('relative' or 'rotary').
        dropout: Dropout module for regularization.
        layers: List of ConformerEncoderLayer instances representing the stacked encoder layers.
        layer_norm: Layer normalization module to normalize the hidden states.
        gradient_checkpointing: Flag indicating whether gradient checkpointing is enabled.

    For detailed usage instructions and examples, refer to the official documentation.
    """
    def __init__(self, config):
        """
        Initializes an instance of the SeamlessM4TConformerEncoder class.

        Args:
            self: An instance of the SeamlessM4TConformerEncoder class.
            config: An object of type Config that contains configuration parameters
                for the SeamlessM4TConformerEncoder.

        Returns:
            None

        Raises:
            None

        This method initializes the SeamlessM4TConformerEncoder with the given configuration parameters.
        It sets the configuration parameters for the instance and initializes the positional embedding based
        on the type of position embedding specified in the configuration. The method also sets the dropout probability,
        creates a list of encoder layers based on the number of layers specified in the configuration, normalizes the
        outputs of the encoder layer using LayerNorm, and sets the gradient checkpointing flag to False.
        """
        super().__init__()
        self.config = config

        if config.position_embeddings_type == "relative":
            self.embed_positions = SeamlessM4TConformerRelPositionalEmbedding(config)
        elif config.position_embeddings_type == "rotary":
            self.embed_positions = SeamlessM4TConformerRotaryPositionalEmbedding(config)
        else:
            self.embed_positions = None

        self.dropout = nn.Dropout(p=config.speech_encoder_dropout)
        self.layers = nn.ModuleList(
            [SeamlessM4TConformerEncoderLayer(config) for _ in range(config.speech_encoder_layers)]
        )

        self.layer_norm = nn.LayerNorm([config.hidden_size], eps=config.layer_norm_eps)

        self.gradient_checkpointing = False

    def forward(
        self,
        hidden_states,
        attention_mask=None,
        output_attentions=False,
        output_hidden_states=False,
        return_dict=True,
    ):
        """
        Construct method in the SeamlessM4TConformerEncoder class.

        Args:
            self: The instance of the SeamlessM4TConformerEncoder class.
            hidden_states (tensor): The input hidden states to be processed by the encoder.
            attention_mask (tensor, optional): A tensor representing the attention mask to be applied during processing.
                Defaults to None.
            output_attentions (bool, optional): A flag indicating whether to output the attention weights.
                Defaults to False.
            output_hidden_states (bool, optional): A flag indicating whether to output the hidden states of each layer.
                Defaults to False.
            return_dict (bool, optional): A flag indicating whether to return the outputs as a dictionary.
                Defaults to True.

        Returns:
            None: The method does not explicitly return a value, but updates hidden_states, all_hidden_states,
                and all_self_attentions within the class instance.

        Raises:
            TypeError: If the input arguments are of incorrect types.
            ValueError: If the input hidden_states and attention_mask have incompatible shapes.
            RuntimeError: If an error occurs during processing or if the input tensors are not well-formed.
        """
        all_hidden_states = () if output_hidden_states else None
        all_self_attentions = () if output_attentions else None

        conv_attention_mask = attention_mask
        if attention_mask is not None:
            # make sure padded tokens output 0
            hidden_states = hidden_states.masked_fill(~attention_mask.bool().unsqueeze(-1), 0.0)
            # extend attention_mask
            attention_mask = 1.0 - attention_mask[:, None, None, :].to(dtype=hidden_states.dtype)
            attention_mask = attention_mask * float(np.finfo(mindspore.dtype_to_nptype(hidden_states.dtype)).min)
            attention_mask = attention_mask.expand(
                attention_mask.shape[0], 1, attention_mask.shape[-1], attention_mask.shape[-1]
            )

        hidden_states = self.dropout(hidden_states)

        if self.embed_positions is not None:
            relative_position_embeddings = self.embed_positions(hidden_states)
        else:
            relative_position_embeddings = None

        for _, layer in enumerate(self.layers):
            if output_hidden_states:
                all_hidden_states = all_hidden_states + (hidden_states,)

            # add LayerDrop (see https://arxiv.org/abs/1909.11556 for description)
            dropout_probability = ops.rand([])

            skip_the_layer = bool(self.training and (dropout_probability < self.config.speech_encoder_layerdrop))
            if not skip_the_layer:
                # under deepspeed zero3 all gpus must run in sync
                layer_outputs = layer(
                    hidden_states,
                    attention_mask=attention_mask,
                    relative_position_embeddings=relative_position_embeddings,
                    output_attentions=output_attentions,
                    conv_attention_mask=conv_attention_mask,
                )
                hidden_states = layer_outputs[0]

            if skip_the_layer:
                layer_outputs = (None, None)

            if output_attentions:
                all_self_attentions = all_self_attentions + (layer_outputs[1],)

        hidden_states = self.layer_norm(hidden_states)
        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        if not return_dict:
            return tuple(v for v in [hidden_states, all_hidden_states, all_self_attentions] if v is not None)
        return BaseModelOutput(
            last_hidden_state=hidden_states,
            hidden_states=all_hidden_states,
            attentions=all_self_attentions,
        )

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerEncoder.__init__(config)

Initializes an instance of the SeamlessM4TConformerEncoder class.

PARAMETER	DESCRIPTION
self	An instance of the SeamlessM4TConformerEncoder class.
config	An object of type Config that contains configuration parameters for the SeamlessM4TConformerEncoder.

RETURNS	DESCRIPTION
	None

This method initializes the SeamlessM4TConformerEncoder with the given configuration parameters. It sets the configuration parameters for the instance and initializes the positional embedding based on the type of position embedding specified in the configuration. The method also sets the dropout probability, creates a list of encoder layers based on the number of layers specified in the configuration, normalizes the outputs of the encoder layer using LayerNorm, and sets the gradient checkpointing flag to False.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def __init__(self, config):
    """
    Initializes an instance of the SeamlessM4TConformerEncoder class.

    Args:
        self: An instance of the SeamlessM4TConformerEncoder class.
        config: An object of type Config that contains configuration parameters
            for the SeamlessM4TConformerEncoder.

    Returns:
        None

    Raises:
        None

    This method initializes the SeamlessM4TConformerEncoder with the given configuration parameters.
    It sets the configuration parameters for the instance and initializes the positional embedding based
    on the type of position embedding specified in the configuration. The method also sets the dropout probability,
    creates a list of encoder layers based on the number of layers specified in the configuration, normalizes the
    outputs of the encoder layer using LayerNorm, and sets the gradient checkpointing flag to False.
    """
    super().__init__()
    self.config = config

    if config.position_embeddings_type == "relative":
        self.embed_positions = SeamlessM4TConformerRelPositionalEmbedding(config)
    elif config.position_embeddings_type == "rotary":
        self.embed_positions = SeamlessM4TConformerRotaryPositionalEmbedding(config)
    else:
        self.embed_positions = None

    self.dropout = nn.Dropout(p=config.speech_encoder_dropout)
    self.layers = nn.ModuleList(
        [SeamlessM4TConformerEncoderLayer(config) for _ in range(config.speech_encoder_layers)]
    )

    self.layer_norm = nn.LayerNorm([config.hidden_size], eps=config.layer_norm_eps)

    self.gradient_checkpointing = False

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerEncoder.forward(hidden_states, attention_mask=None, output_attentions=False, output_hidden_states=False, return_dict=True)

Construct method in the SeamlessM4TConformerEncoder class.

PARAMETER	DESCRIPTION
self	The instance of the SeamlessM4TConformerEncoder class.
hidden_states	The input hidden states to be processed by the encoder. TYPE: tensor
attention_mask	A tensor representing the attention mask to be applied during processing. Defaults to None. TYPE: tensor DEFAULT: None
output_attentions	A flag indicating whether to output the attention weights. Defaults to False. TYPE: bool DEFAULT: False
output_hidden_states	A flag indicating whether to output the hidden states of each layer. Defaults to False. TYPE: bool DEFAULT: False
return_dict	A flag indicating whether to return the outputs as a dictionary. Defaults to True. TYPE: bool DEFAULT: True

RETURNS	DESCRIPTION
None	The method does not explicitly return a value, but updates hidden_states, all_hidden_states, and all_self_attentions within the class instance.

RAISES	DESCRIPTION
TypeError	If the input arguments are of incorrect types.
ValueError	If the input hidden_states and attention_mask have incompatible shapes.
RuntimeError	If an error occurs during processing or if the input tensors are not well-formed.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def forward(
    self,
    hidden_states,
    attention_mask=None,
    output_attentions=False,
    output_hidden_states=False,
    return_dict=True,
):
    """
    Construct method in the SeamlessM4TConformerEncoder class.

    Args:
        self: The instance of the SeamlessM4TConformerEncoder class.
        hidden_states (tensor): The input hidden states to be processed by the encoder.
        attention_mask (tensor, optional): A tensor representing the attention mask to be applied during processing.
            Defaults to None.
        output_attentions (bool, optional): A flag indicating whether to output the attention weights.
            Defaults to False.
        output_hidden_states (bool, optional): A flag indicating whether to output the hidden states of each layer.
            Defaults to False.
        return_dict (bool, optional): A flag indicating whether to return the outputs as a dictionary.
            Defaults to True.

    Returns:
        None: The method does not explicitly return a value, but updates hidden_states, all_hidden_states,
            and all_self_attentions within the class instance.

    Raises:
        TypeError: If the input arguments are of incorrect types.
        ValueError: If the input hidden_states and attention_mask have incompatible shapes.
        RuntimeError: If an error occurs during processing or if the input tensors are not well-formed.
    """
    all_hidden_states = () if output_hidden_states else None
    all_self_attentions = () if output_attentions else None

    conv_attention_mask = attention_mask
    if attention_mask is not None:
        # make sure padded tokens output 0
        hidden_states = hidden_states.masked_fill(~attention_mask.bool().unsqueeze(-1), 0.0)
        # extend attention_mask
        attention_mask = 1.0 - attention_mask[:, None, None, :].to(dtype=hidden_states.dtype)
        attention_mask = attention_mask * float(np.finfo(mindspore.dtype_to_nptype(hidden_states.dtype)).min)
        attention_mask = attention_mask.expand(
            attention_mask.shape[0], 1, attention_mask.shape[-1], attention_mask.shape[-1]
        )

    hidden_states = self.dropout(hidden_states)

    if self.embed_positions is not None:
        relative_position_embeddings = self.embed_positions(hidden_states)
    else:
        relative_position_embeddings = None

    for _, layer in enumerate(self.layers):
        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        # add LayerDrop (see https://arxiv.org/abs/1909.11556 for description)
        dropout_probability = ops.rand([])

        skip_the_layer = bool(self.training and (dropout_probability < self.config.speech_encoder_layerdrop))
        if not skip_the_layer:
            # under deepspeed zero3 all gpus must run in sync
            layer_outputs = layer(
                hidden_states,
                attention_mask=attention_mask,
                relative_position_embeddings=relative_position_embeddings,
                output_attentions=output_attentions,
                conv_attention_mask=conv_attention_mask,
            )
            hidden_states = layer_outputs[0]

        if skip_the_layer:
            layer_outputs = (None, None)

        if output_attentions:
            all_self_attentions = all_self_attentions + (layer_outputs[1],)

    hidden_states = self.layer_norm(hidden_states)
    if output_hidden_states:
        all_hidden_states = all_hidden_states + (hidden_states,)

    if not return_dict:
        return tuple(v for v in [hidden_states, all_hidden_states, all_self_attentions] if v is not None)
    return BaseModelOutput(
        last_hidden_state=hidden_states,
        hidden_states=all_hidden_states,
        attentions=all_self_attentions,
    )

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerEncoderLayer

Bases: Module

Conformer block based on https://arxiv.org/abs/2005.08100.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

class SeamlessM4TConformerEncoderLayer(nn.Module):
    """Conformer block based on https://arxiv.org/abs/2005.08100."""
    # Copied from transformers.models.wav2vec2_conformer.modeling_wav2vec2_conformer.Wav2Vec2ConformerEncoderLayer.__init__ with Wav2Vec2->SeamlessM4T, attention_dropout->speech_encoder_dropout, ops.nn->nn
    def __init__(self, config):
        """
        Initializes a SeamlessM4TConformerEncoderLayer object.

        Args:
            self (SeamlessM4TConformerEncoderLayer): The instance of the class itself.
            config (object): A configuration object containing parameters for the encoder layer.
                It must have the following attributes:

                - hidden_size (int): The dimension of the hidden layers.
                - speech_encoder_dropout (float): The dropout probability for the speech encoder.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        embed_dim = config.hidden_size
        dropout = config.speech_encoder_dropout

        # Feed-forward 1
        self.ffn1_layer_norm = nn.LayerNorm([embed_dim])
        self.ffn1 = SeamlessM4TConformerFeedForward(config)

        # Self-Attention
        self.self_attn_layer_norm = nn.LayerNorm([embed_dim])
        self.self_attn_dropout = nn.Dropout(p=dropout)
        self.self_attn = SeamlessM4TConformerSelfAttention(config)

        # Conformer Convolution
        self.conv_module = SeamlessM4TConformerConvolutionModule(config)

        # Feed-forward 2
        self.ffn2_layer_norm = nn.LayerNorm([embed_dim])
        self.ffn2 = SeamlessM4TConformerFeedForward(config)
        self.final_layer_norm = nn.LayerNorm([embed_dim])

    def forward(
        self,
        hidden_states,
        attention_mask: Optional[mindspore.Tensor] = None,
        relative_position_embeddings: Optional[mindspore.Tensor] = None,
        output_attentions: bool = False,
        conv_attention_mask: Optional[mindspore.Tensor] = None,
    ):
        """
        The 'forward' method in the 'SeamlessM4TConformerEncoderLayer' class forwards the encoder layer of a
        Conformer model.

        Args:
            self: Reference to the current instance of the class.
            hidden_states (mindspore.Tensor): The input hidden states for the encoder layer.
            attention_mask (Optional[mindspore.Tensor]): An optional tensor representing the attention mask.
                Default is None.
            relative_position_embeddings (Optional[mindspore.Tensor]): Optional tensor for relative position embeddings.
                Default is None.
            output_attentions (bool): A flag indicating whether to output attention weights. Default is False.
            conv_attention_mask (Optional[mindspore.Tensor]): An optional tensor representing the convolution attention
                mask. Default is None.

        Returns:
            Tuple[mindspore.Tensor, mindspore.Tensor]: The forwarded hidden states after processing through the
                encoder layer, along with the attention weights.

        Raises:
            None.
        """
        # 1. Feed-Forward 1 layer
        residual = hidden_states
        hidden_states = self.ffn1_layer_norm(hidden_states)
        hidden_states = self.ffn1(hidden_states)
        hidden_states = hidden_states * 0.5 + residual
        residual = hidden_states

        # 2. Self-Attention layer
        hidden_states = self.self_attn_layer_norm(hidden_states)
        hidden_states, attn_weigts = self.self_attn(
            hidden_states=hidden_states,
            attention_mask=attention_mask,
            relative_position_embeddings=relative_position_embeddings,
            output_attentions=output_attentions,
        )
        hidden_states = self.self_attn_dropout(hidden_states)
        hidden_states = hidden_states + residual

        # 3. Convolutional Layer
        residual = hidden_states
        hidden_states = self.conv_module(hidden_states, attention_mask=conv_attention_mask)
        hidden_states = residual + hidden_states

        # 4. Feed-Forward 2 Layer
        residual = hidden_states
        hidden_states = self.ffn2_layer_norm(hidden_states)
        hidden_states = self.ffn2(hidden_states)
        hidden_states = hidden_states * 0.5 + residual
        hidden_states = self.final_layer_norm(hidden_states)

        return hidden_states, attn_weigts

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerEncoderLayer.__init__(config)

Initializes a SeamlessM4TConformerEncoderLayer object.

PARAMETER	DESCRIPTION
self	The instance of the class itself. TYPE: SeamlessM4TConformerEncoderLayer
config	A configuration object containing parameters for the encoder layer. It must have the following attributes: hidden_size (int): The dimension of the hidden layers. speech_encoder_dropout (float): The dropout probability for the speech encoder. TYPE: object

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def __init__(self, config):
    """
    Initializes a SeamlessM4TConformerEncoderLayer object.

    Args:
        self (SeamlessM4TConformerEncoderLayer): The instance of the class itself.
        config (object): A configuration object containing parameters for the encoder layer.
            It must have the following attributes:

            - hidden_size (int): The dimension of the hidden layers.
            - speech_encoder_dropout (float): The dropout probability for the speech encoder.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    embed_dim = config.hidden_size
    dropout = config.speech_encoder_dropout

    # Feed-forward 1
    self.ffn1_layer_norm = nn.LayerNorm([embed_dim])
    self.ffn1 = SeamlessM4TConformerFeedForward(config)

    # Self-Attention
    self.self_attn_layer_norm = nn.LayerNorm([embed_dim])
    self.self_attn_dropout = nn.Dropout(p=dropout)
    self.self_attn = SeamlessM4TConformerSelfAttention(config)

    # Conformer Convolution
    self.conv_module = SeamlessM4TConformerConvolutionModule(config)

    # Feed-forward 2
    self.ffn2_layer_norm = nn.LayerNorm([embed_dim])
    self.ffn2 = SeamlessM4TConformerFeedForward(config)
    self.final_layer_norm = nn.LayerNorm([embed_dim])

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerEncoderLayer.forward(hidden_states, attention_mask=None, relative_position_embeddings=None, output_attentions=False, conv_attention_mask=None)

The 'forward' method in the 'SeamlessM4TConformerEncoderLayer' class forwards the encoder layer of a Conformer model.

PARAMETER	DESCRIPTION
self	Reference to the current instance of the class.
hidden_states	The input hidden states for the encoder layer. TYPE: Tensor
attention_mask	An optional tensor representing the attention mask. Default is None. TYPE: Optional[Tensor] DEFAULT: None
relative_position_embeddings	Optional tensor for relative position embeddings. Default is None. TYPE: Optional[Tensor] DEFAULT: None
output_attentions	A flag indicating whether to output attention weights. Default is False. TYPE: bool DEFAULT: False
conv_attention_mask	An optional tensor representing the convolution attention mask. Default is None. TYPE: Optional[Tensor] DEFAULT: None

RETURNS	DESCRIPTION
	Tuple[mindspore.Tensor, mindspore.Tensor]: The forwarded hidden states after processing through the encoder layer, along with the attention weights.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def forward(
    self,
    hidden_states,
    attention_mask: Optional[mindspore.Tensor] = None,
    relative_position_embeddings: Optional[mindspore.Tensor] = None,
    output_attentions: bool = False,
    conv_attention_mask: Optional[mindspore.Tensor] = None,
):
    """
    The 'forward' method in the 'SeamlessM4TConformerEncoderLayer' class forwards the encoder layer of a
    Conformer model.

    Args:
        self: Reference to the current instance of the class.
        hidden_states (mindspore.Tensor): The input hidden states for the encoder layer.
        attention_mask (Optional[mindspore.Tensor]): An optional tensor representing the attention mask.
            Default is None.
        relative_position_embeddings (Optional[mindspore.Tensor]): Optional tensor for relative position embeddings.
            Default is None.
        output_attentions (bool): A flag indicating whether to output attention weights. Default is False.
        conv_attention_mask (Optional[mindspore.Tensor]): An optional tensor representing the convolution attention
            mask. Default is None.

    Returns:
        Tuple[mindspore.Tensor, mindspore.Tensor]: The forwarded hidden states after processing through the
            encoder layer, along with the attention weights.

    Raises:
        None.
    """
    # 1. Feed-Forward 1 layer
    residual = hidden_states
    hidden_states = self.ffn1_layer_norm(hidden_states)
    hidden_states = self.ffn1(hidden_states)
    hidden_states = hidden_states * 0.5 + residual
    residual = hidden_states

    # 2. Self-Attention layer
    hidden_states = self.self_attn_layer_norm(hidden_states)
    hidden_states, attn_weigts = self.self_attn(
        hidden_states=hidden_states,
        attention_mask=attention_mask,
        relative_position_embeddings=relative_position_embeddings,
        output_attentions=output_attentions,
    )
    hidden_states = self.self_attn_dropout(hidden_states)
    hidden_states = hidden_states + residual

    # 3. Convolutional Layer
    residual = hidden_states
    hidden_states = self.conv_module(hidden_states, attention_mask=conv_attention_mask)
    hidden_states = residual + hidden_states

    # 4. Feed-Forward 2 Layer
    residual = hidden_states
    hidden_states = self.ffn2_layer_norm(hidden_states)
    hidden_states = self.ffn2(hidden_states)
    hidden_states = hidden_states * 0.5 + residual
    hidden_states = self.final_layer_norm(hidden_states)

    return hidden_states, attn_weigts

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerFeatureProjection

Bases: Module

This class represents a feature projection module for the SeamlessM4TConformer model. It inherits from the nn.Module class.

The feature projection module consists of a layer normalization, a dense projection layer, and a dropout layer. It takes in hidden states as input and applies layer normalization, followed by a projection and dropout operation. The resulting hidden states are returned.

ATTRIBUTE	DESCRIPTION
layer_norm	A layer normalization module that normalizes the input hidden states. TYPE: LayerNorm
projection	A dense projection layer that projects the normalized hidden states. TYPE: Linear
dropout	A dropout layer that applies dropout to the projected hidden states. TYPE: Dropout

METHOD	DESCRIPTION
forward	Applies the feature projection to the input hidden states. Args: hidden_states (Tensor): Input hidden states to be projected. Returns: Tensor: The projected hidden states after applying layer normalization, projection, and dropout.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

class SeamlessM4TConformerFeatureProjection(nn.Module):

    """
    This class represents a feature projection module for the SeamlessM4TConformer model.
    It inherits from the nn.Module class.

    The feature projection module consists of a layer normalization, a dense projection layer,
    and a dropout layer. It takes in hidden states as input and applies layer normalization,
    followed by a projection and dropout operation. The resulting hidden states are returned.

    Attributes:
        layer_norm (nn.LayerNorm): A layer normalization module that normalizes the input hidden states.
        projection (nn.Linear): A dense projection layer that projects the normalized hidden states.
        dropout (nn.Dropout): A dropout layer that applies dropout to the projected hidden states.

    Methods:
        forward(hidden_states):
            Applies the feature projection to the input hidden states.

            Args:

            - hidden_states (Tensor): Input hidden states to be projected.

            Returns:

           - Tensor: The projected hidden states after applying layer normalization, projection, and dropout.
    """
    def __init__(self, config):
        """
        Initializes a new instance of the SeamlessM4TConformerFeatureProjection class.

        Args:
            self (SeamlessM4TConformerFeatureProjection): The current instance of the class.
            config:
                The configuration parameters for the feature projection.

                - feature_projection_input_dim (int): The input dimension for the feature projection.
                - layer_norm_eps (float): The epsilon value for layer normalization.
                - hidden_size (int): The hidden size for the projection layer.
                - speech_encoder_dropout (float): The dropout probability for the speech encoder.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.layer_norm = nn.LayerNorm([config.feature_projection_input_dim], eps=config.layer_norm_eps)
        self.projection = nn.Linear(config.feature_projection_input_dim, config.hidden_size)
        self.dropout = nn.Dropout(p=config.speech_encoder_dropout)

    def forward(self, hidden_states):
        """
        Method to forward the feature projection in the SeamlessM4TConformerFeatureProjection class.

        Args:
            self (SeamlessM4TConformerFeatureProjection): The instance of the SeamlessM4TConformerFeatureProjection
                class.
            hidden_states (Tensor): The input hidden states to be processed. Expected to be a tensor.

        Returns:
            None: This method does not return any value directly. The hidden_states are processed and modified in-place.

        Raises:
            None.
        """
        # non-projected hidden states are needed for quantization
        norm_hidden_states = self.layer_norm(hidden_states)
        hidden_states = self.projection(norm_hidden_states)
        hidden_states = self.dropout(hidden_states)
        return hidden_states

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerFeatureProjection.__init__(config)

Initializes a new instance of the SeamlessM4TConformerFeatureProjection class.

PARAMETER	DESCRIPTION
self	The current instance of the class. TYPE: SeamlessM4TConformerFeatureProjection
config	The configuration parameters for the feature projection. feature_projection_input_dim (int): The input dimension for the feature projection. layer_norm_eps (float): The epsilon value for layer normalization. hidden_size (int): The hidden size for the projection layer. speech_encoder_dropout (float): The dropout probability for the speech encoder.

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def __init__(self, config):
    """
    Initializes a new instance of the SeamlessM4TConformerFeatureProjection class.

    Args:
        self (SeamlessM4TConformerFeatureProjection): The current instance of the class.
        config:
            The configuration parameters for the feature projection.

            - feature_projection_input_dim (int): The input dimension for the feature projection.
            - layer_norm_eps (float): The epsilon value for layer normalization.
            - hidden_size (int): The hidden size for the projection layer.
            - speech_encoder_dropout (float): The dropout probability for the speech encoder.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.layer_norm = nn.LayerNorm([config.feature_projection_input_dim], eps=config.layer_norm_eps)
    self.projection = nn.Linear(config.feature_projection_input_dim, config.hidden_size)
    self.dropout = nn.Dropout(p=config.speech_encoder_dropout)

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerFeatureProjection.forward(hidden_states)

Method to forward the feature projection in the SeamlessM4TConformerFeatureProjection class.

PARAMETER	DESCRIPTION
self	The instance of the SeamlessM4TConformerFeatureProjection class. TYPE: SeamlessM4TConformerFeatureProjection
hidden_states	The input hidden states to be processed. Expected to be a tensor. TYPE: Tensor

RETURNS	DESCRIPTION
None	This method does not return any value directly. The hidden_states are processed and modified in-place.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def forward(self, hidden_states):
    """
    Method to forward the feature projection in the SeamlessM4TConformerFeatureProjection class.

    Args:
        self (SeamlessM4TConformerFeatureProjection): The instance of the SeamlessM4TConformerFeatureProjection
            class.
        hidden_states (Tensor): The input hidden states to be processed. Expected to be a tensor.

    Returns:
        None: This method does not return any value directly. The hidden_states are processed and modified in-place.

    Raises:
        None.
    """
    # non-projected hidden states are needed for quantization
    norm_hidden_states = self.layer_norm(hidden_states)
    hidden_states = self.projection(norm_hidden_states)
    hidden_states = self.dropout(hidden_states)
    return hidden_states

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerFeedForward

Bases: Module

The SeamlessM4TConformerFeedForward class represents a feed-forward neural network module for the SeamlessM4TConformer model. It inherits from the nn.Module class and contains methods for initializing the network and forwarding the feed-forward operations.

ATTRIBUTE	DESCRIPTION
config	The configuration parameters for the feed-forward network.
act_fn	The activation function to be used in the network.
dropout	The dropout probability for the network.

METHOD	DESCRIPTION
__init__	Initializes the SeamlessM4TConformerFeedForward module with the given configuration, activation function, and dropout probability.
forward	Constructs the feed-forward operations on the given hidden states, applying intermediate dense layers, activation functions, and dropout. Returns the processed hidden states.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

class SeamlessM4TConformerFeedForward(nn.Module):

    """
    The SeamlessM4TConformerFeedForward class represents a feed-forward neural network module for the
    SeamlessM4TConformer model. It inherits from the nn.Module class and contains methods for initializing the network
    and forwarding the feed-forward operations.

    Attributes:
        config: The configuration parameters for the feed-forward network.
        act_fn: The activation function to be used in the network.
        dropout: The dropout probability for the network.

    Methods:
        __init__:
            Initializes the SeamlessM4TConformerFeedForward module with the given configuration, activation function,
            and dropout probability.

        forward:
            Constructs the feed-forward operations on the given hidden states, applying intermediate dense layers,
            activation functions, and dropout. Returns the processed hidden states.
    """
    def __init__(self, config, act_fn=None, dropout=None):
        """
        Initializes the SeamlessM4TConformerFeedForward class.

        Args:
            self: The instance of the class.
            config:
                An object containing configuration settings.

                - Type: object
                - Purpose: Holds various configuration parameters for the method.
                - Restrictions: Must be provided as an argument.
            act_fn:
                Activation function to be used.

                - Type: str or callable, optional
                - Purpose: Specifies the activation function to apply.
                - Restrictions: If str, it must be a valid key in the ACT2FN dictionary.
            dropout:
                Dropout rate to be applied.

                - Type: float, optional
                - Purpose: Controls the dropout rate for regularization.
                - Restrictions: Must be a float between 0 and 1. If not provided, config.speech_encoder_dropout is used.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        dropout = dropout if dropout is not None else config.speech_encoder_dropout
        act_fn = act_fn if act_fn is not None else config.speech_encoder_hidden_act

        self.intermediate_dropout = nn.Dropout(p=dropout)
        self.intermediate_dense = nn.Linear(config.hidden_size, config.speech_encoder_intermediate_size)
        self.intermediate_act_fn = ACT2FN[act_fn] if isinstance(act_fn, str) else act_fn

        self.output_dense = nn.Linear(config.speech_encoder_intermediate_size, config.hidden_size)
        self.output_dropout = nn.Dropout(p=dropout)

    def forward(self, hidden_states):
        """
        Constructs the feed forward layer for the SeamlessM4TConformerFeedForward class.

        Args:
            self (SeamlessM4TConformerFeedForward): The instance of the SeamlessM4TConformerFeedForward class.
            hidden_states (tensor): The input hidden states to be processed by the feed forward layer.

        Returns:
            None.

        Raises:
            TypeError: If the input hidden_states is not a valid tensor.
            ValueError: If the input hidden_states is empty or has invalid shape.
            RuntimeError: If there is an issue during the feed forward layer forwardion process.
        """
        hidden_states = self.intermediate_dense(hidden_states)
        hidden_states = self.intermediate_act_fn(hidden_states)
        hidden_states = self.intermediate_dropout(hidden_states)

        hidden_states = self.output_dense(hidden_states)
        hidden_states = self.output_dropout(hidden_states)
        return hidden_states

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerFeedForward.__init__(config, act_fn=None, dropout=None)

Initializes the SeamlessM4TConformerFeedForward class.

PARAMETER	DESCRIPTION
self	The instance of the class.
config	An object containing configuration settings. Type: object Purpose: Holds various configuration parameters for the method. Restrictions: Must be provided as an argument.
act_fn	Activation function to be used. Type: str or callable, optional Purpose: Specifies the activation function to apply. Restrictions: If str, it must be a valid key in the ACT2FN dictionary. DEFAULT: None
dropout	Dropout rate to be applied. Type: float, optional Purpose: Controls the dropout rate for regularization. Restrictions: Must be a float between 0 and 1. If not provided, config.speech_encoder_dropout is used. DEFAULT: None

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def __init__(self, config, act_fn=None, dropout=None):
    """
    Initializes the SeamlessM4TConformerFeedForward class.

    Args:
        self: The instance of the class.
        config:
            An object containing configuration settings.

            - Type: object
            - Purpose: Holds various configuration parameters for the method.
            - Restrictions: Must be provided as an argument.
        act_fn:
            Activation function to be used.

            - Type: str or callable, optional
            - Purpose: Specifies the activation function to apply.
            - Restrictions: If str, it must be a valid key in the ACT2FN dictionary.
        dropout:
            Dropout rate to be applied.

            - Type: float, optional
            - Purpose: Controls the dropout rate for regularization.
            - Restrictions: Must be a float between 0 and 1. If not provided, config.speech_encoder_dropout is used.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    dropout = dropout if dropout is not None else config.speech_encoder_dropout
    act_fn = act_fn if act_fn is not None else config.speech_encoder_hidden_act

    self.intermediate_dropout = nn.Dropout(p=dropout)
    self.intermediate_dense = nn.Linear(config.hidden_size, config.speech_encoder_intermediate_size)
    self.intermediate_act_fn = ACT2FN[act_fn] if isinstance(act_fn, str) else act_fn

    self.output_dense = nn.Linear(config.speech_encoder_intermediate_size, config.hidden_size)
    self.output_dropout = nn.Dropout(p=dropout)

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerFeedForward.forward(hidden_states)

Constructs the feed forward layer for the SeamlessM4TConformerFeedForward class.

PARAMETER	DESCRIPTION
self	The instance of the SeamlessM4TConformerFeedForward class. TYPE: SeamlessM4TConformerFeedForward
hidden_states	The input hidden states to be processed by the feed forward layer. TYPE: tensor

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
TypeError	If the input hidden_states is not a valid tensor.
ValueError	If the input hidden_states is empty or has invalid shape.
RuntimeError	If there is an issue during the feed forward layer forwardion process.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def forward(self, hidden_states):
    """
    Constructs the feed forward layer for the SeamlessM4TConformerFeedForward class.

    Args:
        self (SeamlessM4TConformerFeedForward): The instance of the SeamlessM4TConformerFeedForward class.
        hidden_states (tensor): The input hidden states to be processed by the feed forward layer.

    Returns:
        None.

    Raises:
        TypeError: If the input hidden_states is not a valid tensor.
        ValueError: If the input hidden_states is empty or has invalid shape.
        RuntimeError: If there is an issue during the feed forward layer forwardion process.
    """
    hidden_states = self.intermediate_dense(hidden_states)
    hidden_states = self.intermediate_act_fn(hidden_states)
    hidden_states = self.intermediate_dropout(hidden_states)

    hidden_states = self.output_dense(hidden_states)
    hidden_states = self.output_dropout(hidden_states)
    return hidden_states

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerPositionalConvEmbedding

Bases: Module

A Python class representing a SeamlessM4TConformerPositionalConvEmbedding, which is used for positional convolutional embedding within a Conformer neural network model. This class inherits from nn.Module and includes functionality for applying convolution operations with specific configurations for padding and grouping.

ATTRIBUTE	DESCRIPTION
conv	nn.Conv1d A 1D convolutional layer with configurable kernel size, padding, and group settings.
padding	SeamlessM4TConformerSamePadLayer A layer for applying padding to the convolutional output based on specified parameters.
activation	function Activation function to be applied to the output of the convolutional layer.

METHOD	DESCRIPTION
__init__	Constructor method for initializing the SeamlessM4TConformerPositionalConvEmbedding instance.
forward	Method to perform the sequence of operations on the input hidden states, including convolution, padding, activation, and axis swapping.

Usage

Instantiate an object of SeamlessM4TConformerPositionalConvEmbedding with a configuration object and utilize the 'forward' method to process input hidden states.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

class SeamlessM4TConformerPositionalConvEmbedding(nn.Module):

    """
    A Python class representing a SeamlessM4TConformerPositionalConvEmbedding, which is used for positional
    convolutional embedding within a Conformer neural network model.
    This class inherits from nn.Module and includes functionality for applying convolution operations with specific
    configurations for padding and grouping.

    Attributes:
        conv: nn.Conv1d
            A 1D convolutional layer with configurable kernel size, padding, and group settings.

        padding: SeamlessM4TConformerSamePadLayer
            A layer for applying padding to the convolutional output based on specified parameters.

        activation: function
            Activation function to be applied to the output of the convolutional layer.

    Methods:
        __init__:
            Constructor method for initializing the SeamlessM4TConformerPositionalConvEmbedding instance.

        forward:
            Method to perform the sequence of operations on the input hidden states, including convolution,
            padding, activation, and axis swapping.

    Usage:
        Instantiate an object of SeamlessM4TConformerPositionalConvEmbedding with a configuration object and utilize
        the 'forward' method to process input hidden states.
    """
    def __init__(self, config):
        """
        Initialize the SeamlessM4TConformerPositionalConvEmbedding.

        Args:
            self (object): The instance of the class.
            config (object): Configuration object containing parameters for initializing the positional
                convolutional embedding.

                - hidden_size (int): The size of hidden units.
                - num_conv_pos_embeddings (int): The number of convolutional positional embeddings.
                - num_conv_pos_embedding_groups (int): The number of groups for the convolutional positional embeddings.
                - speech_encoder_hidden_act (str): The activation function for the speech encoder hidden layer.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.conv = nn.Conv1d(
            config.hidden_size,
            config.hidden_size,
            kernel_size=config.num_conv_pos_embeddings,
            padding=config.num_conv_pos_embeddings // 2,
            group=config.num_conv_pos_embedding_groups,
        )

        # self.conv = weight_norm(self.conv, name="weight", axis=2)

        self.padding = SeamlessM4TConformerSamePadLayer(config.num_conv_pos_embeddings)
        self.activation = ACT2FN[config.speech_encoder_hidden_act]

    def forward(self, hidden_states):
        """
        Constructs the positional convolutional embedding for the SeamlessM4TConformerPositionalConvEmbedding class.

        Args:
            self (SeamlessM4TConformerPositionalConvEmbedding):
                The instance of the SeamlessM4TConformerPositionalConvEmbedding class.
            hidden_states (numpy.ndarray):
                The input hidden states with shape (batch_size, sequence_length, hidden_size).

        Returns:
            None: The method modifies the hidden_states in place.

        Raises:
            ValueError: If the input hidden_states is not a numpy array.
            ValueError: If the input hidden_states does not have the correct shape
                (batch_size, sequence_length, hidden_size).
            TypeError: If the input hidden_states data type is not compatible with numpy array operations.
        """
        hidden_states = hidden_states.swapaxes(1, 2)

        hidden_states = self.conv(hidden_states)
        hidden_states = self.padding(hidden_states)
        hidden_states = self.activation(hidden_states)

        hidden_states = hidden_states.swapaxes(1, 2)
        return hidden_states

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerPositionalConvEmbedding.__init__(config)

Initialize the SeamlessM4TConformerPositionalConvEmbedding.

PARAMETER	DESCRIPTION
self	The instance of the class. TYPE: object
config	Configuration object containing parameters for initializing the positional convolutional embedding. hidden_size (int): The size of hidden units. num_conv_pos_embeddings (int): The number of convolutional positional embeddings. num_conv_pos_embedding_groups (int): The number of groups for the convolutional positional embeddings. speech_encoder_hidden_act (str): The activation function for the speech encoder hidden layer. TYPE: object

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def __init__(self, config):
    """
    Initialize the SeamlessM4TConformerPositionalConvEmbedding.

    Args:
        self (object): The instance of the class.
        config (object): Configuration object containing parameters for initializing the positional
            convolutional embedding.

            - hidden_size (int): The size of hidden units.
            - num_conv_pos_embeddings (int): The number of convolutional positional embeddings.
            - num_conv_pos_embedding_groups (int): The number of groups for the convolutional positional embeddings.
            - speech_encoder_hidden_act (str): The activation function for the speech encoder hidden layer.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.conv = nn.Conv1d(
        config.hidden_size,
        config.hidden_size,
        kernel_size=config.num_conv_pos_embeddings,
        padding=config.num_conv_pos_embeddings // 2,
        group=config.num_conv_pos_embedding_groups,
    )

    # self.conv = weight_norm(self.conv, name="weight", axis=2)

    self.padding = SeamlessM4TConformerSamePadLayer(config.num_conv_pos_embeddings)
    self.activation = ACT2FN[config.speech_encoder_hidden_act]

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerPositionalConvEmbedding.forward(hidden_states)

Constructs the positional convolutional embedding for the SeamlessM4TConformerPositionalConvEmbedding class.

PARAMETER	DESCRIPTION
self	The instance of the SeamlessM4TConformerPositionalConvEmbedding class. TYPE: SeamlessM4TConformerPositionalConvEmbedding
hidden_states	The input hidden states with shape (batch_size, sequence_length, hidden_size). TYPE: ndarray

RETURNS	DESCRIPTION
None	The method modifies the hidden_states in place.

RAISES	DESCRIPTION
ValueError	If the input hidden_states is not a numpy array.
ValueError	If the input hidden_states does not have the correct shape (batch_size, sequence_length, hidden_size).
TypeError	If the input hidden_states data type is not compatible with numpy array operations.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def forward(self, hidden_states):
    """
    Constructs the positional convolutional embedding for the SeamlessM4TConformerPositionalConvEmbedding class.

    Args:
        self (SeamlessM4TConformerPositionalConvEmbedding):
            The instance of the SeamlessM4TConformerPositionalConvEmbedding class.
        hidden_states (numpy.ndarray):
            The input hidden states with shape (batch_size, sequence_length, hidden_size).

    Returns:
        None: The method modifies the hidden_states in place.

    Raises:
        ValueError: If the input hidden_states is not a numpy array.
        ValueError: If the input hidden_states does not have the correct shape
            (batch_size, sequence_length, hidden_size).
        TypeError: If the input hidden_states data type is not compatible with numpy array operations.
    """
    hidden_states = hidden_states.swapaxes(1, 2)

    hidden_states = self.conv(hidden_states)
    hidden_states = self.padding(hidden_states)
    hidden_states = self.activation(hidden_states)

    hidden_states = hidden_states.swapaxes(1, 2)
    return hidden_states

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerRelPositionalEmbedding

Bases: Module

Relative positional encoding module.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

class SeamlessM4TConformerRelPositionalEmbedding(nn.Module):
    """Relative positional encoding module."""
    def __init__(self, config):
        """
        Initializes an instance of the SeamlessM4TConformerRelPositionalEmbedding class.

        Args:
            self: The instance of the class.
            config: An object of type 'Config' containing the configuration parameters.

        Returns:
            None

        Raises:
            None
        """
        super().__init__()
        self.max_len = config.max_source_positions
        self.d_model = config.hidden_size
        self.pe = None
        self.extend_pe(mindspore.tensor(0.0).expand(1, self.max_len)) # pylint: disable=too-many-function-args

    def extend_pe(self, x):
        """
        Extends the positional embeddings of the SeamlessM4TConformerRelPositionalEmbedding class.

        Args:
            self (SeamlessM4TConformerRelPositionalEmbedding):
                An instance of the SeamlessM4TConformerRelPositionalEmbedding class.
            x (Tensor): The input tensor to extend the positional embeddings.

        Returns:
            None: The method modifies the positional embeddings in-place.

        Raises:
            None.

        Description:
            This method extends the positional embeddings of the SeamlessM4TConformerRelPositionalEmbedding class
            based on the shape of the input tensor, 'x'. If the existing positional embeddings (pe) are already larger
            than or equal to twice the width of 'x', no modifications are made. If the data type of the positional
            embeddings is different from 'x', the positional embeddings are converted to the data type of 'x'.

            The method then calculates positive and negative positional encodings based on the shape of 'x'.
            The positional encodings are calculated using sine and cosine functions with a positional encoding
            matrix. The calculated positional encodings are flipped and concatenated to form the final positional
            embeddings, which are then assigned to the 'pe' attribute of the SeamlessM4TConformerRelPositionalEmbedding
            instance.
        """
        # Reset the positional encodings
        if self.pe is not None:
            # self.pe contains both positive and negative parts
            # the length of self.pe is 2 * input_len - 1
            if self.pe.shape[1] >= x.shape[1] * 2 - 1:
                if self.pe.dtype != x.dtype:
                    self.pe = self.pe.to(dtype=x.dtype)
                return
        # Suppose `i` is the position of query vector and `j` is the
        # position of key vector. We use positive relative positions when keys
        # are to the left (i>j) and negative relative positions otherwise (i<j).
        pe_positive = ops.zeros(x.shape[1], self.d_model)
        pe_negative = ops.zeros(x.shape[1], self.d_model)
        position = ops.arange(0, x.shape[1], dtype=mindspore.float32).unsqueeze(1)
        div_term = ops.exp(
            ops.arange(0, self.d_model, 2, dtype=mindspore.float32) * -(math.log(10000.0) / self.d_model)
        )
        pe_positive[:, 0::2] = ops.sin(position * div_term)
        pe_positive[:, 1::2] = ops.cos(position * div_term)
        pe_negative[:, 0::2] = ops.sin(-1 * position * div_term)
        pe_negative[:, 1::2] = ops.cos(-1 * position * div_term)

        # Reverse the order of positive indices and concat both positive and
        # negative indices. This is used to support the shifting trick
        # as in https://arxiv.org/abs/1901.02860
        pe_positive = ops.flip(pe_positive, [0]).unsqueeze(0)
        pe_negative = pe_negative[1:].unsqueeze(0)
        pe = ops.cat([pe_positive, pe_negative], axis=1)
        self.pe = pe.to(dtype=x.dtype)

    def forward(self, hidden_states: mindspore.Tensor):
        """
        Constructs the relative positional embeddings for the SeamlessM4TConformer model.

        Args:
            self (SeamlessM4TConformerRelPositionalEmbedding): An instance of the
                SeamlessM4TConformerRelPositionalEmbedding class.
            hidden_states (mindspore.Tensor): The hidden states of the model.

        Returns:
            mindspore.Tensor: The relative position embeddings for the given hidden states.

        Raises:
            None.

        Description:
            This method takes the hidden states of the model and forwards the relative position embeddings.
            It first extends the positional encodings (pe) using the extend_pe() method. Then, it calculates the
            start and end indices for selecting the relevant portion of the positional encodings based on the length
            of the hidden states. Finally, it returns the relative position embeddings for the given hidden states.

            The positional encodings are extended to ensure that there are sufficient embeddings to cover the entire
            sequence of hidden states. The start and end indices are calculated to select the relevant
            portion of the positional encodings that corresponds to the hidden states. This ensures that the relative
            position embeddings are aligned with the hidden states.

        Note:
            The relative position embeddings are used to capture the positional information between different elements
            in the hidden states. They help the model understand the relative positions of tokens in the input sequence,
            which is important for tasks such as machine translation.

        Example:
            ```python
            >>> rel_pos_emb = SeamlessM4TConformerRelPositionalEmbedding()
            >>> hidden_states = mindspore.Tensor(...)
            >>> relative_position_embeddings = rel_pos_emb.forward(hidden_states)
            ```
        """
        self.extend_pe(hidden_states)
        start_idx = self.pe.shape[1] // 2 - hidden_states.shape[1] + 1
        end_idx = self.pe.shape[1] // 2 + hidden_states.shape[1]
        relative_position_embeddings = self.pe[:, start_idx:end_idx]

        return relative_position_embeddings

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerRelPositionalEmbedding.__init__(config)

Initializes an instance of the SeamlessM4TConformerRelPositionalEmbedding class.

PARAMETER	DESCRIPTION
self	The instance of the class.
config	An object of type 'Config' containing the configuration parameters.

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def __init__(self, config):
    """
    Initializes an instance of the SeamlessM4TConformerRelPositionalEmbedding class.

    Args:
        self: The instance of the class.
        config: An object of type 'Config' containing the configuration parameters.

    Returns:
        None

    Raises:
        None
    """
    super().__init__()
    self.max_len = config.max_source_positions
    self.d_model = config.hidden_size
    self.pe = None
    self.extend_pe(mindspore.tensor(0.0).expand(1, self.max_len)) # pylint: disable=too-many-function-args

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerRelPositionalEmbedding.extend_pe(x)

Extends the positional embeddings of the SeamlessM4TConformerRelPositionalEmbedding class.

PARAMETER	DESCRIPTION
self	An instance of the SeamlessM4TConformerRelPositionalEmbedding class. TYPE: SeamlessM4TConformerRelPositionalEmbedding
x	The input tensor to extend the positional embeddings. TYPE: Tensor

RETURNS	DESCRIPTION
None	The method modifies the positional embeddings in-place.

Description

This method extends the positional embeddings of the SeamlessM4TConformerRelPositionalEmbedding class based on the shape of the input tensor, 'x'. If the existing positional embeddings (pe) are already larger than or equal to twice the width of 'x', no modifications are made. If the data type of the positional embeddings is different from 'x', the positional embeddings are converted to the data type of 'x'.

The method then calculates positive and negative positional encodings based on the shape of 'x'. The positional encodings are calculated using sine and cosine functions with a positional encoding matrix. The calculated positional encodings are flipped and concatenated to form the final positional embeddings, which are then assigned to the 'pe' attribute of the SeamlessM4TConformerRelPositionalEmbedding instance.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def extend_pe(self, x):
    """
    Extends the positional embeddings of the SeamlessM4TConformerRelPositionalEmbedding class.

    Args:
        self (SeamlessM4TConformerRelPositionalEmbedding):
            An instance of the SeamlessM4TConformerRelPositionalEmbedding class.
        x (Tensor): The input tensor to extend the positional embeddings.

    Returns:
        None: The method modifies the positional embeddings in-place.

    Raises:
        None.

    Description:
        This method extends the positional embeddings of the SeamlessM4TConformerRelPositionalEmbedding class
        based on the shape of the input tensor, 'x'. If the existing positional embeddings (pe) are already larger
        than or equal to twice the width of 'x', no modifications are made. If the data type of the positional
        embeddings is different from 'x', the positional embeddings are converted to the data type of 'x'.

        The method then calculates positive and negative positional encodings based on the shape of 'x'.
        The positional encodings are calculated using sine and cosine functions with a positional encoding
        matrix. The calculated positional encodings are flipped and concatenated to form the final positional
        embeddings, which are then assigned to the 'pe' attribute of the SeamlessM4TConformerRelPositionalEmbedding
        instance.
    """
    # Reset the positional encodings
    if self.pe is not None:
        # self.pe contains both positive and negative parts
        # the length of self.pe is 2 * input_len - 1
        if self.pe.shape[1] >= x.shape[1] * 2 - 1:
            if self.pe.dtype != x.dtype:
                self.pe = self.pe.to(dtype=x.dtype)
            return
    # Suppose `i` is the position of query vector and `j` is the
    # position of key vector. We use positive relative positions when keys
    # are to the left (i>j) and negative relative positions otherwise (i<j).
    pe_positive = ops.zeros(x.shape[1], self.d_model)
    pe_negative = ops.zeros(x.shape[1], self.d_model)
    position = ops.arange(0, x.shape[1], dtype=mindspore.float32).unsqueeze(1)
    div_term = ops.exp(
        ops.arange(0, self.d_model, 2, dtype=mindspore.float32) * -(math.log(10000.0) / self.d_model)
    )
    pe_positive[:, 0::2] = ops.sin(position * div_term)
    pe_positive[:, 1::2] = ops.cos(position * div_term)
    pe_negative[:, 0::2] = ops.sin(-1 * position * div_term)
    pe_negative[:, 1::2] = ops.cos(-1 * position * div_term)

    # Reverse the order of positive indices and concat both positive and
    # negative indices. This is used to support the shifting trick
    # as in https://arxiv.org/abs/1901.02860
    pe_positive = ops.flip(pe_positive, [0]).unsqueeze(0)
    pe_negative = pe_negative[1:].unsqueeze(0)
    pe = ops.cat([pe_positive, pe_negative], axis=1)
    self.pe = pe.to(dtype=x.dtype)

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerRelPositionalEmbedding.forward(hidden_states)

Constructs the relative positional embeddings for the SeamlessM4TConformer model.

PARAMETER	DESCRIPTION
self	An instance of the SeamlessM4TConformerRelPositionalEmbedding class. TYPE: SeamlessM4TConformerRelPositionalEmbedding
hidden_states	The hidden states of the model. TYPE: Tensor

RETURNS	DESCRIPTION
	mindspore.Tensor: The relative position embeddings for the given hidden states.

Description

This method takes the hidden states of the model and forwards the relative position embeddings. It first extends the positional encodings (pe) using the extend_pe() method. Then, it calculates the start and end indices for selecting the relevant portion of the positional encodings based on the length of the hidden states. Finally, it returns the relative position embeddings for the given hidden states.

The positional encodings are extended to ensure that there are sufficient embeddings to cover the entire sequence of hidden states. The start and end indices are calculated to select the relevant portion of the positional encodings that corresponds to the hidden states. This ensures that the relative position embeddings are aligned with the hidden states.

Note

The relative position embeddings are used to capture the positional information between different elements in the hidden states. They help the model understand the relative positions of tokens in the input sequence, which is important for tasks such as machine translation.

Example

>>> rel_pos_emb = SeamlessM4TConformerRelPositionalEmbedding()
>>> hidden_states = mindspore.Tensor(...)
>>> relative_position_embeddings = rel_pos_emb.forward(hidden_states)

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def forward(self, hidden_states: mindspore.Tensor):
    """
    Constructs the relative positional embeddings for the SeamlessM4TConformer model.

    Args:
        self (SeamlessM4TConformerRelPositionalEmbedding): An instance of the
            SeamlessM4TConformerRelPositionalEmbedding class.
        hidden_states (mindspore.Tensor): The hidden states of the model.

    Returns:
        mindspore.Tensor: The relative position embeddings for the given hidden states.

    Raises:
        None.

    Description:
        This method takes the hidden states of the model and forwards the relative position embeddings.
        It first extends the positional encodings (pe) using the extend_pe() method. Then, it calculates the
        start and end indices for selecting the relevant portion of the positional encodings based on the length
        of the hidden states. Finally, it returns the relative position embeddings for the given hidden states.

        The positional encodings are extended to ensure that there are sufficient embeddings to cover the entire
        sequence of hidden states. The start and end indices are calculated to select the relevant
        portion of the positional encodings that corresponds to the hidden states. This ensures that the relative
        position embeddings are aligned with the hidden states.

    Note:
        The relative position embeddings are used to capture the positional information between different elements
        in the hidden states. They help the model understand the relative positions of tokens in the input sequence,
        which is important for tasks such as machine translation.

    Example:
        ```python
        >>> rel_pos_emb = SeamlessM4TConformerRelPositionalEmbedding()
        >>> hidden_states = mindspore.Tensor(...)
        >>> relative_position_embeddings = rel_pos_emb.forward(hidden_states)
        ```
    """
    self.extend_pe(hidden_states)
    start_idx = self.pe.shape[1] // 2 - hidden_states.shape[1] + 1
    end_idx = self.pe.shape[1] // 2 + hidden_states.shape[1]
    relative_position_embeddings = self.pe[:, start_idx:end_idx]

    return relative_position_embeddings

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerRotaryPositionalEmbedding

Bases: Module

Rotary positional embedding Reference : https://blog.eleuther.ai/rotary-embeddings/ Paper: https://arxiv.org/pdf/2104.09864.pdf

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

class SeamlessM4TConformerRotaryPositionalEmbedding(nn.Module):
    """Rotary positional embedding
    Reference : https://blog.eleuther.ai/rotary-embeddings/ Paper: https://arxiv.org/pdf/2104.09864.pdf
    """
    def __init__(self, config):
        """
        __init__(self, config)

        Initialize the SeamlessM4TConformerRotaryPositionalEmbedding instance.

        Args:
            self: The instance of the SeamlessM4TConformerRotaryPositionalEmbedding class.
            config: A configuration object containing the parameters for the rotary positional embedding,
                including hidden_size and speech_encoder_attention_heads. It also includes the rotary_embedding_base
                used for calculating the inverse frequency. It is expected to be a valid configuration object.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        dim = config.hidden_size // config.speech_encoder_attention_heads
        base = config.rotary_embedding_base

        inv_freq = 1.0 / (base ** (ops.arange(0, dim, 2).float() / dim))
        self.inv_freq = inv_freq
        self.cached_sequence_length = None
        self.cached_rotary_positional_embedding = None

    def forward(self, hidden_states):
        """
        Constructs the rotary positional embeddings for the SeamlessM4TConformerRotaryPositionalEmbedding.

        Args:
            self: The instance of the SeamlessM4TConformerRotaryPositionalEmbedding class.
            hidden_states: A tensor representing the hidden states. It should have the shape
                (batch_size, sequence_length, hidden_size).

        Returns:
            None: The method updates the cached_rotary_positional_embedding attribute of the instance.

        Raises:
            None.
        """
        sequence_length = hidden_states.shape[1]

        if sequence_length == self.cached_sequence_length and self.cached_rotary_positional_embedding is not None:
            return self.cached_rotary_positional_embedding

        self.cached_sequence_length = sequence_length
        # Embeddings are computed in the dtype of the inv_freq constant
        time_stamps = ops.arange(sequence_length).type_as(self.inv_freq)
        freqs = ops.einsum("i,j->ij", time_stamps, self.inv_freq)
        embeddings = ops.cat((freqs, freqs), axis=-1)

        cos_embeddings = embeddings.cos()[:, None, None, :]
        sin_embeddings = embeddings.sin()[:, None, None, :]
        # Computed embeddings are cast to the dtype of the hidden state inputs
        self.cached_rotary_positional_embedding = ops.stack([cos_embeddings, sin_embeddings]).type_as(hidden_states)
        return self.cached_rotary_positional_embedding

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerRotaryPositionalEmbedding.__init__(config)

init(self, config)

Initialize the SeamlessM4TConformerRotaryPositionalEmbedding instance.

PARAMETER	DESCRIPTION
self	The instance of the SeamlessM4TConformerRotaryPositionalEmbedding class.
config	A configuration object containing the parameters for the rotary positional embedding, including hidden_size and speech_encoder_attention_heads. It also includes the rotary_embedding_base used for calculating the inverse frequency. It is expected to be a valid configuration object.

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def __init__(self, config):
    """
    __init__(self, config)

    Initialize the SeamlessM4TConformerRotaryPositionalEmbedding instance.

    Args:
        self: The instance of the SeamlessM4TConformerRotaryPositionalEmbedding class.
        config: A configuration object containing the parameters for the rotary positional embedding,
            including hidden_size and speech_encoder_attention_heads. It also includes the rotary_embedding_base
            used for calculating the inverse frequency. It is expected to be a valid configuration object.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    dim = config.hidden_size // config.speech_encoder_attention_heads
    base = config.rotary_embedding_base

    inv_freq = 1.0 / (base ** (ops.arange(0, dim, 2).float() / dim))
    self.inv_freq = inv_freq
    self.cached_sequence_length = None
    self.cached_rotary_positional_embedding = None

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerRotaryPositionalEmbedding.forward(hidden_states)

Constructs the rotary positional embeddings for the SeamlessM4TConformerRotaryPositionalEmbedding.

PARAMETER	DESCRIPTION
self	The instance of the SeamlessM4TConformerRotaryPositionalEmbedding class.
hidden_states	A tensor representing the hidden states. It should have the shape (batch_size, sequence_length, hidden_size).

RETURNS	DESCRIPTION
None	The method updates the cached_rotary_positional_embedding attribute of the instance.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def forward(self, hidden_states):
    """
    Constructs the rotary positional embeddings for the SeamlessM4TConformerRotaryPositionalEmbedding.

    Args:
        self: The instance of the SeamlessM4TConformerRotaryPositionalEmbedding class.
        hidden_states: A tensor representing the hidden states. It should have the shape
            (batch_size, sequence_length, hidden_size).

    Returns:
        None: The method updates the cached_rotary_positional_embedding attribute of the instance.

    Raises:
        None.
    """
    sequence_length = hidden_states.shape[1]

    if sequence_length == self.cached_sequence_length and self.cached_rotary_positional_embedding is not None:
        return self.cached_rotary_positional_embedding

    self.cached_sequence_length = sequence_length
    # Embeddings are computed in the dtype of the inv_freq constant
    time_stamps = ops.arange(sequence_length).type_as(self.inv_freq)
    freqs = ops.einsum("i,j->ij", time_stamps, self.inv_freq)
    embeddings = ops.cat((freqs, freqs), axis=-1)

    cos_embeddings = embeddings.cos()[:, None, None, :]
    sin_embeddings = embeddings.sin()[:, None, None, :]
    # Computed embeddings are cast to the dtype of the hidden state inputs
    self.cached_rotary_positional_embedding = ops.stack([cos_embeddings, sin_embeddings]).type_as(hidden_states)
    return self.cached_rotary_positional_embedding

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerSamePadLayer

Bases: Module

This class represents a seamless M4T Conformer layer with same padding.

Inherits from nn.Module.

ATTRIBUTE	DESCRIPTION
num_pad_remove	The number of padding elements to remove from the input sequence. TYPE: int

METHOD	DESCRIPTION
__init__	Initializes the SeamlessM4TConformerSamePadLayer instance.
forward	Constructs the hidden states of the SeamlessM4TConformerSamePadLayer.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

class SeamlessM4TConformerSamePadLayer(nn.Module):

    """
    This class represents a seamless M4T Conformer layer with same padding.

    Inherits from nn.Module.

    Attributes:
        num_pad_remove (int): The number of padding elements to remove from the input sequence.

    Methods:
        __init__: Initializes the SeamlessM4TConformerSamePadLayer instance.
        forward: Constructs the hidden states of the SeamlessM4TConformerSamePadLayer.

    """
    def __init__(self, num_conv_pos_embeddings):
        """
        Initializes an instance of the SeamlessM4TConformerSamePadLayer class.

        Args:
            self (SeamlessM4TConformerSamePadLayer): The current object instance.
            num_conv_pos_embeddings (int): The number of convolutional position embeddings.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.num_pad_remove = 1 if num_conv_pos_embeddings % 2 == 0 else 0

    def forward(self, hidden_states):
        """
        Constructs the hidden states by removing padding from the input tensor.

        Args:
            self (SeamlessM4TConformerSamePadLayer): An instance of the SeamlessM4TConformerSamePadLayer class.
            hidden_states (torch.Tensor):
                The input tensor containing hidden states.

                - Shape: (batch_size, sequence_length, hidden_size).
                - Purpose: Represents the hidden states to be processed.
                - Restrictions: None.

        Returns:
            None: The hidden states tensor with padding removed is modified in-place.

        Raises:
            None.
        """
        if self.num_pad_remove > 0:
            hidden_states = hidden_states[:, :, : -self.num_pad_remove]
        return hidden_states

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerSamePadLayer.__init__(num_conv_pos_embeddings)

Initializes an instance of the SeamlessM4TConformerSamePadLayer class.

PARAMETER	DESCRIPTION
self	The current object instance. TYPE: SeamlessM4TConformerSamePadLayer
num_conv_pos_embeddings	The number of convolutional position embeddings. TYPE: int

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def __init__(self, num_conv_pos_embeddings):
    """
    Initializes an instance of the SeamlessM4TConformerSamePadLayer class.

    Args:
        self (SeamlessM4TConformerSamePadLayer): The current object instance.
        num_conv_pos_embeddings (int): The number of convolutional position embeddings.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.num_pad_remove = 1 if num_conv_pos_embeddings % 2 == 0 else 0

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerSamePadLayer.forward(hidden_states)

Constructs the hidden states by removing padding from the input tensor.

PARAMETER	DESCRIPTION
self	An instance of the SeamlessM4TConformerSamePadLayer class. TYPE: SeamlessM4TConformerSamePadLayer
hidden_states	The input tensor containing hidden states. Shape: (batch_size, sequence_length, hidden_size). Purpose: Represents the hidden states to be processed. Restrictions: None. TYPE: Tensor

RETURNS	DESCRIPTION
None	The hidden states tensor with padding removed is modified in-place.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def forward(self, hidden_states):
    """
    Constructs the hidden states by removing padding from the input tensor.

    Args:
        self (SeamlessM4TConformerSamePadLayer): An instance of the SeamlessM4TConformerSamePadLayer class.
        hidden_states (torch.Tensor):
            The input tensor containing hidden states.

            - Shape: (batch_size, sequence_length, hidden_size).
            - Purpose: Represents the hidden states to be processed.
            - Restrictions: None.

    Returns:
        None: The hidden states tensor with padding removed is modified in-place.

    Raises:
        None.
    """
    if self.num_pad_remove > 0:
        hidden_states = hidden_states[:, :, : -self.num_pad_remove]
    return hidden_states

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerSelfAttention

Bases: Module

Construct a SeamlessM4TConformerSelfAttention object. Can be enhanced with rotary or relative position embeddings.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

class SeamlessM4TConformerSelfAttention(nn.Module):
    """Construct a SeamlessM4TConformerSelfAttention object.
    Can be enhanced with rotary or relative position embeddings.
    """
    def __init__(self, config, use_position_embeddings=True):
        """
        Initializes a new instance of the SeamlessM4TConformerSelfAttention class.

        Args:
            self: The object instance.
            config (Config): The configuration object.
            use_position_embeddings (bool, optional): Whether to use position embeddings. Default is True.

        Returns:
            None

        Raises:
            None
        """
        super().__init__()

        self.head_size = config.hidden_size // config.speech_encoder_attention_heads
        self.num_heads = config.speech_encoder_attention_heads
        self.position_embeddings_type = config.position_embeddings_type if use_position_embeddings else None

        self.linear_q = nn.Linear(config.hidden_size, config.hidden_size)
        self.linear_k = nn.Linear(config.hidden_size, config.hidden_size)
        self.linear_v = nn.Linear(config.hidden_size, config.hidden_size)
        self.linear_out = nn.Linear(config.hidden_size, config.hidden_size)

        self.dropout = nn.Dropout(p=config.speech_encoder_dropout)

        if self.position_embeddings_type == "relative":
            # linear transformation for positional encoding
            self.linear_pos = nn.Linear(config.hidden_size, config.hidden_size, bias=False)
            # these two learnable bias are used in matrix c and matrix d
            # as described in https://arxiv.org/abs/1901.02860 Section 3.3
            self.pos_bias_u = Parameter(ops.zeros(self.num_heads, self.head_size))
            self.pos_bias_v = Parameter(ops.zeros(self.num_heads, self.head_size))

    # Copied from transformers.models.wav2vec2_conformer.modeling_wav2vec2_conformer.Wav2Vec2ConformerSelfAttention.forward
    def forward(
        self,
        hidden_states: mindspore.Tensor,
        attention_mask: Optional[mindspore.Tensor] = None,
        relative_position_embeddings: Optional[mindspore.Tensor] = None,
        output_attentions: bool = False,
    ) -> Tuple[mindspore.Tensor, Optional[mindspore.Tensor], Optional[Tuple[mindspore.Tensor]]]:
        """
        Constructs the self-attention mechanism in the SeamlessM4TConformerSelfAttention class.

        Args:
            self (SeamlessM4TConformerSelfAttention): An instance of the SeamlessM4TConformerSelfAttention class.
            hidden_states (mindspore.Tensor): The input hidden states tensor of shape
                (batch_size, sequence_length, hidden_size).
            attention_mask (Optional[mindspore.Tensor]): An optional attention mask tensor of shape
                (batch_size, sequence_length, sequence_length), where each value is either 0 or 1. It is used to mask
                positions in the attention scores that should be ignored.
            relative_position_embeddings (Optional[mindspore.Tensor]): An optional tensor of shape
                (sequence_length, sequence_length, hidden_size) used for relative position embeddings. Required when
                self.position_embeddings_type is 'rotary' or 'relative'.
            output_attentions (bool): A flag indicating whether to output attention probabilities. Defaults to False.

        Returns:
            Tuple[mindspore.Tensor, Optional[mindspore.Tensor], Optional[Tuple[mindspore.Tensor]]]:
                A tuple containing:

                - hidden_states (mindspore.Tensor): The output hidden states tensor of shape
                (batch_size, sequence_length, hidden_size).
                - probs (Optional[mindspore.Tensor]): An optional tensor of shape
                (batch_size, num_heads, sequence_length, sequence_length) containing the attention probabilities.
                - None (Optional[Tuple[mindspore.Tensor]]): An optional tuple of attention weights tensors, each of
                shape (batch_size, num_heads, sequence_length, sequence_length). Only returned when output_attentions
                is True.

        Raises:
            ValueError: If self.position_embeddings_type is 'rotary' but relative_position_embeddings is not defined.
            ValueError: If self.position_embeddings_type is 'relative' but relative_position_embeddings is not defined.
        """
        # self-attention mechanism
        batch_size, _, _ = hidden_states.shape

        # make sure query/key states can be != value states
        query_key_states = hidden_states
        value_states = hidden_states

        if self.position_embeddings_type == "rotary":
            if relative_position_embeddings is None:
                raise ValueError(
                    "`relative_position_embeddings` has to be defined when `self.position_embeddings_type == 'rotary'"
                )
            query_key_states = self._apply_rotary_embedding(query_key_states, relative_position_embeddings)

        # project query_key_states and value_states
        query = self.linear_q(query_key_states).view(batch_size, -1, self.num_heads, self.head_size)
        key = self.linear_k(query_key_states).view(batch_size, -1, self.num_heads, self.head_size)
        value = self.linear_v(value_states).view(batch_size, -1, self.num_heads, self.head_size)

        # => (batch, head, time1, d_k)
        query = query.swapaxes(1, 2)
        key = key.swapaxes(1, 2)
        value = value.swapaxes(1, 2)

        if self.position_embeddings_type == "relative":
            if relative_position_embeddings is None:
                raise ValueError(
                    "`relative_position_embeddings` has to be defined when `self.position_embeddings_type =="
                    " 'relative'"
                )
            # apply relative_position_embeddings to qk scores
            # as proposed in Transformer_XL: https://arxiv.org/abs/1901.02860
            scores = self._apply_relative_embeddings(
                query=query, key=key, relative_position_embeddings=relative_position_embeddings
            )
        else:
            scores = ops.matmul(query, key.swapaxes(-2, -1)) / math.sqrt(self.head_size)

        # apply attention_mask if necessary
        if attention_mask is not None:
            scores = scores + attention_mask

        # => (batch, head, time1, time2)
        probs = ops.softmax(scores, axis=-1)
        probs = self.dropout(probs)

        # => (batch, head, time1, d_k)
        hidden_states = ops.matmul(probs, value)

        # => (batch, time1, hidden_size)
        hidden_states = hidden_states.swapaxes(1, 2).reshape(batch_size, -1, self.num_heads * self.head_size)
        hidden_states = self.linear_out(hidden_states)

        return hidden_states, probs

    # Copied from transformers.models.wav2vec2_conformer.modeling_wav2vec2_conformer.Wav2Vec2ConformerSelfAttention._apply_rotary_embedding
    def _apply_rotary_embedding(self, hidden_states, relative_position_embeddings):
        """
        Apply rotary embedding to the hidden states in the SeamlessM4TConformerSelfAttention class.

        Args:
            self: Reference to the instance of the class.
            hidden_states (torch.Tensor): A 3D tensor of shape (batch_size, sequence_length, _) representing the
                input hidden states.
            relative_position_embeddings (torch.Tensor): A 3D tensor of shape (2, sequence_length, ...) containing the
                relative position embeddings.

        Returns:
            torch.Tensor: A 3D tensor of shape (batch_size, sequence_length, self.num_heads * self.head_size)
                representing the modified hidden states after applying rotary embedding.

        Raises:
            None
        """
        batch_size, sequence_length, _ = hidden_states.shape
        hidden_states = hidden_states.view(batch_size, sequence_length, self.num_heads, self.head_size)

        cos = relative_position_embeddings[0, :sequence_length, ...]
        sin = relative_position_embeddings[1, :sequence_length, ...]

        # rotate hidden_states with rotary embeddings
        hidden_states = hidden_states.swapaxes(0, 1)
        rotated_states_begin = hidden_states[..., : self.head_size // 2]
        rotated_states_end = hidden_states[..., self.head_size // 2 :]
        rotated_states = ops.cat((-rotated_states_end, rotated_states_begin), axis=rotated_states_begin.ndim - 1)
        hidden_states = (hidden_states * cos) + (rotated_states * sin)
        hidden_states = hidden_states.swapaxes(0, 1)

        hidden_states = hidden_states.view(batch_size, sequence_length, self.num_heads * self.head_size)

        return hidden_states

    # Copied from transformers.models.wav2vec2_conformer.modeling_wav2vec2_conformer.Wav2Vec2ConformerSelfAttention._apply_relative_embeddings
    def _apply_relative_embeddings(self, query, key, relative_position_embeddings):
        """Apply relative embeddings to the given query and key.

        This method applies relative position embeddings to the query and key tensors in the
        SeamlessM4TConformerSelfAttention class.

        Args:
            self (SeamlessM4TConformerSelfAttention): The instance of the SeamlessM4TConformerSelfAttention class.
            query (Tensor): The query tensor.
            key (Tensor): The key tensor.
            relative_position_embeddings (Tensor): The tensor containing relative position embeddings.

        Returns:
            None.

        Raises:
            None.
        """
        # 1. project positional embeddings
        # => (batch, head, 2*time1-1, d_k)
        proj_relative_position_embeddings = self.linear_pos(relative_position_embeddings)
        proj_relative_position_embeddings = proj_relative_position_embeddings.view(
            relative_position_embeddings.shape[0], -1, self.num_heads, self.head_size
        )
        proj_relative_position_embeddings = proj_relative_position_embeddings.swapaxes(1, 2)
        proj_relative_position_embeddings = proj_relative_position_embeddings.swapaxes(2, 3)

        # 2. Add bias to query
        # => (batch, head, time1, d_k)
        query = query.swapaxes(1, 2)
        q_with_bias_u = (query + self.pos_bias_u).swapaxes(1, 2)
        q_with_bias_v = (query + self.pos_bias_v).swapaxes(1, 2)

        # 3. attention score: first compute matrix a and matrix c
        # as described in https://arxiv.org/abs/1901.02860 Section 3.3
        # => (batch, head, time1, time2)
        scores_ac = ops.matmul(q_with_bias_u, key.swapaxes(-2, -1))

        # 4. then compute matrix b and matrix d
        # => (batch, head, time1, 2*time1-1)
        scores_bd = ops.matmul(q_with_bias_v, proj_relative_position_embeddings)

        # 5. shift matrix b and matrix d
        zero_pad = ops.zeros((*scores_bd.shape[:3], 1), dtype=scores_bd.dtype)
        scores_bd_padded = ops.cat([zero_pad, scores_bd], axis=-1)
        scores_bd_padded_shape = scores_bd.shape[:2] + (scores_bd.shape[3] + 1, scores_bd.shape[2])
        scores_bd_padded = scores_bd_padded.view(*scores_bd_padded_shape)
        scores_bd = scores_bd_padded[:, :, 1:].view_as(scores_bd)
        scores_bd = scores_bd[:, :, :, : scores_bd.shape[-1] // 2 + 1]

        # 6. sum matrices
        # => (batch, head, time1, time2)
        scores = (scores_ac + scores_bd) / math.sqrt(self.head_size)

        return scores

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerSelfAttention.__init__(config, use_position_embeddings=True)

Initializes a new instance of the SeamlessM4TConformerSelfAttention class.

PARAMETER	DESCRIPTION
self	The object instance.
config	The configuration object. TYPE: Config
use_position_embeddings	Whether to use position embeddings. Default is True. TYPE: bool DEFAULT: True

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def __init__(self, config, use_position_embeddings=True):
    """
    Initializes a new instance of the SeamlessM4TConformerSelfAttention class.

    Args:
        self: The object instance.
        config (Config): The configuration object.
        use_position_embeddings (bool, optional): Whether to use position embeddings. Default is True.

    Returns:
        None

    Raises:
        None
    """
    super().__init__()

    self.head_size = config.hidden_size // config.speech_encoder_attention_heads
    self.num_heads = config.speech_encoder_attention_heads
    self.position_embeddings_type = config.position_embeddings_type if use_position_embeddings else None

    self.linear_q = nn.Linear(config.hidden_size, config.hidden_size)
    self.linear_k = nn.Linear(config.hidden_size, config.hidden_size)
    self.linear_v = nn.Linear(config.hidden_size, config.hidden_size)
    self.linear_out = nn.Linear(config.hidden_size, config.hidden_size)

    self.dropout = nn.Dropout(p=config.speech_encoder_dropout)

    if self.position_embeddings_type == "relative":
        # linear transformation for positional encoding
        self.linear_pos = nn.Linear(config.hidden_size, config.hidden_size, bias=False)
        # these two learnable bias are used in matrix c and matrix d
        # as described in https://arxiv.org/abs/1901.02860 Section 3.3
        self.pos_bias_u = Parameter(ops.zeros(self.num_heads, self.head_size))
        self.pos_bias_v = Parameter(ops.zeros(self.num_heads, self.head_size))

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TConformerSelfAttention.forward(hidden_states, attention_mask=None, relative_position_embeddings=None, output_attentions=False)

Constructs the self-attention mechanism in the SeamlessM4TConformerSelfAttention class.

PARAMETER	DESCRIPTION
self	An instance of the SeamlessM4TConformerSelfAttention class. TYPE: SeamlessM4TConformerSelfAttention
hidden_states	The input hidden states tensor of shape (batch_size, sequence_length, hidden_size). TYPE: Tensor
attention_mask	An optional attention mask tensor of shape (batch_size, sequence_length, sequence_length), where each value is either 0 or 1. It is used to mask positions in the attention scores that should be ignored. TYPE: Optional[Tensor] DEFAULT: None
relative_position_embeddings	An optional tensor of shape (sequence_length, sequence_length, hidden_size) used for relative position embeddings. Required when self.position_embeddings_type is 'rotary' or 'relative'. TYPE: Optional[Tensor] DEFAULT: None
output_attentions	A flag indicating whether to output attention probabilities. Defaults to False. TYPE: bool DEFAULT: False

RETURNS

DESCRIPTION

Tuple[Tensor, Optional[Tensor], Optional[Tuple[Tensor]]]

Tuple[mindspore.Tensor, Optional[mindspore.Tensor], Optional[Tuple[mindspore.Tensor]]]: A tuple containing: hidden_states (mindspore.Tensor): The output hidden states tensor of shape (batch_size, sequence_length, hidden_size). probs (Optional[mindspore.Tensor]): An optional tensor of shape (batch_size, num_heads, sequence_length, sequence_length) containing the attention probabilities. None (Optional[Tuple[mindspore.Tensor]]): An optional tuple of attention weights tensors, each of shape (batch_size, num_heads, sequence_length, sequence_length). Only returned when output_attentions is True.

RAISES	DESCRIPTION
ValueError	If self.position_embeddings_type is 'rotary' but relative_position_embeddings is not defined.
ValueError	If self.position_embeddings_type is 'relative' but relative_position_embeddings is not defined.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def forward(
    self,
    hidden_states: mindspore.Tensor,
    attention_mask: Optional[mindspore.Tensor] = None,
    relative_position_embeddings: Optional[mindspore.Tensor] = None,
    output_attentions: bool = False,
) -> Tuple[mindspore.Tensor, Optional[mindspore.Tensor], Optional[Tuple[mindspore.Tensor]]]:
    """
    Constructs the self-attention mechanism in the SeamlessM4TConformerSelfAttention class.

    Args:
        self (SeamlessM4TConformerSelfAttention): An instance of the SeamlessM4TConformerSelfAttention class.
        hidden_states (mindspore.Tensor): The input hidden states tensor of shape
            (batch_size, sequence_length, hidden_size).
        attention_mask (Optional[mindspore.Tensor]): An optional attention mask tensor of shape
            (batch_size, sequence_length, sequence_length), where each value is either 0 or 1. It is used to mask
            positions in the attention scores that should be ignored.
        relative_position_embeddings (Optional[mindspore.Tensor]): An optional tensor of shape
            (sequence_length, sequence_length, hidden_size) used for relative position embeddings. Required when
            self.position_embeddings_type is 'rotary' or 'relative'.
        output_attentions (bool): A flag indicating whether to output attention probabilities. Defaults to False.

    Returns:
        Tuple[mindspore.Tensor, Optional[mindspore.Tensor], Optional[Tuple[mindspore.Tensor]]]:
            A tuple containing:

            - hidden_states (mindspore.Tensor): The output hidden states tensor of shape
            (batch_size, sequence_length, hidden_size).
            - probs (Optional[mindspore.Tensor]): An optional tensor of shape
            (batch_size, num_heads, sequence_length, sequence_length) containing the attention probabilities.
            - None (Optional[Tuple[mindspore.Tensor]]): An optional tuple of attention weights tensors, each of
            shape (batch_size, num_heads, sequence_length, sequence_length). Only returned when output_attentions
            is True.

    Raises:
        ValueError: If self.position_embeddings_type is 'rotary' but relative_position_embeddings is not defined.
        ValueError: If self.position_embeddings_type is 'relative' but relative_position_embeddings is not defined.
    """
    # self-attention mechanism
    batch_size, _, _ = hidden_states.shape

    # make sure query/key states can be != value states
    query_key_states = hidden_states
    value_states = hidden_states

    if self.position_embeddings_type == "rotary":
        if relative_position_embeddings is None:
            raise ValueError(
                "`relative_position_embeddings` has to be defined when `self.position_embeddings_type == 'rotary'"
            )
        query_key_states = self._apply_rotary_embedding(query_key_states, relative_position_embeddings)

    # project query_key_states and value_states
    query = self.linear_q(query_key_states).view(batch_size, -1, self.num_heads, self.head_size)
    key = self.linear_k(query_key_states).view(batch_size, -1, self.num_heads, self.head_size)
    value = self.linear_v(value_states).view(batch_size, -1, self.num_heads, self.head_size)

    # => (batch, head, time1, d_k)
    query = query.swapaxes(1, 2)
    key = key.swapaxes(1, 2)
    value = value.swapaxes(1, 2)

    if self.position_embeddings_type == "relative":
        if relative_position_embeddings is None:
            raise ValueError(
                "`relative_position_embeddings` has to be defined when `self.position_embeddings_type =="
                " 'relative'"
            )
        # apply relative_position_embeddings to qk scores
        # as proposed in Transformer_XL: https://arxiv.org/abs/1901.02860
        scores = self._apply_relative_embeddings(
            query=query, key=key, relative_position_embeddings=relative_position_embeddings
        )
    else:
        scores = ops.matmul(query, key.swapaxes(-2, -1)) / math.sqrt(self.head_size)

    # apply attention_mask if necessary
    if attention_mask is not None:
        scores = scores + attention_mask

    # => (batch, head, time1, time2)
    probs = ops.softmax(scores, axis=-1)
    probs = self.dropout(probs)

    # => (batch, head, time1, d_k)
    hidden_states = ops.matmul(probs, value)

    # => (batch, time1, hidden_size)
    hidden_states = hidden_states.swapaxes(1, 2).reshape(batch_size, -1, self.num_heads * self.head_size)
    hidden_states = self.linear_out(hidden_states)

    return hidden_states, probs

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TDecoder

Bases: SeamlessM4TPreTrainedModel

SeamlessM4TDecoder

This class represents a decoder module for the SeamlessM4T model. It inherits from SeamlessM4TPreTrainedModel and implements methods for initializing the decoder, forwarding the decoder, and getting/setting input embeddings.

ATTRIBUTE	DESCRIPTION
config	An instance of SeamlessM4TConfig containing the configuration settings for the decoder.
dropout	The dropout rate specified in the configuration.
layerdrop	The layer drop rate specified in the configuration.
padding_idx	The padding token index specified in the configuration.
vocab_size	The size of the vocabulary specified in the configuration.
max_target_positions	The maximum target positions specified in the configuration.
embed_scale	The scaling factor for embedding specified in the configuration.
embed_tokens	An instance of nn.Embedding for embedding tokens.
embed_positions	An instance of SeamlessM4TSinusoidalPositionalEmbedding for embedding positions.
layers	A list of SeamlessM4TDecoderLayer instances representing the decoder layers.
layer_norm	An instance of nn.LayerNorm for layer normalization.
gradient_checkpointing	A boolean specifying whether gradient checkpointing is enabled.

METHOD	DESCRIPTION
__init__	Initializes the SeamlessM4TDecoder with the given configuration and embed_tokens.
get_input_embeddings	Returns the input embeddings.
set_input_embeddings	Sets the input embeddings for the decoder.
forward	Constructs the decoder with the given input and optional arguments.

PARAMETER	DESCRIPTION
input_ids	A mindspore.Tensor of shape (batch_size, sequence_length) representing input sequence token indices.
attention_mask	A mindspore.Tensor of shape (batch_size, sequence_length) representing attention mask to avoid padding tokens.
encoder_hidden_states	A mindspore.Tensor of shape (batch_size, encoder_sequence_length, hidden_size) representing hidden states of the encoder.
encoder_attention_mask	A mindspore.Tensor of shape (batch_size, encoder_sequence_length) representing attention mask for cross-attention.
past_key_values	A tuple of tuples of mindspore.Tensor representing pre-computed hidden-states for sequential decoding.
inputs_embeds	A mindspore.Tensor of shape (batch_size, sequence_length, hidden_size) representing embedded input representation.
use_cache	A boolean specifying whether to use cache for sequential decoding.
output_attentions	A boolean specifying whether to return attentions tensors of all attention layers.
output_hidden_states	A boolean specifying whether to return hidden states of all layers.
return_dict	A boolean specifying whether to return a ModelOutput instead of a plain tuple.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

class SeamlessM4TDecoder(SeamlessM4TPreTrainedModel):

    """
    SeamlessM4TDecoder

    This class represents a decoder module for the SeamlessM4T model. It inherits from SeamlessM4TPreTrainedModel and
    implements methods for initializing the decoder, forwarding the decoder, and getting/setting input embeddings.

    Attributes:
        config: An instance of SeamlessM4TConfig containing the configuration settings for the decoder.
        dropout: The dropout rate specified in the configuration.
        layerdrop: The layer drop rate specified in the configuration.
        padding_idx: The padding token index specified in the configuration.
        vocab_size: The size of the vocabulary specified in the configuration.
        max_target_positions: The maximum target positions specified in the configuration.
        embed_scale: The scaling factor for embedding specified in the configuration.
        embed_tokens: An instance of nn.Embedding for embedding tokens.
        embed_positions: An instance of SeamlessM4TSinusoidalPositionalEmbedding for embedding positions.
        layers: A list of SeamlessM4TDecoderLayer instances representing the decoder layers.
        layer_norm: An instance of nn.LayerNorm for layer normalization.
        gradient_checkpointing: A boolean specifying whether gradient checkpointing is enabled.

    Methods:
        __init__: Initializes the SeamlessM4TDecoder with the given configuration and embed_tokens.
        get_input_embeddings: Returns the input embeddings.
        set_input_embeddings: Sets the input embeddings for the decoder.
        forward: Constructs the decoder with the given input and optional arguments.

    Args:
        input_ids: A mindspore.Tensor of shape (batch_size, sequence_length) representing input sequence token indices.
        attention_mask: A mindspore.Tensor of shape (batch_size, sequence_length) representing attention mask to avoid
            padding tokens.
        encoder_hidden_states: A mindspore.Tensor of shape (batch_size, encoder_sequence_length, hidden_size)
            representing hidden states of the encoder.
        encoder_attention_mask: A mindspore.Tensor of shape (batch_size, encoder_sequence_length) representing
            attention mask for cross-attention.
        past_key_values: A tuple of tuples of mindspore.Tensor representing pre-computed hidden-states for sequential
            decoding.
        inputs_embeds: A mindspore.Tensor of shape (batch_size, sequence_length, hidden_size) representing embedded
            input representation.
        use_cache: A boolean specifying whether to use cache for sequential decoding.
        output_attentions: A boolean specifying whether to return attentions tensors of all attention layers.
        output_hidden_states: A boolean specifying whether to return hidden states of all layers.
        return_dict: A boolean specifying whether to return a ModelOutput instead of a plain tuple.
    """
    def __init__(
        self,
        config: SeamlessM4TConfig,
        embed_tokens: Optional[nn.Embedding] = None,
    ):
        """
        Initializes an instance of the 'SeamlessM4TDecoder' class.

        Args:
            self: An instance of the 'SeamlessM4TDecoder' class.
            config (SeamlessM4TConfig): An object containing configuration options for the decoder.
            embed_tokens (Optional[nn.Embedding]): An optional embedding object to be used for token embeddings.

        Returns:
            None

        Raises:
            None

        This method initializes the 'SeamlessM4TDecoder' instance by setting various attributes and creating necessary
        objects. It takes the following parameters:

        - self: An instance of the 'SeamlessM4TDecoder' class.
        - config (SeamlessM4TConfig): An object that holds configuration options for the decoder.
        It provides access to various hyperparameters and settings.
        - embed_tokens (Optional[nn.Embedding]): An optional embedding object that can be used for token embeddings.
        If provided, the 'embed_tokens' attribute of the decoder will be set to this object. Otherwise, a new embedding
        object will be created using the 'vocab_size' and 'hidden_size' from the 'config' object.

        Note:
            The 'config' parameter is mandatory, while the 'embed_tokens' parameter is optional.

        The method performs the following actions:

        1. Calls the superclass '__init__' method with the 'config' parameter.
        2. Sets the 'dropout' attribute to the 'dropout' value from the 'config' object.
        3. Sets the 'layerdrop' attribute to the 'decoder_layerdrop' value from the 'config' object.
        4. Sets the 'padding_idx' attribute to the 'pad_token_id' value from the 'config' object.
        5. Sets the 'vocab_size' attribute to the 'vocab_size' value from the 'config' object.
        6. Sets the 'max_target_positions' attribute to the 'max_position_embeddings' value from the 'config' object.
        7. Sets the 'embed_scale' attribute based on the 'scale_embedding' value from the 'config' object.
        If 'scale_embedding' is True, it sets 'embed_scale' to the square root of 'hidden_size'; otherwise, it
        sets 'embed_scale' to 1.0.
        8. If 'embed_tokens' is not None:

            - Creates a new 'nn.Embedding' object named 'self.embed_tokens' with 'embed_tokens.vocab_size',
            'embed_tokens.embedding_size', and 'self.padding_idx' as arguments.
            - Sets the weight of 'self.embed_tokens' to the weight of 'embed_tokens'.
        9. If 'embed_tokens' is None:

            - Creates a new 'nn.Embedding' object named 'self.embed_tokens' with 'self.vocab_size', 'config.hidden_size',
            and 'self.padding_idx' as arguments.
        10. Creates a 'SeamlessM4TSinusoidalPositionalEmbedding' object named 'self.embed_positions' with
        'self.max_target_positions', 'config.hidden_size', and 'self.padding_idx' as arguments.
        11. Creates a list named 'layers'.
        12. Iterates 'config.decoder_layers' times and appends a 'SeamlessM4TDecoderLayer' object to 'layers',
        using 'config', 'config.decoder_attention_heads', and 'config.decoder_ffn_dim' as arguments.
        13. Sets the 'layers' attribute to a 'nn.ModuleList' containing the objects in 'layers'.
        14. Creates a 'nn.LayerNorm' object named 'self.layer_norm' with a list containing 'config.hidden_size'
        as the argument.
        15. Sets the 'gradient_checkpointing' attribute to False.
        16. Calls the 'post_init' method.

        Note: The 'post_init' method is not defined in the given code snippet.
        """
        super().__init__(config)
        self.dropout = config.dropout
        self.layerdrop = config.decoder_layerdrop
        self.padding_idx = config.pad_token_id
        self.vocab_size = config.vocab_size
        self.max_target_positions = config.max_position_embeddings
        self.embed_scale = math.sqrt(config.hidden_size) if config.scale_embedding else 1.0

        if embed_tokens is not None:
            # if embed_tokens defined, use its shape instead
            self.embed_tokens = nn.Embedding(embed_tokens.vocab_size, embed_tokens.embedding_size, self.padding_idx)
            self.embed_tokens.weight = embed_tokens.weight
        else:
            self.embed_tokens = nn.Embedding(self.vocab_size, config.hidden_size, self.padding_idx)

        self.embed_positions = SeamlessM4TSinusoidalPositionalEmbedding(
            self.max_target_positions,
            config.hidden_size,
            padding_idx=self.padding_idx,
        )

        layers = []
        for _ in range(config.decoder_layers):
            layers.append(
                SeamlessM4TDecoderLayer(
                    config,
                    decoder_attention_heads=config.decoder_attention_heads,
                    decoder_ffn_dim=config.decoder_ffn_dim,
                )
            )
        self.layers = nn.ModuleList(layers)
        self.layer_norm = nn.LayerNorm([config.hidden_size])

        self.gradient_checkpointing = False
        # Initialize weights and apply final processing
        self.post_init()

    def get_input_embeddings(self):
        """
        Retrieve the input embeddings from the SeamlessM4TDecoder.

        Args:
            self (SeamlessM4TDecoder): An instance of the SeamlessM4TDecoder class.

        Returns:
            None.

        Raises:
            None.
        """
        return self.embed_tokens

    def set_input_embeddings(self, value):
        """
        Sets the input embeddings for the SeamlessM4TDecoder.

        Args:
            self (SeamlessM4TDecoder): The instance of SeamlessM4TDecoder.
            value: The input embeddings to be set.

        Returns:
            None.

        Raises:
            None.
        """
        self.embed_tokens = value

    def forward(
        self,
        input_ids: mindspore.Tensor = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        encoder_hidden_states: Optional[mindspore.Tensor] = None,
        encoder_attention_mask: Optional[mindspore.Tensor] = None,
        past_key_values: Optional[Tuple[Tuple[mindspore.Tensor]]] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        use_cache: Optional[bool] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, BaseModelOutputWithPastAndCrossAttentions]:
        r"""
        Args:
            input_ids (`mindspore.Tensor` of shape `(batch_size, sequence_length)`):
                Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you
                provide it.

                Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
                [`PreTrainedTokenizer.__call__`] for details.

                [What are input IDs?](../glossary#input-ids)
            attention_mask (`mindspore.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
                Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:

                - 1 for tokens that are **not masked**,
                - 0 for tokens that are **masked**.

                [What are attention masks?](../glossary#attention-mask)
            encoder_hidden_states (`mindspore.Tensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*):
                Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention
                of the decoder.
            encoder_attention_mask (`mindspore.Tensor` of shape `(batch_size, encoder_sequence_length)`, *optional*):
                Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values
                selected in `[0, 1]`:

                - 1 for tokens that are **not masked**,
                - 0 for tokens that are **masked**.

                [What are attention masks?](../glossary#attention-mask)
            past_key_values (`tuple(tuple(mindspore.Tensor))`, *optional*, returned when `use_cache=True` is passed
                or when `config.use_cache=True`):
                Tuple of `tuple(mindspore.Tensor)` of length `config.n_layers`, with each tuple having 2 tensors of
                shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of
                shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`.

                Contains pre-computed hidden-states (key and values in the self-attention blocks and in the
                cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding.

                If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those
                that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of
                all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`mindspore.Tensor` of
                shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing
                `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more
                control over how to convert `input_ids` indices into associated vectors than the model's internal
                embedding lookup matrix.
            output_attentions (`bool`, *optional*):
                Whether or not to return the attentions tensors of all attention layers. See `attentions` under
                returned tensors for more detail.
            output_hidden_states (`bool`, *optional*):
                Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors
                for more detail.
            return_dict (`bool`, *optional*):
                Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
        """
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )
        use_cache = use_cache if use_cache is not None else self.config.use_cache
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        # retrieve input_ids and inputs_embeds
        if input_ids is not None and inputs_embeds is not None:
            raise ValueError("You cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time")
        elif input_ids is not None:
            input = input_ids
            input_shape = input.shape
            input_ids = input_ids.view(-1, input_shape[-1])
        elif inputs_embeds is not None:
            input_shape = inputs_embeds.shape[:-1]
            input = inputs_embeds[:, :, -1]
        else:
            raise ValueError("You have to specify either decoder_input_ids or decoder_inputs_embeds")

        # past_key_values_length
        past_key_values_length = past_key_values[0][0].shape[2] if past_key_values is not None else 0

        if inputs_embeds is None:
            inputs_embeds = self.embed_tokens(input_ids) * self.embed_scale

        attention_mask = _prepare_4d_causal_attention_mask(
            attention_mask, input_shape, inputs_embeds, past_key_values_length
        )

        # expand encoder attention mask
        if encoder_hidden_states is not None and encoder_attention_mask is not None:
            # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]
            encoder_attention_mask = _prepare_4d_attention_mask(
                encoder_attention_mask, inputs_embeds.dtype, tgt_len=input_shape[-1]
            )

        # embed positions
        positions = self.embed_positions(input, past_key_values_length=past_key_values_length)

        hidden_states = inputs_embeds + positions

        hidden_states = ops.dropout(hidden_states, p=self.dropout, training=self.training)

        if self.gradient_checkpointing and self.training:
            if use_cache:
                logger.warning_once(
                    "`use_cache=True` is incompatible with gradient checkpointing`. Setting `use_cache=False`..."
                )
                use_cache = False

        # decoder layers
        all_hidden_states = () if output_hidden_states else None
        all_self_attns = () if output_attentions else None
        all_cross_attentions = () if (output_attentions and encoder_hidden_states is not None) else None
        next_decoder_cache = () if use_cache else None

        for idx, decoder_layer in enumerate(self.layers):
            # add LayerDrop (see https://arxiv.org/abs/1909.11556 for description)
            if output_hidden_states:
                all_hidden_states += (hidden_states,)
            if self.training:
                dropout_probability = ops.rand([])
                if dropout_probability < self.layerdrop:
                    continue

            past_key_value = past_key_values[idx] if past_key_values is not None else None

            layer_outputs = decoder_layer(
                hidden_states,
                attention_mask=attention_mask,
                encoder_hidden_states=encoder_hidden_states,
                encoder_attention_mask=encoder_attention_mask,
                past_key_value=past_key_value,
                output_attentions=output_attentions,
                use_cache=use_cache,
            )
            hidden_states = layer_outputs[0]

            if use_cache:
                next_decoder_cache += (layer_outputs[1],)

            if output_attentions:
                all_self_attns += (layer_outputs[2],)

                if encoder_hidden_states is not None:
                    all_cross_attentions += (layer_outputs[3],)

        hidden_states = self.layer_norm(hidden_states)

        # add hidden states from the last decoder layer
        if output_hidden_states:
            all_hidden_states += (hidden_states,)

        next_cache = next_decoder_cache if use_cache else None
        if not return_dict:
            return tuple(
                v
                for v in [hidden_states, next_cache, all_hidden_states, all_self_attns, all_cross_attentions]
                if v is not None
            )
        return BaseModelOutputWithPastAndCrossAttentions(
            last_hidden_state=hidden_states,
            past_key_values=next_cache,
            hidden_states=all_hidden_states,
            attentions=all_self_attns,
            cross_attentions=all_cross_attentions,
        )

mindnlp.transformers.models.seamless_m4t.modeling_seamless_m4t.SeamlessM4TDecoder.__init__(config, embed_tokens=None)

Initializes an instance of the 'SeamlessM4TDecoder' class.

PARAMETER	DESCRIPTION
self	An instance of the 'SeamlessM4TDecoder' class.
config	An object containing configuration options for the decoder. TYPE: SeamlessM4TConfig
embed_tokens	An optional embedding object to be used for token embeddings. TYPE: Optional[Embedding] DEFAULT: None

RETURNS	DESCRIPTION
	None

This method initializes the 'SeamlessM4TDecoder' instance by setting various attributes and creating necessary objects. It takes the following parameters:

• self: An instance of the 'SeamlessM4TDecoder' class.
• config (SeamlessM4TConfig): An object that holds configuration options for the decoder. It provides access to various hyperparameters and settings.
• embed_tokens (Optional[nn.Embedding]): An optional embedding object that can be used for token embeddings. If provided, the 'embed_tokens' attribute of the decoder will be set to this object. Otherwise, a new embedding object will be created using the 'vocab_size' and 'hidden_size' from the 'config' object.

Note

The 'config' parameter is mandatory, while the 'embed_tokens' parameter is optional.

The method performs the following actions:

1. Calls the superclass 'init' method with the 'config' parameter.
2. Sets the 'dropout' attribute to the 'dropout' value from the 'config' object.
3. Sets the 'layerdrop' attribute to the 'decoder_layerdrop' value from the 'config' object.
4. Sets the 'padding_idx' attribute to the 'pad_token_id' value from the 'config' object.
5. Sets the 'vocab_size' attribute to the 'vocab_size' value from the 'config' object.
6. Sets the 'max_target_positions' attribute to the 'max_position_embeddings' value from the 'config' object.
7. Sets the 'embed_scale' attribute based on the 'scale_embedding' value from the 'config' object. If 'scale_embedding' is True, it sets 'embed_scale' to the square root of 'hidden_size'; otherwise, it sets 'embed_scale' to 1.0.

8. If 'embed_tokens' is not None:

   • Creates a new 'nn.Embedding' object named 'self.embed_tokens' with 'embed_tokens.vocab_size', 'embed_tokens.embedding_size', and 'self.padding_idx' as arguments.

   • Sets the weight of 'self.embed_tokens' to the weight of 'embed_tokens'. 9. If 'embed_tokens' is None:

   • Creates a new 'nn.Embedding' object named 'self.embed_tokens' with 'self.vocab_size', 'config.hidden_size', and 'self.padding_idx' as arguments. 10. Creates a 'SeamlessM4TSinusoidalPositionalEmbedding' object named 'self.embed_positions' with 'self.max_target_positions', 'config.hidden_size', and 'self.padding_idx' as arguments. 11. Creates a list named 'layers'. 12. Iterates 'config.decoder_layers' times and appends a 'SeamlessM4TDecoderLayer' object to 'layers', using 'config', 'config.decoder_attention_heads', and 'config.decoder_ffn_dim' as arguments. 13. Sets the 'layers' attribute to a 'nn.ModuleList' containing the objects in 'layers'. 14. Creates a 'nn.LayerNorm' object named 'self.layer_norm' with a list containing 'config.hidden_size' as the argument. 15. Sets the 'gradient_checkpointing' attribute to False. 16. Calls the 'post_init' method.

Note: The 'post_init' method is not defined in the given code snippet.

Source code in mindnlp/transformers/models/seamless_m4t/modeling_seamless_m4t.py

def __init__(
    self,
    config: SeamlessM4TConfig,
    embed_tokens: Optional[nn.Embedding] = None,
):
    """
    Initializes an instance of the 'SeamlessM4TDecoder' class.

    Args:
        self: An instance of the 'SeamlessM4TDecoder' class.
        config (SeamlessM4TConfig): An object containing configuration options for the decoder.
        embed_tokens (Optional[nn.Embedding]): An optional embedding object to be used for token embeddings.

    Returns:
        None

    Raises:
        None

    This method initializes the 'SeamlessM4TDecoder' instance by setting various attributes and creating necessary
    objects. It takes the following parameters:

    - self: An instance of the 'SeamlessM4TDecoder' class.
    - config (SeamlessM4TConfig): An object that holds configuration options for the decoder.
    It provides access to various hyperparameters and settings.
    - embed_tokens (Optional[nn.Embedding]): An optional embedding object that can be used for token embeddings.
    If provided, the 'embed_tokens' attribute of the decoder will be set to this object. Otherwise, a new embedding
    object will be created using the 'vocab_size' and 'hidden_size' from the 'config' object.

    Note:
        The 'config' parameter is mandatory, while the 'embed_tokens' parameter is optional.

    The method performs the following actions:

    1. Calls the superclass '__init__' method with the 'config' parameter.
    2. Sets the 'dropout' attribute to the 'dropout' value from the 'config' object.
    3. Sets the 'layerdrop' attribute to the 'decoder_layerdrop' value from the 'config' object.
    4. Sets the 'padding_idx' attribute to the 'pad_token_id' value from the 'config' object.
    5. Sets the 'vocab_size' attribute to the 'vocab_size' value from the 'config' object.
    6. Sets the 'max_target_positions' attribute to the 'max_position_embeddings' value from the 'config' object.
    7. Sets the 'embed_scale' attribute based on the 'scale_embedding' value from the 'config' object.
    If 'scale_embedding' is True, it sets 'embed_scale' to the square root of 'hidden_size'; otherwise, it
    sets 'embed_scale' to 1.0.
    8. If 'embed_tokens' is not None:

        - Creates a new 'nn.Embedding' object named 'self.embed_tokens' with 'embed_tokens.vocab_size',