hubert

mindnlp.transformers.models.hubert.configuration_hubert

Hubert model configuration

mindnlp.transformers.models.hubert.configuration_hubert.HubertConfig

Bases: PretrainedConfig

This is the configuration class to store the configuration of a [HubertModel]. It is used to instantiate an Hubert model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Hubert facebook/hubert-base-ls960 architecture.

Configuration objects inherit from [PretrainedConfig] and can be used to control the model outputs. Read the documentation from [PretrainedConfig] for more information.

PARAMETER	DESCRIPTION
vocab_size	Vocabulary size of the Hubert model. Defines the number of different tokens that can be represented by the inputs_ids passed when calling [HubertModel]. Vocabulary size of the model. Defines the different tokens that can be represented by the inputs_ids passed to the forward method of [HubertModel]. TYPE: `int`, *optional*, defaults to 32 DEFAULT: 32
hidden_size	Dimensionality of the encoder layers and the pooler layer. TYPE: `int`, *optional*, defaults to 768 DEFAULT: 768
num_hidden_layers	Number of hidden layers in the Transformer encoder. TYPE: `int`, *optional*, defaults to 12 DEFAULT: 12
num_attention_heads	Number of attention heads for each attention layer in the Transformer encoder. TYPE: `int`, *optional*, defaults to 12 DEFAULT: 12
intermediate_size	Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. TYPE: `int`, *optional*, defaults to 3072 DEFAULT: 3072
hidden_act	The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu", "relu", "selu" and "gelu_new" are supported. TYPE: `str` or `function`, *optional*, defaults to `"gelu"` DEFAULT: 'gelu'
hidden_dropout(`float`,	The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. TYPE: *optional*, defaults to 0.1
activation_dropout	The dropout ratio for activations inside the fully connected layer. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
attention_dropout(`float`,	The dropout ratio for the attention probabilities. TYPE: *optional*, defaults to 0.1
final_dropout	The dropout probabilitiy for the final projection layer of [Wav2Vec2ForCTC]. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
layerdrop	The LayerDrop probability. See the LayerDrop paper for more details. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
initializer_range	The standard deviation of the truncated_normal_initializer for initializing all weight matrices. TYPE: `float`, *optional*, defaults to 0.02 DEFAULT: 0.02
layer_norm_eps	The epsilon used by the layer normalization layers. TYPE: `float`, *optional*, defaults to 1e-12 DEFAULT: 1e-05
feat_extract_norm	The norm to be applied to 1D convolutional layers in feature encoder. One of "group" for group normalization of only the first 1D convolutional layer or "layer" for layer normalization of all 1D convolutional layers. TYPE: `str`, *optional*, defaults to `"group"` DEFAULT: 'group'
feat_proj_dropout	The dropout probability for output of the feature encoder. TYPE: `float`, *optional*, defaults to 0.0 DEFAULT: 0.0
feat_proj_layer_norm	Whether to apply LayerNorm to the output of the feature encoder. TYPE: `bool`, *optional*, defaults to `True` DEFAULT: True
feat_extract_activation	The non-linear activation function (function or string) in the 1D convolutional layers of the feature extractor. If string, "gelu", "relu", "selu" and "gelu_new" are supported. TYPE: `str, `optional`, defaults to `"gelu"` DEFAULT: 'gelu'
conv_dim	A tuple of integers defining the number of input and output channels of each 1D convolutional layer in the feature encoder. The length of conv_dim defines the number of 1D convolutional layers. TYPE: `Tuple[int]`, *optional*, defaults to `(512, 512, 512, 512, 512, 512, 512)` DEFAULT: (512, 512, 512, 512, 512, 512, 512)
conv_stride	A tuple of integers defining the stride of each 1D convolutional layer in the feature encoder. The length of conv_stride defines the number of convolutional layers and has to match the length of conv_dim. TYPE: `Tuple[int]`, *optional*, defaults to `(5, 2, 2, 2, 2, 2, 2)` DEFAULT: (5, 2, 2, 2, 2, 2, 2)
conv_kernel	A tuple of integers defining the kernel size of each 1D convolutional layer in the feature encoder. The length of conv_kernel defines the number of convolutional layers and has to match the length of conv_dim. TYPE: `Tuple[int]`, *optional*, defaults to `(10, 3, 3, 3, 3, 3, 3)` DEFAULT: (10, 3, 3, 3, 3, 2, 2)
conv_bias	Whether the 1D convolutional layers have a bias. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
num_conv_pos_embeddings	Number of convolutional positional embeddings. Defines the kernel size of 1D convolutional positional embeddings layer. TYPE: `int`, *optional*, defaults to 128 DEFAULT: 128
num_conv_pos_embedding_groups	Number of groups of 1D convolutional positional embeddings layer. TYPE: `int`, *optional*, defaults to 16 DEFAULT: 16
do_stable_layer_norm	Whether do apply stable layer norm architecture of the Transformer encoder. do_stable_layer_norm is True corresponds to applying layer norm before the attention layer, whereas do_stable_layer_norm is False corresponds to applying layer norm after the attention layer. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
apply_spec_augment	Whether to apply SpecAugment data augmentation to the outputs of the feature encoder. For reference see SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition. TYPE: `bool`, *optional*, defaults to `True` DEFAULT: True
mask_time_prob	Percentage (between 0 and 1) of all feature vectors along the time axis which will be masked. The masking procecure generates ''mask_time_prob*len(time_axis)/mask_time_length'' independent masks over the axis. If reasoning from the propability of each feature vector to be chosen as the start of the vector span to be masked, mask_time_prob should be prob_vector_start*mask_time_length. Note that overlap may decrease the actual percentage of masked vectors. This is only relevant if apply_spec_augment is True. TYPE: `float`, *optional*, defaults to 0.05 DEFAULT: 0.05
mask_time_length	Length of vector span along the time axis. TYPE: `int`, *optional*, defaults to 10 DEFAULT: 10
mask_time_min_masks	The minimum number of masks of length mask_feature_length generated along the time axis, each time step, irrespectively of mask_feature_prob. Only relevant if ''mask_time_prob*len(time_axis)/mask_time_length < mask_time_min_masks'' TYPE: `int`, *optional*, defaults to 2), DEFAULT: 2
mask_feature_prob	Percentage (between 0 and 1) of all feature vectors along the feature axis which will be masked. The masking procecure generates ''mask_feature_prob*len(feature_axis)/mask_time_length'' independent masks over the axis. If reasoning from the propability of each feature vector to be chosen as the start of the vector span to be masked, mask_feature_prob should be prob_vector_start*mask_feature_length. Note that overlap may decrease the actual percentage of masked vectors. This is only relevant if apply_spec_augment is True. TYPE: `float`, *optional*, defaults to 0.0 DEFAULT: 0.0
mask_feature_length	Length of vector span along the feature axis. TYPE: `int`, *optional*, defaults to 10 DEFAULT: 10
mask_feature_min_masks	The minimum number of masks of length mask_feature_length generated along the feature axis, each time step, irrespectively of mask_feature_prob. Only relevant if ''mask_feature_prob*len(feature_axis)/mask_feature_length < mask_feature_min_masks'' TYPE: `int`, *optional*, defaults to 0), DEFAULT: 0
ctc_loss_reduction	Specifies the reduction to apply to the output of torch.nn.CTCLoss. Only relevant when training an instance of [HubertForCTC]. TYPE: `str`, *optional*, defaults to `"sum"` DEFAULT: 'sum'
ctc_zero_infinity	Whether to zero infinite losses and the associated gradients of torch.nn.CTCLoss. Infinite losses mainly occur when the inputs are too short to be aligned to the targets. Only relevant when training an instance of [HubertForCTC]. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
use_weighted_layer_sum	Whether to use a weighted average of layer outputs with learned weights. Only relevant when using an instance of [HubertForSequenceClassification]. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
classifier_proj_size	Dimensionality of the projection before token mean-pooling for classification. TYPE: `int`, *optional*, defaults to 256 DEFAULT: 256

Example

>>> from transformers import HubertModel, HubertConfig
...
>>> # Initializing a Hubert facebook/hubert-base-ls960 style configuration
>>> configuration = HubertConfig()
...
>>> # Initializing a model from the facebook/hubert-base-ls960 style configuration
>>> model = HubertModel(configuration)
... 
>>> # Accessing the model configuration
>>> configuration = model.config

Source code in mindnlp/transformers/models/hubert/configuration_hubert.py

class HubertConfig(PretrainedConfig):
    r"""
    This is the configuration class to store the configuration of a [`HubertModel`]. It is used to instantiate an
    Hubert model according to the specified arguments, defining the model architecture. Instantiating a configuration
    with the defaults will yield a similar configuration to that of the Hubert
    [facebook/hubert-base-ls960](https://hf-mirror.com/facebook/hubert-base-ls960) architecture.

    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
    documentation from [`PretrainedConfig`] for more information.


    Args:
        vocab_size (`int`, *optional*, defaults to 32):
            Vocabulary size of the Hubert model. Defines the number of different tokens that can be represented by the
            `inputs_ids` passed when calling [`HubertModel`]. Vocabulary size of the model. Defines the different
            tokens that can be represented by the *inputs_ids* passed to the forward method of [`HubertModel`].
        hidden_size (`int`, *optional*, defaults to 768):
            Dimensionality of the encoder layers and the pooler layer.
        num_hidden_layers (`int`, *optional*, defaults to 12):
            Number of hidden layers in the Transformer encoder.
        num_attention_heads (`int`, *optional*, defaults to 12):
            Number of attention heads for each attention layer in the Transformer encoder.
        intermediate_size (`int`, *optional*, defaults to 3072):
            Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder.
        hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`):
            The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`,
            `"relu"`, `"selu"` and `"gelu_new"` are supported.
        hidden_dropout(`float`, *optional*, defaults to 0.1):
            The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
        activation_dropout (`float`, *optional*, defaults to 0.1):
            The dropout ratio for activations inside the fully connected layer.
        attention_dropout(`float`, *optional*, defaults to 0.1):
            The dropout ratio for the attention probabilities.
        final_dropout (`float`, *optional*, defaults to 0.1):
            The dropout probabilitiy for the final projection layer of [`Wav2Vec2ForCTC`].
        layerdrop (`float`, *optional*, defaults to 0.1):
            The LayerDrop probability. See the [LayerDrop paper](see https://arxiv.org/abs/1909.11556) for more
            details.
        initializer_range (`float`, *optional*, defaults to 0.02):
            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
        layer_norm_eps (`float`, *optional*, defaults to 1e-12):
            The epsilon used by the layer normalization layers.
        feat_extract_norm (`str`, *optional*, defaults to `"group"`):
            The norm to be applied to 1D convolutional layers in feature encoder. One of `"group"` for group
            normalization of only the first 1D convolutional layer or `"layer"` for layer normalization of all 1D
            convolutional layers.
        feat_proj_dropout (`float`, *optional*, defaults to 0.0):
            The dropout probability for output of the feature encoder.
        feat_proj_layer_norm (`bool`, *optional*, defaults to `True`):
            Whether to apply LayerNorm to the output of the feature encoder.
        feat_extract_activation (`str, `optional`, defaults to `"gelu"`):
            The non-linear activation function (function or string) in the 1D convolutional layers of the feature
            extractor. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` are supported.
        conv_dim (`Tuple[int]`, *optional*, defaults to `(512, 512, 512, 512, 512, 512, 512)`):
            A tuple of integers defining the number of input and output channels of each 1D convolutional layer in the
            feature encoder. The length of *conv_dim* defines the number of 1D convolutional layers.
        conv_stride (`Tuple[int]`, *optional*, defaults to `(5, 2, 2, 2, 2, 2, 2)`):
            A tuple of integers defining the stride of each 1D convolutional layer in the feature encoder. The length
            of *conv_stride* defines the number of convolutional layers and has to match the length of *conv_dim*.
        conv_kernel (`Tuple[int]`, *optional*, defaults to `(10, 3, 3, 3, 3, 3, 3)`):
            A tuple of integers defining the kernel size of each 1D convolutional layer in the feature encoder. The
            length of *conv_kernel* defines the number of convolutional layers and has to match the length of
            *conv_dim*.
        conv_bias (`bool`, *optional*, defaults to `False`):
            Whether the 1D convolutional layers have a bias.
        num_conv_pos_embeddings (`int`, *optional*, defaults to 128):
            Number of convolutional positional embeddings. Defines the kernel size of 1D convolutional positional
            embeddings layer.
        num_conv_pos_embedding_groups (`int`, *optional*, defaults to 16):
            Number of groups of 1D convolutional positional embeddings layer.
        do_stable_layer_norm (`bool`, *optional*, defaults to `False`):
            Whether do apply *stable* layer norm architecture of the Transformer encoder. `do_stable_layer_norm is
            True` corresponds to applying layer norm before the attention layer, whereas `do_stable_layer_norm is
            False` corresponds to applying layer norm after the attention layer.
        apply_spec_augment (`bool`, *optional*, defaults to `True`):
            Whether to apply *SpecAugment* data augmentation to the outputs of the feature encoder. For reference see
            [SpecAugment: A Simple Data Augmentation Method for Automatic Speech
            Recognition](https://arxiv.org/abs/1904.08779).
        mask_time_prob (`float`, *optional*, defaults to 0.05):
            Percentage (between 0 and 1) of all feature vectors along the time axis which will be masked. The masking
            procecure generates ''mask_time_prob*len(time_axis)/mask_time_length'' independent masks over the axis. If
            reasoning from the propability of each feature vector to be chosen as the start of the vector span to be
            masked, *mask_time_prob* should be `prob_vector_start*mask_time_length`. Note that overlap may decrease the
            actual percentage of masked vectors. This is only relevant if `apply_spec_augment is True`.
        mask_time_length (`int`, *optional*, defaults to 10):
            Length of vector span along the time axis.
        mask_time_min_masks (`int`, *optional*, defaults to 2),:
            The minimum number of masks of length `mask_feature_length` generated along the time axis, each time step,
            irrespectively of `mask_feature_prob`. Only relevant if ''mask_time_prob*len(time_axis)/mask_time_length <
            mask_time_min_masks''
        mask_feature_prob (`float`, *optional*, defaults to 0.0):
            Percentage (between 0 and 1) of all feature vectors along the feature axis which will be masked. The
            masking procecure generates ''mask_feature_prob*len(feature_axis)/mask_time_length'' independent masks over
            the axis. If reasoning from the propability of each feature vector to be chosen as the start of the vector
            span to be masked, *mask_feature_prob* should be `prob_vector_start*mask_feature_length`. Note that overlap
            may decrease the actual percentage of masked vectors. This is only relevant if `apply_spec_augment is
            True`.
        mask_feature_length (`int`, *optional*, defaults to 10):
            Length of vector span along the feature axis.
        mask_feature_min_masks (`int`, *optional*, defaults to 0),:
            The minimum number of masks of length `mask_feature_length` generated along the feature axis, each time
            step, irrespectively of `mask_feature_prob`. Only relevant if
            ''mask_feature_prob*len(feature_axis)/mask_feature_length < mask_feature_min_masks''
        ctc_loss_reduction (`str`, *optional*, defaults to `"sum"`):
            Specifies the reduction to apply to the output of `torch.nn.CTCLoss`. Only relevant when training an
            instance of [`HubertForCTC`].
        ctc_zero_infinity (`bool`, *optional*, defaults to `False`):
            Whether to zero infinite losses and the associated gradients of `torch.nn.CTCLoss`. Infinite losses mainly
            occur when the inputs are too short to be aligned to the targets. Only relevant when training an instance
            of [`HubertForCTC`].
        use_weighted_layer_sum (`bool`, *optional*, defaults to `False`):
            Whether to use a weighted average of layer outputs with learned weights. Only relevant when using an
            instance of [`HubertForSequenceClassification`].
        classifier_proj_size (`int`, *optional*, defaults to 256):
            Dimensionality of the projection before token mean-pooling for classification.

    Example:
        ```python
        >>> from transformers import HubertModel, HubertConfig
        ...
        >>> # Initializing a Hubert facebook/hubert-base-ls960 style configuration
        >>> configuration = HubertConfig()
        ...
        >>> # Initializing a model from the facebook/hubert-base-ls960 style configuration
        >>> model = HubertModel(configuration)
        ... 
        >>> # Accessing the model configuration
        >>> configuration = model.config
        ```
    """
    model_type = "hubert"

    def __init__(
        self,
        vocab_size=32,
        hidden_size=768,
        num_hidden_layers=12,
        num_attention_heads=12,
        intermediate_size=3072,
        hidden_act="gelu",
        hidden_dropout=0.1,
        activation_dropout=0.1,
        attention_dropout=0.1,
        feat_proj_layer_norm=True,
        feat_proj_dropout=0.0,
        final_dropout=0.1,
        layerdrop=0.1,
        initializer_range=0.02,
        layer_norm_eps=1e-5,
        feat_extract_norm="group",
        feat_extract_activation="gelu",
        conv_dim=(512, 512, 512, 512, 512, 512, 512),
        conv_stride=(5, 2, 2, 2, 2, 2, 2),
        conv_kernel=(10, 3, 3, 3, 3, 2, 2),
        conv_bias=False,
        num_conv_pos_embeddings=128,
        num_conv_pos_embedding_groups=16,
        do_stable_layer_norm=False,
        apply_spec_augment=True,
        mask_time_prob=0.05,
        mask_time_length=10,
        mask_time_min_masks=2,
        mask_feature_prob=0.0,
        mask_feature_length=10,
        mask_feature_min_masks=0,
        ctc_loss_reduction="sum",
        ctc_zero_infinity=False,
        use_weighted_layer_sum=False,
        classifier_proj_size=256,
        pad_token_id=0,
        bos_token_id=1,
        eos_token_id=2,
        **kwargs,
    ):
        """
        Initializes a new instance of the HubertConfig class.

        Args:
            vocab_size (int): The size of the vocabulary.
            hidden_size (int): The size of the hidden layers.
            num_hidden_layers (int): The number of hidden layers.
            num_attention_heads (int): The number of attention heads.
            intermediate_size (int): The size of the intermediate layers.
            hidden_act (str): The activation function for the hidden layers.
            hidden_dropout (float): The dropout rate for the hidden layers.
            activation_dropout (float): The dropout rate for activations.
            attention_dropout (float): The dropout rate for attention mechanisms.
            feat_proj_layer_norm (bool): Whether to apply layer normalization to projection features.
            feat_proj_dropout (float): The dropout rate for feature projection.
            final_dropout (float): The final dropout rate.
            layerdrop (float): The layer drop probability.
            initializer_range (float): The range for parameter initialization.
            layer_norm_eps (float): The epsilon value for layer normalization.
            feat_extract_norm (str): The normalization type for feature extraction.
            feat_extract_activation (str): The activation function for feature extraction.
            conv_dim (tuple): The dimensions for convolutional layers.
            conv_stride (tuple): The stride values for convolutional layers.
            conv_kernel (tuple): The kernel sizes for convolutional layers.
            conv_bias (bool): Whether to use bias in convolutional layers.
            num_conv_pos_embeddings (int): The number of positional embeddings for convolutional layers.
            num_conv_pos_embedding_groups (int): The number of groups for positional embeddings.
            do_stable_layer_norm (bool): Whether to use stable layer normalization.
            apply_spec_augment (bool): Whether to apply SpecAugment during training.
            mask_time_prob (float): The probability of masking in the time dimension.
            mask_time_length (int): The maximum length of time masking.
            mask_time_min_masks (int): The minimum number of time masks.
            mask_feature_prob (float): The probability of masking in the feature dimension.
            mask_feature_length (int): The maximum length of feature masking.
            mask_feature_min_masks (int): The minimum number of feature masks.
            ctc_loss_reduction (str): The reduction type for CTC loss.
            ctc_zero_infinity (bool): Whether to set positive infinity to zero in CTC loss.
            use_weighted_layer_sum (bool): Whether to use weighted layer sum for classification.
            classifier_proj_size (int): The size of the classifier projection layer.
            pad_token_id (int): The token ID for padding.
            bos_token_id (int): The token ID for the beginning of sequence.
            eos_token_id (int): The token ID for the end of sequence.

        Returns:
            None

        Raises:
            ValueError: If the configuration for convolutional layers is incorrect.
        """
        super().__init__(**kwargs, pad_token_id=pad_token_id, bos_token_id=bos_token_id, eos_token_id=eos_token_id)

        self.hidden_size = hidden_size
        self.feat_extract_norm = feat_extract_norm
        self.feat_extract_activation = feat_extract_activation
        self.conv_dim = list(conv_dim)
        self.conv_stride = list(conv_stride)
        self.conv_kernel = list(conv_kernel)
        self.conv_bias = conv_bias
        self.num_conv_pos_embeddings = num_conv_pos_embeddings
        self.num_conv_pos_embedding_groups = num_conv_pos_embedding_groups
        self.num_feat_extract_layers = len(self.conv_dim)
        self.num_hidden_layers = num_hidden_layers
        self.intermediate_size = intermediate_size
        self.hidden_act = hidden_act
        self.num_attention_heads = num_attention_heads
        self.hidden_dropout = hidden_dropout
        self.attention_dropout = attention_dropout
        self.activation_dropout = activation_dropout
        self.feat_proj_layer_norm = feat_proj_layer_norm
        self.feat_proj_dropout = feat_proj_dropout
        self.final_dropout = final_dropout
        self.layerdrop = layerdrop
        self.layer_norm_eps = layer_norm_eps
        self.initializer_range = initializer_range
        self.vocab_size = vocab_size
        self.do_stable_layer_norm = do_stable_layer_norm
        self.use_weighted_layer_sum = use_weighted_layer_sum
        self.classifier_proj_size = classifier_proj_size

        if (
            (len(self.conv_stride) != self.num_feat_extract_layers)
            or (len(self.conv_kernel) != self.num_feat_extract_layers)
            or (len(self.conv_dim) != self.num_feat_extract_layers)
        ):
            raise ValueError(
                "Configuration for convolutional layers is incorrect. It is required that `len(config.conv_dim)` =="
                " `len(config.conv_stride)` == `len(config.conv_kernel)`, but is `len(config.conv_dim) ="
                f" {len(self.conv_dim)}`, `len(config.conv_stride) = {len(self.conv_stride)}`,"
                f" `len(config.conv_kernel) = {len(self.conv_kernel)}`."
            )

        # fine-tuning config parameters for SpecAugment: https://arxiv.org/abs/1904.08779
        self.apply_spec_augment = apply_spec_augment
        self.mask_time_prob = mask_time_prob
        self.mask_time_length = mask_time_length
        self.mask_time_min_masks = mask_time_min_masks
        self.mask_feature_prob = mask_feature_prob
        self.mask_feature_length = mask_feature_length
        self.mask_feature_min_masks = mask_feature_min_masks

        # ctc loss
        self.ctc_loss_reduction = ctc_loss_reduction
        self.ctc_zero_infinity = ctc_zero_infinity

    @property
    def inputs_to_logits_ratio(self):
        """
        Calculates the ratio of inputs to logits based on the convolutional strides in the Hubert configuration.

        Args:
            self (HubertConfig): The instance of HubertConfig.

        Returns:
            int: The ratio of inputs to logits calculated as the product of convolutional strides.

        Raises:
            None.
        """
        return functools.reduce(operator.mul, self.conv_stride, 1)

mindnlp.transformers.models.hubert.configuration_hubert.HubertConfig.inputs_to_logits_ratio property

Calculates the ratio of inputs to logits based on the convolutional strides in the Hubert configuration.

PARAMETER	DESCRIPTION
self	The instance of HubertConfig. TYPE: HubertConfig

RETURNS	DESCRIPTION
int	The ratio of inputs to logits calculated as the product of convolutional strides.

mindnlp.transformers.models.hubert.configuration_hubert.HubertConfig.__init__(vocab_size=32, hidden_size=768, num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072, hidden_act='gelu', hidden_dropout=0.1, activation_dropout=0.1, attention_dropout=0.1, feat_proj_layer_norm=True, feat_proj_dropout=0.0, final_dropout=0.1, layerdrop=0.1, initializer_range=0.02, layer_norm_eps=1e-05, feat_extract_norm='group', feat_extract_activation='gelu', conv_dim=(512, 512, 512, 512, 512, 512, 512), conv_stride=(5, 2, 2, 2, 2, 2, 2), conv_kernel=(10, 3, 3, 3, 3, 2, 2), conv_bias=False, num_conv_pos_embeddings=128, num_conv_pos_embedding_groups=16, do_stable_layer_norm=False, apply_spec_augment=True, mask_time_prob=0.05, mask_time_length=10, mask_time_min_masks=2, mask_feature_prob=0.0, mask_feature_length=10, mask_feature_min_masks=0, ctc_loss_reduction='sum', ctc_zero_infinity=False, use_weighted_layer_sum=False, classifier_proj_size=256, pad_token_id=0, bos_token_id=1, eos_token_id=2, **kwargs)

Initializes a new instance of the HubertConfig class.

PARAMETER	DESCRIPTION
vocab_size	The size of the vocabulary. TYPE: int DEFAULT: 32
hidden_size	The size of the hidden layers. TYPE: int DEFAULT: 768
num_hidden_layers	The number of hidden layers. TYPE: int DEFAULT: 12
num_attention_heads	The number of attention heads. TYPE: int DEFAULT: 12
intermediate_size	The size of the intermediate layers. TYPE: int DEFAULT: 3072
hidden_act	The activation function for the hidden layers. TYPE: str DEFAULT: 'gelu'
hidden_dropout	The dropout rate for the hidden layers. TYPE: float DEFAULT: 0.1
activation_dropout	The dropout rate for activations. TYPE: float DEFAULT: 0.1
attention_dropout	The dropout rate for attention mechanisms. TYPE: float DEFAULT: 0.1
feat_proj_layer_norm	Whether to apply layer normalization to projection features. TYPE: bool DEFAULT: True
feat_proj_dropout	The dropout rate for feature projection. TYPE: float DEFAULT: 0.0
final_dropout	The final dropout rate. TYPE: float DEFAULT: 0.1
layerdrop	The layer drop probability. TYPE: float DEFAULT: 0.1
initializer_range	The range for parameter initialization. TYPE: float DEFAULT: 0.02
layer_norm_eps	The epsilon value for layer normalization. TYPE: float DEFAULT: 1e-05
feat_extract_norm	The normalization type for feature extraction. TYPE: str DEFAULT: 'group'
feat_extract_activation	The activation function for feature extraction. TYPE: str DEFAULT: 'gelu'
conv_dim	The dimensions for convolutional layers. TYPE: tuple DEFAULT: (512, 512, 512, 512, 512, 512, 512)
conv_stride	The stride values for convolutional layers. TYPE: tuple DEFAULT: (5, 2, 2, 2, 2, 2, 2)
conv_kernel	The kernel sizes for convolutional layers. TYPE: tuple DEFAULT: (10, 3, 3, 3, 3, 2, 2)
conv_bias	Whether to use bias in convolutional layers. TYPE: bool DEFAULT: False
num_conv_pos_embeddings	The number of positional embeddings for convolutional layers. TYPE: int DEFAULT: 128
num_conv_pos_embedding_groups	The number of groups for positional embeddings. TYPE: int DEFAULT: 16
do_stable_layer_norm	Whether to use stable layer normalization. TYPE: bool DEFAULT: False
apply_spec_augment	Whether to apply SpecAugment during training. TYPE: bool DEFAULT: True
mask_time_prob	The probability of masking in the time dimension. TYPE: float DEFAULT: 0.05
mask_time_length	The maximum length of time masking. TYPE: int DEFAULT: 10
mask_time_min_masks	The minimum number of time masks. TYPE: int DEFAULT: 2
mask_feature_prob	The probability of masking in the feature dimension. TYPE: float DEFAULT: 0.0
mask_feature_length	The maximum length of feature masking. TYPE: int DEFAULT: 10
mask_feature_min_masks	The minimum number of feature masks. TYPE: int DEFAULT: 0
ctc_loss_reduction	The reduction type for CTC loss. TYPE: str DEFAULT: 'sum'
ctc_zero_infinity	Whether to set positive infinity to zero in CTC loss. TYPE: bool DEFAULT: False
use_weighted_layer_sum	Whether to use weighted layer sum for classification. TYPE: bool DEFAULT: False
classifier_proj_size	The size of the classifier projection layer. TYPE: int DEFAULT: 256
pad_token_id	The token ID for padding. TYPE: int DEFAULT: 0
bos_token_id	The token ID for the beginning of sequence. TYPE: int DEFAULT: 1
eos_token_id	The token ID for the end of sequence. TYPE: int DEFAULT: 2

RETURNS	DESCRIPTION
	None

RAISES	DESCRIPTION
ValueError	If the configuration for convolutional layers is incorrect.

Source code in mindnlp/transformers/models/hubert/configuration_hubert.py

def __init__(
    self,
    vocab_size=32,
    hidden_size=768,
    num_hidden_layers=12,
    num_attention_heads=12,
    intermediate_size=3072,
    hidden_act="gelu",
    hidden_dropout=0.1,
    activation_dropout=0.1,
    attention_dropout=0.1,
    feat_proj_layer_norm=True,
    feat_proj_dropout=0.0,
    final_dropout=0.1,
    layerdrop=0.1,
    initializer_range=0.02,
    layer_norm_eps=1e-5,
    feat_extract_norm="group",
    feat_extract_activation="gelu",
    conv_dim=(512, 512, 512, 512, 512, 512, 512),
    conv_stride=(5, 2, 2, 2, 2, 2, 2),
    conv_kernel=(10, 3, 3, 3, 3, 2, 2),
    conv_bias=False,
    num_conv_pos_embeddings=128,
    num_conv_pos_embedding_groups=16,
    do_stable_layer_norm=False,
    apply_spec_augment=True,
    mask_time_prob=0.05,
    mask_time_length=10,
    mask_time_min_masks=2,
    mask_feature_prob=0.0,
    mask_feature_length=10,
    mask_feature_min_masks=0,
    ctc_loss_reduction="sum",
    ctc_zero_infinity=False,
    use_weighted_layer_sum=False,
    classifier_proj_size=256,
    pad_token_id=0,
    bos_token_id=1,
    eos_token_id=2,
    **kwargs,
):
    """
    Initializes a new instance of the HubertConfig class.

    Args:
        vocab_size (int): The size of the vocabulary.
        hidden_size (int): The size of the hidden layers.
        num_hidden_layers (int): The number of hidden layers.
        num_attention_heads (int): The number of attention heads.
        intermediate_size (int): The size of the intermediate layers.
        hidden_act (str): The activation function for the hidden layers.
        hidden_dropout (float): The dropout rate for the hidden layers.
        activation_dropout (float): The dropout rate for activations.
        attention_dropout (float): The dropout rate for attention mechanisms.
        feat_proj_layer_norm (bool): Whether to apply layer normalization to projection features.
        feat_proj_dropout (float): The dropout rate for feature projection.
        final_dropout (float): The final dropout rate.
        layerdrop (float): The layer drop probability.
        initializer_range (float): The range for parameter initialization.
        layer_norm_eps (float): The epsilon value for layer normalization.
        feat_extract_norm (str): The normalization type for feature extraction.
        feat_extract_activation (str): The activation function for feature extraction.
        conv_dim (tuple): The dimensions for convolutional layers.
        conv_stride (tuple): The stride values for convolutional layers.
        conv_kernel (tuple): The kernel sizes for convolutional layers.
        conv_bias (bool): Whether to use bias in convolutional layers.
        num_conv_pos_embeddings (int): The number of positional embeddings for convolutional layers.
        num_conv_pos_embedding_groups (int): The number of groups for positional embeddings.
        do_stable_layer_norm (bool): Whether to use stable layer normalization.
        apply_spec_augment (bool): Whether to apply SpecAugment during training.
        mask_time_prob (float): The probability of masking in the time dimension.
        mask_time_length (int): The maximum length of time masking.
        mask_time_min_masks (int): The minimum number of time masks.
        mask_feature_prob (float): The probability of masking in the feature dimension.
        mask_feature_length (int): The maximum length of feature masking.
        mask_feature_min_masks (int): The minimum number of feature masks.
        ctc_loss_reduction (str): The reduction type for CTC loss.
        ctc_zero_infinity (bool): Whether to set positive infinity to zero in CTC loss.
        use_weighted_layer_sum (bool): Whether to use weighted layer sum for classification.
        classifier_proj_size (int): The size of the classifier projection layer.
        pad_token_id (int): The token ID for padding.
        bos_token_id (int): The token ID for the beginning of sequence.
        eos_token_id (int): The token ID for the end of sequence.

    Returns:
        None

    Raises:
        ValueError: If the configuration for convolutional layers is incorrect.
    """
    super().__init__(**kwargs, pad_token_id=pad_token_id, bos_token_id=bos_token_id, eos_token_id=eos_token_id)

    self.hidden_size = hidden_size
    self.feat_extract_norm = feat_extract_norm
    self.feat_extract_activation = feat_extract_activation
    self.conv_dim = list(conv_dim)
    self.conv_stride = list(conv_stride)
    self.conv_kernel = list(conv_kernel)
    self.conv_bias = conv_bias
    self.num_conv_pos_embeddings = num_conv_pos_embeddings
    self.num_conv_pos_embedding_groups = num_conv_pos_embedding_groups
    self.num_feat_extract_layers = len(self.conv_dim)
    self.num_hidden_layers = num_hidden_layers
    self.intermediate_size = intermediate_size
    self.hidden_act = hidden_act
    self.num_attention_heads = num_attention_heads
    self.hidden_dropout = hidden_dropout
    self.attention_dropout = attention_dropout
    self.activation_dropout = activation_dropout
    self.feat_proj_layer_norm = feat_proj_layer_norm
    self.feat_proj_dropout = feat_proj_dropout
    self.final_dropout = final_dropout
    self.layerdrop = layerdrop
    self.layer_norm_eps = layer_norm_eps
    self.initializer_range = initializer_range
    self.vocab_size = vocab_size
    self.do_stable_layer_norm = do_stable_layer_norm
    self.use_weighted_layer_sum = use_weighted_layer_sum
    self.classifier_proj_size = classifier_proj_size

    if (
        (len(self.conv_stride) != self.num_feat_extract_layers)
        or (len(self.conv_kernel) != self.num_feat_extract_layers)
        or (len(self.conv_dim) != self.num_feat_extract_layers)
    ):
        raise ValueError(
            "Configuration for convolutional layers is incorrect. It is required that `len(config.conv_dim)` =="
            " `len(config.conv_stride)` == `len(config.conv_kernel)`, but is `len(config.conv_dim) ="
            f" {len(self.conv_dim)}`, `len(config.conv_stride) = {len(self.conv_stride)}`,"
            f" `len(config.conv_kernel) = {len(self.conv_kernel)}`."
        )

    # fine-tuning config parameters for SpecAugment: https://arxiv.org/abs/1904.08779
    self.apply_spec_augment = apply_spec_augment
    self.mask_time_prob = mask_time_prob
    self.mask_time_length = mask_time_length
    self.mask_time_min_masks = mask_time_min_masks
    self.mask_feature_prob = mask_feature_prob
    self.mask_feature_length = mask_feature_length
    self.mask_feature_min_masks = mask_feature_min_masks

    # ctc loss
    self.ctc_loss_reduction = ctc_loss_reduction
    self.ctc_zero_infinity = ctc_zero_infinity

mindnlp.transformers.models.hubert.modeling_hubert

MindSpore Hubert model.

mindnlp.transformers.models.hubert.modeling_hubert.HubertAttention

Bases: Module

Multi-headed attention from 'Attention Is All You Need' paper

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

class HubertAttention(nn.Module):
    """Multi-headed attention from 'Attention Is All You Need' paper"""
    def __init__(
        self,
        embed_dim: int,
        num_heads: int,
        dropout: float = 0.0,
        is_decoder: bool = False,
        bias: bool = True,
        is_causal: bool = False,
        config: Optional[HubertConfig] = None,
    ):
        """
        Initializes a HubertAttention object.

        Args:
            embed_dim (int): The dimension of the input embeddings.
            num_heads (int): The number of attention heads to use.
            dropout (float, optional): The dropout probability. Default is 0.0.
            is_decoder (bool, optional): Whether the attention mechanism is used in a decoder. Default is False.
            bias (bool, optional): Whether to include bias in the linear projections. Default is True.
            is_causal (bool, optional): Whether the attention is causal. Default is False.
            config (Optional[HubertConfig], optional): The configuration object for the attention mechanism.
                Default is None.

        Returns:
            None.

        Raises:
            ValueError: If `embed_dim` is not divisible by `num_heads`.
        """
        super().__init__()
        self.embed_dim = embed_dim
        self.num_heads = num_heads
        self.dropout = dropout
        self.head_dim = embed_dim // num_heads
        self.config = config

        if (self.head_dim * num_heads) != self.embed_dim:
            raise ValueError(
                f"embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim}"
                f" and `num_heads`: {num_heads})."
            )
        self.scaling = self.head_dim**-0.5
        self.is_decoder = is_decoder
        self.is_causal = is_causal

        self.k_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.v_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.q_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.out_proj = nn.Linear(embed_dim, embed_dim, bias=bias)

    def _shape(self, tensor: Tensor, seq_len: int, bsz: int):
        """
        Method _shape in class HubertAttention.

        Args:
            self (HubertAttention): The instance of the HubertAttention class.
            tensor (Tensor): The input tensor to be reshaped.
            seq_len (int): The length of the sequence.
            bsz (int): The batch size.

        Returns:
            None: This method reshapes the input tensor based on the provided sequence length and batch size.

        Raises:
            None.
        """
        return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).swapaxes(1, 2)

    def forward(
        self,
        hidden_states: Tensor,
        key_value_states: Optional[Tensor] = None,
        past_key_value: Optional[Tuple[Tensor]] = None,
        attention_mask: Optional[Tensor] = None,
        layer_head_mask: Optional[Tensor] = None,
        output_attentions: bool = False,
    ) -> Tuple[Tensor, Optional[Tensor], Optional[Tuple[Tensor]]]:
        """Input shape: Batch x Time x Channel"""
        # if key_value_states are provided this layer is used as a cross-attention layer
        # for the decoder
        is_cross_attention = key_value_states is not None

        bsz, tgt_len, _ = hidden_states.shape

        # get query proj
        query_states = self.q_proj(hidden_states) * self.scaling
        # get key, value proj
        # `past_key_value[0].shape[2] == key_value_states.shape[1]`
        # is checking that the `sequence_length` of the `past_key_value` is the same as
        # the provided `key_value_states` to support prefix tuning
        if (
            is_cross_attention
            and past_key_value is not None
            and past_key_value[0].shape[2] == key_value_states.shape[1]
        ):
            # reuse k,v, cross_attentions
            key_states = past_key_value[0]
            value_states = past_key_value[1]
        elif is_cross_attention:
            # cross_attentions
            key_states = self._shape(self.k_proj(key_value_states), -1, bsz)
            value_states = self._shape(self.v_proj(key_value_states), -1, bsz)
        elif past_key_value is not None:
            # reuse k, v, self_attention
            key_states = self._shape(self.k_proj(hidden_states), -1, bsz)
            value_states = self._shape(self.v_proj(hidden_states), -1, bsz)
            key_states = ops.cat([past_key_value[0], key_states], axis=2)
            value_states = ops.cat([past_key_value[1], value_states], axis=2)
        else:
            # self_attention
            key_states = self._shape(self.k_proj(hidden_states), -1, bsz)
            value_states = self._shape(self.v_proj(hidden_states), -1, bsz)

        if self.is_decoder:
            # if cross_attention save Tuple(Tensor, Tensor) of all cross attention key/value_states.
            # Further calls to cross_attention layer can then reuse all cross-attention
            # key/value_states (first "if" case)
            # if uni-directional self-attention (decoder) save Tuple(Tensor, Tensor) of
            # all previous decoder key/value_states. Further calls to uni-directional self-attention
            # can concat previous decoder key/value_states to current projected key/value_states (third "elif" case)
            # if encoder bi-directional self-attention `past_key_value` is always `None`
            past_key_value = (key_states, value_states)

        proj_shape = (bsz * self.num_heads, -1, self.head_dim)
        query_states = self._shape(query_states, tgt_len, bsz).view(*proj_shape)
        key_states = key_states.reshape(*proj_shape)
        value_states = value_states.reshape(*proj_shape)

        src_len = key_states.shape[1]
        attn_weights = ops.bmm(query_states, key_states.swapaxes(1, 2))

        if attn_weights.shape != (bsz * self.num_heads, tgt_len, src_len):
            raise ValueError(
                f"Attention weights should be of size {(bsz * self.num_heads, tgt_len, src_len)}, but is"
                f" {attn_weights.shape}"
            )

        if attention_mask is not None:
            if attention_mask.shape != (bsz, 1, tgt_len, src_len):
                raise ValueError(f"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is {attention_mask.shape}")
            attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attention_mask
            attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)

        attn_weights = ops.softmax(attn_weights, axis=-1)

        if layer_head_mask is not None:
            if layer_head_mask.shape != (self.num_heads,):
                raise ValueError(
                    f"Head mask for a single layer should be of size {(self.num_heads,)}, but is"
                    f" {layer_head_mask.shape}"
                )
            attn_weights = layer_head_mask.view(1, -1, 1, 1) * attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
            attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)

        if output_attentions:
            # this operation is a bit awkward, but it's required to
            # make sure that attn_weights keeps its gradient.
            # In order to do so, attn_weights have to be reshaped
            # twice and have to be reused in the following
            attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
            attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len)
        else:
            attn_weights_reshaped = None

        attn_probs = ops.dropout(attn_weights, p=self.dropout, training=self.training)
        attn_output = ops.bmm(attn_probs, value_states)

        if attn_output.shape != (bsz * self.num_heads, tgt_len, self.head_dim):
            raise ValueError(
                f"`attn_output` should be of size {(bsz * self.num_heads, tgt_len, self.head_dim)}, but is"
                f" {attn_output.shape}"
            )

        attn_output = attn_output.view(bsz, self.num_heads, tgt_len, self.head_dim)
        attn_output = attn_output.swapaxes(1, 2)

        # Use the `embed_dim` from the config (stored in the class) rather than `hidden_state` because `attn_output` can be
        # partitioned across GPUs when using tensor-parallelism.
        attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim)
        attn_output = self.out_proj(attn_output)

        return attn_output, attn_weights_reshaped, past_key_value

mindnlp.transformers.models.hubert.modeling_hubert.HubertAttention.__init__(embed_dim, num_heads, dropout=0.0, is_decoder=False, bias=True, is_causal=False, config=None)

Initializes a HubertAttention object.

PARAMETER	DESCRIPTION
embed_dim	The dimension of the input embeddings. TYPE: int
num_heads	The number of attention heads to use. TYPE: int
dropout	The dropout probability. Default is 0.0. TYPE: float DEFAULT: 0.0
is_decoder	Whether the attention mechanism is used in a decoder. Default is False. TYPE: bool DEFAULT: False
bias	Whether to include bias in the linear projections. Default is True. TYPE: bool DEFAULT: True
is_causal	Whether the attention is causal. Default is False. TYPE: bool DEFAULT: False
config	The configuration object for the attention mechanism. Default is None. TYPE: Optional[HubertConfig] DEFAULT: None

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If embed_dim is not divisible by num_heads.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def __init__(
    self,
    embed_dim: int,
    num_heads: int,
    dropout: float = 0.0,
    is_decoder: bool = False,
    bias: bool = True,
    is_causal: bool = False,
    config: Optional[HubertConfig] = None,
):
    """
    Initializes a HubertAttention object.

    Args:
        embed_dim (int): The dimension of the input embeddings.
        num_heads (int): The number of attention heads to use.
        dropout (float, optional): The dropout probability. Default is 0.0.
        is_decoder (bool, optional): Whether the attention mechanism is used in a decoder. Default is False.
        bias (bool, optional): Whether to include bias in the linear projections. Default is True.
        is_causal (bool, optional): Whether the attention is causal. Default is False.
        config (Optional[HubertConfig], optional): The configuration object for the attention mechanism.
            Default is None.

    Returns:
        None.

    Raises:
        ValueError: If `embed_dim` is not divisible by `num_heads`.
    """
    super().__init__()
    self.embed_dim = embed_dim
    self.num_heads = num_heads
    self.dropout = dropout
    self.head_dim = embed_dim // num_heads
    self.config = config

    if (self.head_dim * num_heads) != self.embed_dim:
        raise ValueError(
            f"embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim}"
            f" and `num_heads`: {num_heads})."
        )
    self.scaling = self.head_dim**-0.5
    self.is_decoder = is_decoder
    self.is_causal = is_causal

    self.k_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
    self.v_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
    self.q_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
    self.out_proj = nn.Linear(embed_dim, embed_dim, bias=bias)

mindnlp.transformers.models.hubert.modeling_hubert.HubertAttention.forward(hidden_states, key_value_states=None, past_key_value=None, attention_mask=None, layer_head_mask=None, output_attentions=False)

Input shape: Batch x Time x Channel

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def forward(
    self,
    hidden_states: Tensor,
    key_value_states: Optional[Tensor] = None,
    past_key_value: Optional[Tuple[Tensor]] = None,
    attention_mask: Optional[Tensor] = None,
    layer_head_mask: Optional[Tensor] = None,
    output_attentions: bool = False,
) -> Tuple[Tensor, Optional[Tensor], Optional[Tuple[Tensor]]]:
    """Input shape: Batch x Time x Channel"""
    # if key_value_states are provided this layer is used as a cross-attention layer
    # for the decoder
    is_cross_attention = key_value_states is not None

    bsz, tgt_len, _ = hidden_states.shape

    # get query proj
    query_states = self.q_proj(hidden_states) * self.scaling
    # get key, value proj
    # `past_key_value[0].shape[2] == key_value_states.shape[1]`
    # is checking that the `sequence_length` of the `past_key_value` is the same as
    # the provided `key_value_states` to support prefix tuning
    if (
        is_cross_attention
        and past_key_value is not None
        and past_key_value[0].shape[2] == key_value_states.shape[1]
    ):
        # reuse k,v, cross_attentions
        key_states = past_key_value[0]
        value_states = past_key_value[1]
    elif is_cross_attention:
        # cross_attentions
        key_states = self._shape(self.k_proj(key_value_states), -1, bsz)
        value_states = self._shape(self.v_proj(key_value_states), -1, bsz)
    elif past_key_value is not None:
        # reuse k, v, self_attention
        key_states = self._shape(self.k_proj(hidden_states), -1, bsz)
        value_states = self._shape(self.v_proj(hidden_states), -1, bsz)
        key_states = ops.cat([past_key_value[0], key_states], axis=2)
        value_states = ops.cat([past_key_value[1], value_states], axis=2)
    else:
        # self_attention
        key_states = self._shape(self.k_proj(hidden_states), -1, bsz)
        value_states = self._shape(self.v_proj(hidden_states), -1, bsz)

    if self.is_decoder:
        # if cross_attention save Tuple(Tensor, Tensor) of all cross attention key/value_states.
        # Further calls to cross_attention layer can then reuse all cross-attention
        # key/value_states (first "if" case)
        # if uni-directional self-attention (decoder) save Tuple(Tensor, Tensor) of
        # all previous decoder key/value_states. Further calls to uni-directional self-attention
        # can concat previous decoder key/value_states to current projected key/value_states (third "elif" case)
        # if encoder bi-directional self-attention `past_key_value` is always `None`
        past_key_value = (key_states, value_states)

    proj_shape = (bsz * self.num_heads, -1, self.head_dim)
    query_states = self._shape(query_states, tgt_len, bsz).view(*proj_shape)
    key_states = key_states.reshape(*proj_shape)
    value_states = value_states.reshape(*proj_shape)

    src_len = key_states.shape[1]
    attn_weights = ops.bmm(query_states, key_states.swapaxes(1, 2))

    if attn_weights.shape != (bsz * self.num_heads, tgt_len, src_len):
        raise ValueError(
            f"Attention weights should be of size {(bsz * self.num_heads, tgt_len, src_len)}, but is"
            f" {attn_weights.shape}"
        )

    if attention_mask is not None:
        if attention_mask.shape != (bsz, 1, tgt_len, src_len):
            raise ValueError(f"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is {attention_mask.shape}")
        attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attention_mask
        attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)

    attn_weights = ops.softmax(attn_weights, axis=-1)

    if layer_head_mask is not None:
        if layer_head_mask.shape != (self.num_heads,):
            raise ValueError(
                f"Head mask for a single layer should be of size {(self.num_heads,)}, but is"
                f" {layer_head_mask.shape}"
            )
        attn_weights = layer_head_mask.view(1, -1, 1, 1) * attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
        attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)

    if output_attentions:
        # this operation is a bit awkward, but it's required to
        # make sure that attn_weights keeps its gradient.
        # In order to do so, attn_weights have to be reshaped
        # twice and have to be reused in the following
        attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
        attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len)
    else:
        attn_weights_reshaped = None

    attn_probs = ops.dropout(attn_weights, p=self.dropout, training=self.training)
    attn_output = ops.bmm(attn_probs, value_states)

    if attn_output.shape != (bsz * self.num_heads, tgt_len, self.head_dim):
        raise ValueError(
            f"`attn_output` should be of size {(bsz * self.num_heads, tgt_len, self.head_dim)}, but is"
            f" {attn_output.shape}"
        )

    attn_output = attn_output.view(bsz, self.num_heads, tgt_len, self.head_dim)
    attn_output = attn_output.swapaxes(1, 2)

    # Use the `embed_dim` from the config (stored in the class) rather than `hidden_state` because `attn_output` can be
    # partitioned across GPUs when using tensor-parallelism.
    attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim)
    attn_output = self.out_proj(attn_output)

    return attn_output, attn_weights_reshaped, past_key_value

mindnlp.transformers.models.hubert.modeling_hubert.HubertAttnAdapterLayer

Bases: Module

Implements an adapter layer for attention modules in the Hubert model, optimizing training throughput by utilizing 3D tensor weights as parameters and bypassing the use of ModuleList.

This class inherits from nn.Module and provides functionality to forward adapter modules directly with 3D tensor weights as parameters, without using ModuleList, resulting in improved training throughput.

ATTRIBUTE	DESCRIPTION
input_dim	The dimension of the input tensor for the adapter layer. TYPE: int
hidden_dim	The hidden size dimension of the adapter layer. TYPE: int
norm	An instance of LayerNorm for normalizing the hidden states. TYPE: LayerNorm
linear_1	An instance of Dense representing the first linear transformation. TYPE: Dense
act_fn	An instance of ReLU activation function. TYPE: ReLU
linear_2	An instance of Dense representing the second linear transformation. TYPE: Dense

METHOD	DESCRIPTION
forward	Constructs the adapter layer by applying normalization, linear transformations, and activation function to the given hidden_states tensor, and returns the resulting tensor.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

class HubertAttnAdapterLayer(nn.Module):

    """
    Implements an adapter layer for attention modules in the Hubert model, optimizing training throughput
    by utilizing 3D tensor weights as parameters and bypassing the use of ModuleList.

    This class inherits from nn.Module and provides functionality to forward adapter modules directly with 3D tensor
    weights as parameters, without using ModuleList, resulting in improved training throughput.

    Attributes:
        input_dim (int): The dimension of the input tensor for the adapter layer.
        hidden_dim (int): The hidden size dimension of the adapter layer.
        norm (LayerNorm): An instance of LayerNorm for normalizing the hidden states.
        linear_1 (Dense): An instance of Dense representing the first linear transformation.
        act_fn (ReLU): An instance of ReLU activation function.
        linear_2 (Dense): An instance of Dense representing the second linear transformation.

    Methods:
        forward:
            Constructs the adapter layer by applying normalization, linear transformations, and activation function
            to the given hidden_states tensor, and returns the resulting tensor.

    """
    def __init__(self, config: HubertConfig):
        """
        Implements adapter modules directly with 3D tensor weight as parameters and without using ModuleList to speed
        up training throughput.
        """
        super().__init__()
        self.input_dim = config.adapter_attn_dim
        self.hidden_dim = config.hidden_size

        self.norm = nn.LayerNorm(self.hidden_dim)
        self.linear_1 = nn.Linear(self.hidden_dim, self.input_dim)
        self.act_fn = nn.ReLU()
        self.linear_2 = nn.Linear(self.input_dim, self.hidden_dim)

    def forward(self, hidden_states: Tensor):
        """
        Method to forward the attention adapter layer in the HubertAttnAdapterLayer class.

        Args:
            self (HubertAttnAdapterLayer): The instance of the HubertAttnAdapterLayer class.
            hidden_states (Tensor): The input hidden states tensor for the layer.

        Returns:
            hidden_states: The forwarded hidden states after passing through the layer operations.

        Raises:
            None.
        """
        hidden_states = self.norm(hidden_states)
        hidden_states = self.linear_1(hidden_states)
        hidden_states = self.act_fn(hidden_states)
        hidden_states = self.linear_2(hidden_states)
        return hidden_states

mindnlp.transformers.models.hubert.modeling_hubert.HubertAttnAdapterLayer.__init__(config)

Implements adapter modules directly with 3D tensor weight as parameters and without using ModuleList to speed up training throughput.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def __init__(self, config: HubertConfig):
    """
    Implements adapter modules directly with 3D tensor weight as parameters and without using ModuleList to speed
    up training throughput.
    """
    super().__init__()
    self.input_dim = config.adapter_attn_dim
    self.hidden_dim = config.hidden_size

    self.norm = nn.LayerNorm(self.hidden_dim)
    self.linear_1 = nn.Linear(self.hidden_dim, self.input_dim)
    self.act_fn = nn.ReLU()
    self.linear_2 = nn.Linear(self.input_dim, self.hidden_dim)

mindnlp.transformers.models.hubert.modeling_hubert.HubertAttnAdapterLayer.forward(hidden_states)

Method to forward the attention adapter layer in the HubertAttnAdapterLayer class.

PARAMETER	DESCRIPTION
self	The instance of the HubertAttnAdapterLayer class. TYPE: HubertAttnAdapterLayer
hidden_states	The input hidden states tensor for the layer. TYPE: Tensor

RETURNS	DESCRIPTION
hidden_states	The forwarded hidden states after passing through the layer operations.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def forward(self, hidden_states: Tensor):
    """
    Method to forward the attention adapter layer in the HubertAttnAdapterLayer class.

    Args:
        self (HubertAttnAdapterLayer): The instance of the HubertAttnAdapterLayer class.
        hidden_states (Tensor): The input hidden states tensor for the layer.

    Returns:
        hidden_states: The forwarded hidden states after passing through the layer operations.

    Raises:
        None.
    """
    hidden_states = self.norm(hidden_states)
    hidden_states = self.linear_1(hidden_states)
    hidden_states = self.act_fn(hidden_states)
    hidden_states = self.linear_2(hidden_states)
    return hidden_states

mindnlp.transformers.models.hubert.modeling_hubert.HubertEncoder

Bases: Module

A class representing the encoder component of the Hubert model. This class is responsible for processing input hidden states through multiple layers of HubertEncoderLayer.

This class inherits from nn.Module.

ATTRIBUTE	DESCRIPTION
config	The configuration object for the Hubert model. TYPE: HubertConfig
pos_conv_embed	Instance of HubertPositionalConvEmbedding for positional convolutional embeddings. TYPE: HubertPositionalConvEmbedding
layer_norm	Layer normalization module. TYPE: LayerNorm
dropout	Dropout module for regularization. TYPE: Dropout
layers	List of HubertEncoderLayer instances representing the encoder layers. TYPE: ModuleList

METHOD	DESCRIPTION
forward	Processes the input hidden states through the encoder layers and returns the final hidden states along with optional hidden states and attentions.

PARAMETER	DESCRIPTION
hidden_states	The input hidden states to be processed by the encoder. TYPE: Tensor
attention_mask	Optional attention mask to mask out specific tokens during processing. TYPE: Optional[Tensor]
output_attentions	Flag indicating whether to output attention weights. TYPE: bool
output_hidden_states	Flag indicating whether to output hidden states of each layer. TYPE: bool
return_dict	Flag indicating whether to return the output as a BaseModelOutput dictionary. TYPE: bool

RETURNS	DESCRIPTION
	BaseModelOutput or tuple: A BaseModelOutput object containing the last hidden state, hidden states of all layers, and attention weights, or a tuple containing these elements based on the value of 'return_dict'.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

class HubertEncoder(nn.Module):

    """
    A class representing the encoder component of the Hubert model.
    This class is responsible for processing input hidden states through multiple layers of HubertEncoderLayer.

    This class inherits from nn.Module.

    Attributes:
        config (HubertConfig): The configuration object for the Hubert model.
        pos_conv_embed (HubertPositionalConvEmbedding):
            Instance of HubertPositionalConvEmbedding for positional convolutional embeddings.
        layer_norm (nn.LayerNorm): Layer normalization module.
        dropout (nn.Dropout): Dropout module for regularization.
        layers (nn.ModuleList): List of HubertEncoderLayer instances representing the encoder layers.

    Methods:
        forward(hidden_states, attention_mask, output_attentions, output_hidden_states, return_dict):
            Processes the input hidden states through the encoder layers and returns the final hidden states
            along with optional hidden states and attentions.

    Args:
        hidden_states (Tensor): The input hidden states to be processed by the encoder.
        attention_mask (Optional[Tensor]): Optional attention mask to mask out specific tokens during processing.
        output_attentions (bool): Flag indicating whether to output attention weights.
        output_hidden_states (bool): Flag indicating whether to output hidden states of each layer.
        return_dict (bool): Flag indicating whether to return the output as a BaseModelOutput dictionary.

    Returns:
        BaseModelOutput or tuple: A BaseModelOutput object containing the last hidden state, hidden states of all layers,
            and attention weights, or a tuple containing these elements based on the value of 'return_dict'.

    """
    def __init__(self, config: HubertConfig):
        """
        Initializes an instance of the HubertEncoder class.

        Args:
            self: The instance of the class.
            config (HubertConfig):
                The configuration object containing various settings for the HubertEncoder.

                - type: HubertConfig
                - purpose: It provides necessary configurations for initializing the encoder.
                - restrictions: Must be an instance of HubertConfig.

        Returns:
            None

        Raises:
            None
        """
        super().__init__()
        self.config = config
        self.pos_conv_embed = HubertPositionalConvEmbedding(config)
        self.layer_norm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
        self.dropout = nn.Dropout(p=config.hidden_dropout)
        self.layers = nn.ModuleList([HubertEncoderLayer(config) for _ in range(config.num_hidden_layers)])

    def forward(
        self,
        hidden_states: Tensor,
        attention_mask: Optional[Tensor] = None,
        output_attentions: bool = False,
        output_hidden_states: bool = False,
        return_dict: bool = True,
    ):
        """
        This method forwards the Hubert encoder using the provided parameters and returns the final output.

        Args:
            self: The instance of the class.
            hidden_states (Tensor): The hidden states input tensor for the encoder.
            attention_mask (Optional[Tensor]):
                Optional attention mask tensor to mask certain elements in the hidden states. Default is None.
            output_attentions (bool): Flag indicating whether to output attentions. Default is False.
            output_hidden_states (bool): Flag indicating whether to output hidden states. Default is False.
            return_dict (bool): Flag indicating whether to return the output as a dictionary. Default is True.

        Returns:
            None: This method does not return any value explicitly,
                as it updates hidden states and hidden state-related outputs within the class instance.

        Raises:
            ValueError: If the attention mask dimensions are incompatible with the hidden states tensor.
            RuntimeError: If there is an issue during the execution of the encoder layers.
            TypeError: If the input types are incorrect or incompatible with the expected types.
        """
        all_hidden_states = () if output_hidden_states else None
        all_self_attentions = () if output_attentions else None

        if attention_mask is not None:
            # make sure padded tokens output 0
            expand_attention_mask = attention_mask.unsqueeze(-1).repeat(1, 1, hidden_states.shape[2])
            hidden_states[~expand_attention_mask] = 0

            # extend attention_mask
            attention_mask = 1.0 - attention_mask[:, None, None, :].to(dtype=hidden_states.dtype)
            attention_mask = attention_mask * finfo(hidden_states.dtype, 'min')
            attention_mask = attention_mask.expand(
                attention_mask.shape[0], 1, attention_mask.shape[-1], attention_mask.shape[-1]
            )

        position_embeddings = self.pos_conv_embed(hidden_states)
        hidden_states = hidden_states + position_embeddings
        hidden_states = self.layer_norm(hidden_states)
        hidden_states = self.dropout(hidden_states)

        for layer in self.layers:
            if output_hidden_states:
                all_hidden_states = all_hidden_states + (hidden_states,)

            # add LayerDrop (see https://arxiv.org/abs/1909.11556 for description)
            dropout_probability = ops.rand([])

            skip_the_layer = self.training and (dropout_probability < self.config.layerdrop)
            if not skip_the_layer:
                layer_outputs = layer(hidden_states, attention_mask=attention_mask, output_attentions=output_attentions)
                hidden_states = layer_outputs[0]

            if skip_the_layer:
                layer_outputs = (None, None)

            if output_attentions:
                all_self_attentions = all_self_attentions + (layer_outputs[1],)

        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        if not return_dict:
            return tuple(v for v in [hidden_states, all_hidden_states, all_self_attentions] if v is not None)

        return BaseModelOutput(
            last_hidden_state=hidden_states,
            hidden_states=all_hidden_states,
            attentions=all_self_attentions,
        )

mindnlp.transformers.models.hubert.modeling_hubert.HubertEncoder.__init__(config)

Initializes an instance of the HubertEncoder class.

PARAMETER	DESCRIPTION
self	The instance of the class.
config	The configuration object containing various settings for the HubertEncoder. type: HubertConfig purpose: It provides necessary configurations for initializing the encoder. restrictions: Must be an instance of HubertConfig. TYPE: HubertConfig

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def __init__(self, config: HubertConfig):
    """
    Initializes an instance of the HubertEncoder class.

    Args:
        self: The instance of the class.
        config (HubertConfig):
            The configuration object containing various settings for the HubertEncoder.

            - type: HubertConfig
            - purpose: It provides necessary configurations for initializing the encoder.
            - restrictions: Must be an instance of HubertConfig.

    Returns:
        None

    Raises:
        None
    """
    super().__init__()
    self.config = config
    self.pos_conv_embed = HubertPositionalConvEmbedding(config)
    self.layer_norm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
    self.dropout = nn.Dropout(p=config.hidden_dropout)
    self.layers = nn.ModuleList([HubertEncoderLayer(config) for _ in range(config.num_hidden_layers)])

mindnlp.transformers.models.hubert.modeling_hubert.HubertEncoder.forward(hidden_states, attention_mask=None, output_attentions=False, output_hidden_states=False, return_dict=True)

This method forwards the Hubert encoder using the provided parameters and returns the final output.

PARAMETER	DESCRIPTION
self	The instance of the class.
hidden_states	The hidden states input tensor for the encoder. TYPE: Tensor
attention_mask	Optional attention mask tensor to mask certain elements in the hidden states. Default is None. TYPE: Optional[Tensor] DEFAULT: None
output_attentions	Flag indicating whether to output attentions. Default is False. TYPE: bool DEFAULT: False
output_hidden_states	Flag indicating whether to output hidden states. Default is False. TYPE: bool DEFAULT: False
return_dict	Flag indicating whether to return the output as a dictionary. Default is True. TYPE: bool DEFAULT: True

RETURNS	DESCRIPTION
None	This method does not return any value explicitly, as it updates hidden states and hidden state-related outputs within the class instance.

RAISES	DESCRIPTION
ValueError	If the attention mask dimensions are incompatible with the hidden states tensor.
RuntimeError	If there is an issue during the execution of the encoder layers.
TypeError	If the input types are incorrect or incompatible with the expected types.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def forward(
    self,
    hidden_states: Tensor,
    attention_mask: Optional[Tensor] = None,
    output_attentions: bool = False,
    output_hidden_states: bool = False,
    return_dict: bool = True,
):
    """
    This method forwards the Hubert encoder using the provided parameters and returns the final output.

    Args:
        self: The instance of the class.
        hidden_states (Tensor): The hidden states input tensor for the encoder.
        attention_mask (Optional[Tensor]):
            Optional attention mask tensor to mask certain elements in the hidden states. Default is None.
        output_attentions (bool): Flag indicating whether to output attentions. Default is False.
        output_hidden_states (bool): Flag indicating whether to output hidden states. Default is False.
        return_dict (bool): Flag indicating whether to return the output as a dictionary. Default is True.

    Returns:
        None: This method does not return any value explicitly,
            as it updates hidden states and hidden state-related outputs within the class instance.

    Raises:
        ValueError: If the attention mask dimensions are incompatible with the hidden states tensor.
        RuntimeError: If there is an issue during the execution of the encoder layers.
        TypeError: If the input types are incorrect or incompatible with the expected types.
    """
    all_hidden_states = () if output_hidden_states else None
    all_self_attentions = () if output_attentions else None

    if attention_mask is not None:
        # make sure padded tokens output 0
        expand_attention_mask = attention_mask.unsqueeze(-1).repeat(1, 1, hidden_states.shape[2])
        hidden_states[~expand_attention_mask] = 0

        # extend attention_mask
        attention_mask = 1.0 - attention_mask[:, None, None, :].to(dtype=hidden_states.dtype)
        attention_mask = attention_mask * finfo(hidden_states.dtype, 'min')
        attention_mask = attention_mask.expand(
            attention_mask.shape[0], 1, attention_mask.shape[-1], attention_mask.shape[-1]
        )

    position_embeddings = self.pos_conv_embed(hidden_states)
    hidden_states = hidden_states + position_embeddings
    hidden_states = self.layer_norm(hidden_states)
    hidden_states = self.dropout(hidden_states)

    for layer in self.layers:
        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        # add LayerDrop (see https://arxiv.org/abs/1909.11556 for description)
        dropout_probability = ops.rand([])

        skip_the_layer = self.training and (dropout_probability < self.config.layerdrop)
        if not skip_the_layer:
            layer_outputs = layer(hidden_states, attention_mask=attention_mask, output_attentions=output_attentions)
            hidden_states = layer_outputs[0]

        if skip_the_layer:
            layer_outputs = (None, None)

        if output_attentions:
            all_self_attentions = all_self_attentions + (layer_outputs[1],)

    if output_hidden_states:
        all_hidden_states = all_hidden_states + (hidden_states,)

    if not return_dict:
        return tuple(v for v in [hidden_states, all_hidden_states, all_self_attentions] if v is not None)

    return BaseModelOutput(
        last_hidden_state=hidden_states,
        hidden_states=all_hidden_states,
        attentions=all_self_attentions,
    )

mindnlp.transformers.models.hubert.modeling_hubert.HubertEncoderLayer

Bases: Module

HubertEncoderLayer represents a single layer of the HubertEncoder.

This class inherits from nn.Module and contains methods to initialize the layer and forward the layer. The init method initializes the layer with the given configuration, while the forward method applies the attention mechanism, dropout, layer normalization, feed forward, and final layer normalization to the input hidden states.

ATTRIBUTE	DESCRIPTION
attention	An instance of HubertAttention representing the attention mechanism with specified parameters.
dropout	An instance of nn.Dropout representing the dropout layer with a specified dropout rate.
layer_norm	An instance of nn.LayerNorm representing the layer normalization with a specified epsilon.
feed_forward	An instance of HubertFeedForward representing the feed forward layer with the given configuration.
final_layer_norm	An instance of nn.LayerNorm representing the final layer normalization with a specified epsilon.

METHOD	DESCRIPTION
__init__	Initializes the HubertEncoderLayer instance with the given configuration.
forward	Applies the attention mechanism, dropout, layer normalization, feed forward, and final

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

class HubertEncoderLayer(nn.Module):

    '''
    HubertEncoderLayer represents a single layer of the HubertEncoder.

    This class inherits from nn.Module and contains methods to initialize the layer and forward the layer.
    The __init__ method initializes the layer with the given configuration, while the forward method applies the
    attention mechanism, dropout, layer normalization, feed forward, and final layer normalization to the input
    hidden states.

    Attributes:
        attention: An instance of HubertAttention representing the attention mechanism with specified parameters.
        dropout: An instance of nn.Dropout representing the dropout layer with a specified dropout rate.
        layer_norm: An instance of nn.LayerNorm representing the layer normalization with a specified epsilon.
        feed_forward: An instance of HubertFeedForward representing the feed forward layer with the given configuration.
        final_layer_norm: An instance of nn.LayerNorm representing the final layer normalization with a specified epsilon.

    Methods:
        __init__: Initializes the HubertEncoderLayer instance with the given configuration.
        forward: Applies the attention mechanism, dropout, layer normalization, feed forward, and final
        layer normalization to the input hidden states.

    '''
    def __init__(self, config: HubertConfig):
        """
        Initializes a HubertEncoderLayer instance.

        Args:
            self: The instance of HubertEncoderLayer.
            config (HubertConfig):
                An instance of HubertConfig containing configuration parameters.

                - config.hidden_size (int): The size of hidden layers.
                - config.num_attention_heads (int): The number of attention heads.
                - config.attention_dropout (float): The dropout probability for attention layers.
                - config.hidden_dropout (float): The dropout probability for hidden layers.
                - config.layer_norm_eps (float): The epsilon value for layer normalization.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.attention = HubertAttention(
            embed_dim=config.hidden_size,
            num_heads=config.num_attention_heads,
            dropout=config.attention_dropout,
            is_decoder=False,
        )
        self.dropout = nn.Dropout(p=config.hidden_dropout)
        self.layer_norm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
        self.feed_forward = HubertFeedForward(config)
        self.final_layer_norm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)

    def forward(self, hidden_states, attention_mask=None, output_attentions=False):
        """
        Method to forward the Hubert Encoder Layer.

        Args:
            self: Reference to the instance of the class.
            hidden_states (Tensor): Input hidden states to be processed.
            attention_mask (Tensor, optional): Mask to avoid attending over padding tokens. Default is None.
            output_attentions (bool, optional): Flag to indicate whether to output attention weights. Default is False.

        Returns:
            Tuple: A tuple containing the processed hidden states.
                If output_attentions is True, the tuple also includes the attention weights.

        Raises:
            None
        """
        attn_residual = hidden_states
        hidden_states, attn_weights, _ = self.attention(
            hidden_states, attention_mask=attention_mask, output_attentions=output_attentions
        )
        hidden_states = self.dropout(hidden_states)
        hidden_states = attn_residual + hidden_states

        hidden_states = self.layer_norm(hidden_states)
        hidden_states = hidden_states + self.feed_forward(hidden_states)
        hidden_states = self.final_layer_norm(hidden_states)

        outputs = (hidden_states,)
        if output_attentions:
            outputs += (attn_weights,)
        return outputs

mindnlp.transformers.models.hubert.modeling_hubert.HubertEncoderLayer.__init__(config)

Initializes a HubertEncoderLayer instance.

PARAMETER	DESCRIPTION
self	The instance of HubertEncoderLayer.
config	An instance of HubertConfig containing configuration parameters. config.hidden_size (int): The size of hidden layers. config.num_attention_heads (int): The number of attention heads. config.attention_dropout (float): The dropout probability for attention layers. config.hidden_dropout (float): The dropout probability for hidden layers. config.layer_norm_eps (float): The epsilon value for layer normalization. TYPE: HubertConfig

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def __init__(self, config: HubertConfig):
    """
    Initializes a HubertEncoderLayer instance.

    Args:
        self: The instance of HubertEncoderLayer.
        config (HubertConfig):
            An instance of HubertConfig containing configuration parameters.

            - config.hidden_size (int): The size of hidden layers.
            - config.num_attention_heads (int): The number of attention heads.
            - config.attention_dropout (float): The dropout probability for attention layers.
            - config.hidden_dropout (float): The dropout probability for hidden layers.
            - config.layer_norm_eps (float): The epsilon value for layer normalization.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.attention = HubertAttention(
        embed_dim=config.hidden_size,
        num_heads=config.num_attention_heads,
        dropout=config.attention_dropout,
        is_decoder=False,
    )
    self.dropout = nn.Dropout(p=config.hidden_dropout)
    self.layer_norm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
    self.feed_forward = HubertFeedForward(config)
    self.final_layer_norm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)

mindnlp.transformers.models.hubert.modeling_hubert.HubertEncoderLayer.forward(hidden_states, attention_mask=None, output_attentions=False)

Method to forward the Hubert Encoder Layer.

PARAMETER	DESCRIPTION
self	Reference to the instance of the class.
hidden_states	Input hidden states to be processed. TYPE: Tensor
attention_mask	Mask to avoid attending over padding tokens. Default is None. TYPE: Tensor DEFAULT: None
output_attentions	Flag to indicate whether to output attention weights. Default is False. TYPE: bool DEFAULT: False

RETURNS	DESCRIPTION
Tuple	A tuple containing the processed hidden states. If output_attentions is True, the tuple also includes the attention weights.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def forward(self, hidden_states, attention_mask=None, output_attentions=False):
    """
    Method to forward the Hubert Encoder Layer.

    Args:
        self: Reference to the instance of the class.
        hidden_states (Tensor): Input hidden states to be processed.
        attention_mask (Tensor, optional): Mask to avoid attending over padding tokens. Default is None.
        output_attentions (bool, optional): Flag to indicate whether to output attention weights. Default is False.

    Returns:
        Tuple: A tuple containing the processed hidden states.
            If output_attentions is True, the tuple also includes the attention weights.

    Raises:
        None
    """
    attn_residual = hidden_states
    hidden_states, attn_weights, _ = self.attention(
        hidden_states, attention_mask=attention_mask, output_attentions=output_attentions
    )
    hidden_states = self.dropout(hidden_states)
    hidden_states = attn_residual + hidden_states

    hidden_states = self.layer_norm(hidden_states)
    hidden_states = hidden_states + self.feed_forward(hidden_states)
    hidden_states = self.final_layer_norm(hidden_states)

    outputs = (hidden_states,)
    if output_attentions:
        outputs += (attn_weights,)
    return outputs

mindnlp.transformers.models.hubert.modeling_hubert.HubertEncoderLayerStableLayerNorm

Bases: Module

Represents a stable layer normalization encoder layer for the Hubert model.

This class inherits from nn.Module and contains methods for initializing the layer and forwarding the layer with attention and feed-forward operations. It also includes an optional adapter layer.

ATTRIBUTE	DESCRIPTION
attention	HubertAttention The attention mechanism for the encoder layer.
dropout	nn.Dropout The dropout layer for the encoder layer.
layer_norm	nn.LayerNorm The layer normalization for the encoder layer.
feed_forward	HubertFeedForward The feed-forward network for the encoder layer.
final_layer_norm	nn.LayerNorm The final layer normalization for the encoder layer.
adapter_layer	HubertAttnAdapterLayer or None The optional adapter layer for the encoder layer.

METHOD	DESCRIPTION
__init__	Initializes the encoder layer with the provided configuration.
forward	Constructs the encoder layer with attention and feed-forward operations, and an optional adapter layer.

RETURNS	DESCRIPTION
outputs	Tuple[Tensor, ...] The outputs of the encoder layer, including hidden states and optionally attention weights.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

class HubertEncoderLayerStableLayerNorm(nn.Module):

    """
    Represents a stable layer normalization encoder layer for the Hubert model.

    This class inherits from nn.Module and contains methods for initializing the layer and forwarding the layer
    with attention and feed-forward operations. It also includes an optional adapter layer.

    Attributes:
        attention: HubertAttention
            The attention mechanism for the encoder layer.
        dropout: nn.Dropout
            The dropout layer for the encoder layer.
        layer_norm: nn.LayerNorm
            The layer normalization for the encoder layer.
        feed_forward: HubertFeedForward
            The feed-forward network for the encoder layer.
        final_layer_norm: nn.LayerNorm
            The final layer normalization for the encoder layer.
        adapter_layer: HubertAttnAdapterLayer or None
            The optional adapter layer for the encoder layer.

    Methods:
        __init__:
            Initializes the encoder layer with the provided configuration.
        forward:
            Constructs the encoder layer with attention and feed-forward operations, and an optional adapter layer.

    Returns:
        outputs: Tuple[Tensor, ...]
            The outputs of the encoder layer, including hidden states and optionally attention weights.
    """
    def __init__(self, config: HubertConfig):
        """
        Initializes a HubertEncoderLayerStableLayerNorm instance.

        Args:
            self: The instance of the HubertEncoderLayerStableLayerNorm class.
            config (HubertConfig):
                An instance of the HubertConfig class containing configuration parameters for the encoder layer.

                Parameters:

                - embed_dim (int): The dimension of the embedding.
                - num_heads (int): The number of attention heads.
                - dropout (float): The dropout probability for attention.
                - hidden_dropout (float): The dropout probability for hidden layers.
                - layer_norm_eps (float): The epsilon value for layer normalization.
                - adapter_attn_dim (int, optional): The dimension of the attention adapter layer. Defaults to None.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.attention = HubertAttention(
            embed_dim=config.hidden_size,
            num_heads=config.num_attention_heads,
            dropout=config.attention_dropout,
            is_decoder=False,
        )
        self.dropout = nn.Dropout(p=config.hidden_dropout)
        self.layer_norm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
        self.feed_forward = HubertFeedForward(config)
        self.final_layer_norm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)

        if getattr(config, "adapter_attn_dim", None) is not None:
            self.adapter_layer = HubertAttnAdapterLayer(config)
        else:
            self.adapter_layer = None

    def forward(
        self,
        hidden_states: Tensor,
        attention_mask: Optional[Tensor] = None,
        output_attentions: bool = False,
    ):
        """
        This method forwards the Hubert encoder layer with stable layer normalization.

        Args:
            self: The instance of the HubertEncoderLayerStableLayerNorm class.
            hidden_states (Tensor): The input tensor representing the hidden states.
                This parameter is required for the forwardion of the encoder layer.
            attention_mask (Optional[Tensor]): An optional tensor representing the attention mask.
                It defaults to None and is used to mask padded tokens during attention computation.
            output_attentions (bool): A boolean flag indicating whether to output the attention weights.
                It defaults to False and is used to control whether the attention weights should be returned.

        Returns:
            tuple: A tuple containing the forwarded hidden states. If output_attentions is True, the tuple also contains
                the attention weights.

        Raises:
            None
        """
        attn_residual = hidden_states
        hidden_states = self.layer_norm(hidden_states)
        hidden_states, attn_weights, _ = self.attention(
            hidden_states, attention_mask=attention_mask, output_attentions=output_attentions
        )
        hidden_states = self.dropout(hidden_states)
        hidden_states = attn_residual + hidden_states
        hidden_states = hidden_states + self.feed_forward(self.final_layer_norm(hidden_states))

        if self.adapter_layer is not None:
            hidden_states = hidden_states + self.adapter_layer(hidden_states)

        outputs = (hidden_states,)
        if output_attentions:
            outputs += (attn_weights,)
        return outputs

mindnlp.transformers.models.hubert.modeling_hubert.HubertEncoderLayerStableLayerNorm.__init__(config)

Initializes a HubertEncoderLayerStableLayerNorm instance.

PARAMETER	DESCRIPTION
self	The instance of the HubertEncoderLayerStableLayerNorm class.
config	An instance of the HubertConfig class containing configuration parameters for the encoder layer. Parameters: embed_dim (int): The dimension of the embedding. num_heads (int): The number of attention heads. dropout (float): The dropout probability for attention. hidden_dropout (float): The dropout probability for hidden layers. layer_norm_eps (float): The epsilon value for layer normalization. adapter_attn_dim (int, optional): The dimension of the attention adapter layer. Defaults to None. TYPE: HubertConfig

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def __init__(self, config: HubertConfig):
    """
    Initializes a HubertEncoderLayerStableLayerNorm instance.

    Args:
        self: The instance of the HubertEncoderLayerStableLayerNorm class.
        config (HubertConfig):
            An instance of the HubertConfig class containing configuration parameters for the encoder layer.

            Parameters:

            - embed_dim (int): The dimension of the embedding.
            - num_heads (int): The number of attention heads.
            - dropout (float): The dropout probability for attention.
            - hidden_dropout (float): The dropout probability for hidden layers.
            - layer_norm_eps (float): The epsilon value for layer normalization.
            - adapter_attn_dim (int, optional): The dimension of the attention adapter layer. Defaults to None.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.attention = HubertAttention(
        embed_dim=config.hidden_size,
        num_heads=config.num_attention_heads,
        dropout=config.attention_dropout,
        is_decoder=False,
    )
    self.dropout = nn.Dropout(p=config.hidden_dropout)
    self.layer_norm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
    self.feed_forward = HubertFeedForward(config)
    self.final_layer_norm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)

    if getattr(config, "adapter_attn_dim", None) is not None:
        self.adapter_layer = HubertAttnAdapterLayer(config)
    else:
        self.adapter_layer = None

mindnlp.transformers.models.hubert.modeling_hubert.HubertEncoderLayerStableLayerNorm.forward(hidden_states, attention_mask=None, output_attentions=False)

This method forwards the Hubert encoder layer with stable layer normalization.

PARAMETER	DESCRIPTION
self	The instance of the HubertEncoderLayerStableLayerNorm class.
hidden_states	The input tensor representing the hidden states. This parameter is required for the forwardion of the encoder layer. TYPE: Tensor
attention_mask	An optional tensor representing the attention mask. It defaults to None and is used to mask padded tokens during attention computation. TYPE: Optional[Tensor] DEFAULT: None
output_attentions	A boolean flag indicating whether to output the attention weights. It defaults to False and is used to control whether the attention weights should be returned. TYPE: bool DEFAULT: False

RETURNS	DESCRIPTION
tuple	A tuple containing the forwarded hidden states. If output_attentions is True, the tuple also contains the attention weights.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def forward(
    self,
    hidden_states: Tensor,
    attention_mask: Optional[Tensor] = None,
    output_attentions: bool = False,
):
    """
    This method forwards the Hubert encoder layer with stable layer normalization.

    Args:
        self: The instance of the HubertEncoderLayerStableLayerNorm class.
        hidden_states (Tensor): The input tensor representing the hidden states.
            This parameter is required for the forwardion of the encoder layer.
        attention_mask (Optional[Tensor]): An optional tensor representing the attention mask.
            It defaults to None and is used to mask padded tokens during attention computation.
        output_attentions (bool): A boolean flag indicating whether to output the attention weights.
            It defaults to False and is used to control whether the attention weights should be returned.

    Returns:
        tuple: A tuple containing the forwarded hidden states. If output_attentions is True, the tuple also contains
            the attention weights.

    Raises:
        None
    """
    attn_residual = hidden_states
    hidden_states = self.layer_norm(hidden_states)
    hidden_states, attn_weights, _ = self.attention(
        hidden_states, attention_mask=attention_mask, output_attentions=output_attentions
    )
    hidden_states = self.dropout(hidden_states)
    hidden_states = attn_residual + hidden_states
    hidden_states = hidden_states + self.feed_forward(self.final_layer_norm(hidden_states))

    if self.adapter_layer is not None:
        hidden_states = hidden_states + self.adapter_layer(hidden_states)

    outputs = (hidden_states,)
    if output_attentions:
        outputs += (attn_weights,)
    return outputs

mindnlp.transformers.models.hubert.modeling_hubert.HubertEncoderStableLayerNorm

Bases: Module

Class representing a Hubert encoder with stable layer normalization.

This class implements an encoder model for the Hubert architecture with stable layer normalization. The encoder consists of multiple layers, each containing positional convolutional embeddings, layer normalization, and dropout, followed by a series of encoder layers. The encoder can process input hidden states, apply attention masks, and optionally output hidden states and self-attentions.

ATTRIBUTE	DESCRIPTION
config	The configuration object for the Hubert model. TYPE: HubertConfig
pos_conv_embed	Positional convolutional embedding layer. TYPE: HubertPositionalConvEmbedding
layer_norm	Layer normalization for the hidden states. TYPE: LayerNorm
dropout	Dropout layer. TYPE: Dropout
layers	List of encoder layers for processing the hidden states. TYPE: ModuleList

METHOD

DESCRIPTION

forward

Processes the input hidden states through the encoder layers. Args: hidden_states (Tensor): Input hidden states to be processed. attention_mask (Tensor, optional): Mask for attention scores. Defaults to None. output_attentions (bool, optional): Whether to output self-attention matrices. Defaults to False. output_hidden_states (bool, optional): Whether to output hidden states of each layer. Defaults to False. return_dict (bool, optional): Whether to return the output as a dictionary. Defaults to True. Returns: BaseModelOutput: Object containing the last hidden state, hidden states of all layers, and self-attention matrices.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

class HubertEncoderStableLayerNorm(nn.Module):

    """
    Class representing a Hubert encoder with stable layer normalization.

    This class implements an encoder model for the Hubert architecture with stable layer normalization.
    The encoder consists of multiple layers, each containing positional convolutional embeddings,
    layer normalization, and dropout, followed by a series of encoder layers.
    The encoder can process input hidden states, apply attention masks,
    and optionally output hidden states and self-attentions.

    Attributes:
        config (HubertConfig): The configuration object for the Hubert model.
        pos_conv_embed (HubertPositionalConvEmbedding): Positional convolutional embedding layer.
        layer_norm (nn.LayerNorm): Layer normalization for the hidden states.
        dropout (nn.Dropout): Dropout layer.
        layers (nn.ModuleList): List of encoder layers for processing the hidden states.

    Methods:
        forward(self, hidden_states, attention_mask=None, output_attentions=False, output_hidden_states=False, return_dict=True):
            Processes the input hidden states through the encoder layers.

            Args:

            - hidden_states (Tensor): Input hidden states to be processed.
            - attention_mask (Tensor, optional): Mask for attention scores. Defaults to None.
            - output_attentions (bool, optional): Whether to output self-attention matrices. Defaults to False.
            - output_hidden_states (bool, optional): Whether to output hidden states of each layer. Defaults to False.
            - return_dict (bool, optional): Whether to return the output as a dictionary. Defaults to True.

            Returns:

            - BaseModelOutput: Object containing the last hidden state, hidden states of all layers, and self-attention matrices.
    """
    def __init__(self, config: HubertConfig):
        """
        Initializes an instance of the HubertEncoderStableLayerNorm class.

        Args:
            self: The instance of the HubertEncoderStableLayerNorm class.
            config (HubertConfig): An instance of HubertConfig containing configuration parameters for the encoder.
                This parameter specifies the configuration settings for the encoder.
                It is a required parameter and must be of type HubertConfig.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.config = config
        self.pos_conv_embed = HubertPositionalConvEmbedding(config)
        self.layer_norm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
        self.dropout = nn.Dropout(p=config.hidden_dropout)
        self.layers = nn.ModuleList([HubertEncoderLayerStableLayerNorm(config) for _ in range(config.num_hidden_layers)])

    def forward(
        self,
        hidden_states,
        attention_mask=None,
        output_attentions=False,
        output_hidden_states=False,
        return_dict=True,
    ):
        """
        Constructs the Hubert encoder stable layer norm.

        Args:
            self (HubertEncoderStableLayerNorm): The object instance.
            hidden_states (torch.Tensor): The input hidden states of shape (batch_size, sequence_length, hidden_size).
            attention_mask (torch.Tensor, optional):
                The attention mask of shape (batch_size, sequence_length) or (batch_size, 1, 1, sequence_length)
                indicating which tokens should be attended to. Defaults to None.
            output_attentions (bool, optional): Whether to return the attentions. Defaults to False.
            output_hidden_states (bool, optional): Whether to return the hidden states. Defaults to False.
            return_dict (bool, optional): Whether to return a dictionary instead of a BaseModelOutput. Defaults to True.

        Returns:
            None:
                This method does not return any value. It operates in place on the hidden_states
                and other internal buffers.

        Raises:
            ValueError: If the shapes of the input tensors are not compatible or if there are issues in the
                internal computations.
            RuntimeError: If there are errors during the computation or if the method is called in an invalid state.
        """
        all_hidden_states = () if output_hidden_states else None
        all_self_attentions = () if output_attentions else None

        if attention_mask is not None:
            # make sure padded tokens are not attended to
            expand_attention_mask = attention_mask.unsqueeze(-1).repeat(1, 1, hidden_states.shape[2])
            hidden_states[~expand_attention_mask] = 0

            # extend attention_mask
            attention_mask = 1.0 - attention_mask[:, None, None, :].to(dtype=hidden_states.dtype)
            attention_mask = attention_mask * finfo(hidden_states.dtype, 'min')
            attention_mask = attention_mask.expand(
                attention_mask.shape[0], 1, attention_mask.shape[-1], attention_mask.shape[-1]
            )

        position_embeddings = self.pos_conv_embed(hidden_states)
        hidden_states = hidden_states + position_embeddings
        hidden_states = self.dropout(hidden_states)

        for layer in self.layers:
            if output_hidden_states:
                all_hidden_states = all_hidden_states + (hidden_states,)

            # add LayerDrop (see https://arxiv.org/abs/1909.11556 for description)
            dropout_probability =ops.rand([])

            skip_the_layer = self.training and (dropout_probability < self.config.layerdrop)
            if not skip_the_layer:
                layer_outputs = layer(hidden_states, attention_mask=attention_mask, output_attentions=output_attentions)
                hidden_states = layer_outputs[0]

            if skip_the_layer:
                layer_outputs = (None, None)

            if output_attentions:
                all_self_attentions = all_self_attentions + (layer_outputs[1],)

        hidden_states = self.layer_norm(hidden_states)

        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        if not return_dict:
            return tuple(v for v in [hidden_states, all_hidden_states, all_self_attentions] if v is not None)

        return BaseModelOutput(
            last_hidden_state=hidden_states,
            hidden_states=all_hidden_states,
            attentions=all_self_attentions,
        )

mindnlp.transformers.models.hubert.modeling_hubert.HubertEncoderStableLayerNorm.__init__(config)

Initializes an instance of the HubertEncoderStableLayerNorm class.

PARAMETER	DESCRIPTION
self	The instance of the HubertEncoderStableLayerNorm class.
config	An instance of HubertConfig containing configuration parameters for the encoder. This parameter specifies the configuration settings for the encoder. It is a required parameter and must be of type HubertConfig. TYPE: HubertConfig

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def __init__(self, config: HubertConfig):
    """
    Initializes an instance of the HubertEncoderStableLayerNorm class.

    Args:
        self: The instance of the HubertEncoderStableLayerNorm class.
        config (HubertConfig): An instance of HubertConfig containing configuration parameters for the encoder.
            This parameter specifies the configuration settings for the encoder.
            It is a required parameter and must be of type HubertConfig.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.config = config
    self.pos_conv_embed = HubertPositionalConvEmbedding(config)
    self.layer_norm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
    self.dropout = nn.Dropout(p=config.hidden_dropout)
    self.layers = nn.ModuleList([HubertEncoderLayerStableLayerNorm(config) for _ in range(config.num_hidden_layers)])

mindnlp.transformers.models.hubert.modeling_hubert.HubertEncoderStableLayerNorm.forward(hidden_states, attention_mask=None, output_attentions=False, output_hidden_states=False, return_dict=True)

Constructs the Hubert encoder stable layer norm.

PARAMETER	DESCRIPTION
self	The object instance. TYPE: HubertEncoderStableLayerNorm
hidden_states	The input hidden states of shape (batch_size, sequence_length, hidden_size). TYPE: Tensor
attention_mask	The attention mask of shape (batch_size, sequence_length) or (batch_size, 1, 1, sequence_length) indicating which tokens should be attended to. Defaults to None. TYPE: Tensor DEFAULT: None
output_attentions	Whether to return the attentions. Defaults to False. TYPE: bool DEFAULT: False
output_hidden_states	Whether to return the hidden states. Defaults to False. TYPE: bool DEFAULT: False
return_dict	Whether to return a dictionary instead of a BaseModelOutput. Defaults to True. TYPE: bool DEFAULT: True

RETURNS	DESCRIPTION
None	This method does not return any value. It operates in place on the hidden_states and other internal buffers.

RAISES	DESCRIPTION
ValueError	If the shapes of the input tensors are not compatible or if there are issues in the internal computations.
RuntimeError	If there are errors during the computation or if the method is called in an invalid state.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def forward(
    self,
    hidden_states,
    attention_mask=None,
    output_attentions=False,
    output_hidden_states=False,
    return_dict=True,
):
    """
    Constructs the Hubert encoder stable layer norm.

    Args:
        self (HubertEncoderStableLayerNorm): The object instance.
        hidden_states (torch.Tensor): The input hidden states of shape (batch_size, sequence_length, hidden_size).
        attention_mask (torch.Tensor, optional):
            The attention mask of shape (batch_size, sequence_length) or (batch_size, 1, 1, sequence_length)
            indicating which tokens should be attended to. Defaults to None.
        output_attentions (bool, optional): Whether to return the attentions. Defaults to False.
        output_hidden_states (bool, optional): Whether to return the hidden states. Defaults to False.
        return_dict (bool, optional): Whether to return a dictionary instead of a BaseModelOutput. Defaults to True.

    Returns:
        None:
            This method does not return any value. It operates in place on the hidden_states
            and other internal buffers.

    Raises:
        ValueError: If the shapes of the input tensors are not compatible or if there are issues in the
            internal computations.
        RuntimeError: If there are errors during the computation or if the method is called in an invalid state.
    """
    all_hidden_states = () if output_hidden_states else None
    all_self_attentions = () if output_attentions else None

    if attention_mask is not None:
        # make sure padded tokens are not attended to
        expand_attention_mask = attention_mask.unsqueeze(-1).repeat(1, 1, hidden_states.shape[2])
        hidden_states[~expand_attention_mask] = 0

        # extend attention_mask
        attention_mask = 1.0 - attention_mask[:, None, None, :].to(dtype=hidden_states.dtype)
        attention_mask = attention_mask * finfo(hidden_states.dtype, 'min')
        attention_mask = attention_mask.expand(
            attention_mask.shape[0], 1, attention_mask.shape[-1], attention_mask.shape[-1]
        )

    position_embeddings = self.pos_conv_embed(hidden_states)
    hidden_states = hidden_states + position_embeddings
    hidden_states = self.dropout(hidden_states)

    for layer in self.layers:
        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        # add LayerDrop (see https://arxiv.org/abs/1909.11556 for description)
        dropout_probability =ops.rand([])

        skip_the_layer = self.training and (dropout_probability < self.config.layerdrop)
        if not skip_the_layer:
            layer_outputs = layer(hidden_states, attention_mask=attention_mask, output_attentions=output_attentions)
            hidden_states = layer_outputs[0]

        if skip_the_layer:
            layer_outputs = (None, None)

        if output_attentions:
            all_self_attentions = all_self_attentions + (layer_outputs[1],)

    hidden_states = self.layer_norm(hidden_states)

    if output_hidden_states:
        all_hidden_states = all_hidden_states + (hidden_states,)

    if not return_dict:
        return tuple(v for v in [hidden_states, all_hidden_states, all_self_attentions] if v is not None)

    return BaseModelOutput(
        last_hidden_state=hidden_states,
        hidden_states=all_hidden_states,
        attentions=all_self_attentions,
    )

mindnlp.transformers.models.hubert.modeling_hubert.HubertFeatureEncoder

Bases: Module

Construct the features from raw audio waveform

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

class HubertFeatureEncoder(nn.Module):
    """Construct the features from raw audio waveform"""
    def __init__(self, config: HubertConfig):
        """
        Initializes a new instance of HubertFeatureEncoder.

        Args:
            self: The instance of the class.
            config (HubertConfig): An instance of HubertConfig containing configuration parameters for the feature encoder.
                It specifies the normalization type to be used for feature extraction.

                - config.feat_extract_norm (str): Specifies the normalization type, should be either 'group' or 'layer'.

        Returns:
            None.

        Raises:
            ValueError: If the normalization type specified in config.feat_extract_norm is not 'group' or 'layer'.
        """
        super().__init__()
        if config.feat_extract_norm == "group":
            conv_layers = [HubertGroupNormConvLayer(config, layer_id=0)] + [
                HubertNoLayerNormConvLayer(config, layer_id=i + 1) for i in range(config.num_feat_extract_layers - 1)
            ]
        elif config.feat_extract_norm == "layer":
            conv_layers = [HubertLayerNormConvLayer(config, layer_id=i) for i in range(config.num_feat_extract_layers)]
        else:
            raise ValueError(
                f"`config.feat_extract_norm` is {config.feat_extract_norm}, but has to be one of ['group', 'layer']"
            )
        self.conv_layers = nn.ModuleList(conv_layers)
        self._requires_grad = True

    def _freeze_parameters(self):
        """
        Method _freeze_parameters in the class HubertFeatureEncoder freezes the parameters of the model
        by setting their 'requires_grad' attribute to False.

        Args:
            self (HubertFeatureEncoder): The instance of the HubertFeatureEncoder class.

        Returns:
            None.

        Raises:
            None.
        """
        for _, param in self.parameters_and_names():
            param.requires_grad = False
        self._requires_grad = False

    def forward(self, input_values):
        """
        Constructs the hidden states of the HubertFeatureEncoder.

        Args:
            self (HubertFeatureEncoder): An instance of the HubertFeatureEncoder class.
            input_values (array-like): The input values for forwarding the hidden states.
                It should be a 2-dimensional array with shape (n_samples, n_features).

        Returns:
            None.

        Raises:
            None.
        """
        hidden_states = input_values[:, None]
        for conv_layer in self.conv_layers:
            hidden_states = conv_layer(hidden_states)
        return hidden_states

mindnlp.transformers.models.hubert.modeling_hubert.HubertFeatureEncoder.__init__(config)

Initializes a new instance of HubertFeatureEncoder.

PARAMETER	DESCRIPTION
self	The instance of the class.
config	An instance of HubertConfig containing configuration parameters for the feature encoder. It specifies the normalization type to be used for feature extraction. config.feat_extract_norm (str): Specifies the normalization type, should be either 'group' or 'layer'. TYPE: HubertConfig

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If the normalization type specified in config.feat_extract_norm is not 'group' or 'layer'.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def __init__(self, config: HubertConfig):
    """
    Initializes a new instance of HubertFeatureEncoder.

    Args:
        self: The instance of the class.
        config (HubertConfig): An instance of HubertConfig containing configuration parameters for the feature encoder.
            It specifies the normalization type to be used for feature extraction.

            - config.feat_extract_norm (str): Specifies the normalization type, should be either 'group' or 'layer'.

    Returns:
        None.

    Raises:
        ValueError: If the normalization type specified in config.feat_extract_norm is not 'group' or 'layer'.
    """
    super().__init__()
    if config.feat_extract_norm == "group":
        conv_layers = [HubertGroupNormConvLayer(config, layer_id=0)] + [
            HubertNoLayerNormConvLayer(config, layer_id=i + 1) for i in range(config.num_feat_extract_layers - 1)
        ]
    elif config.feat_extract_norm == "layer":
        conv_layers = [HubertLayerNormConvLayer(config, layer_id=i) for i in range(config.num_feat_extract_layers)]
    else:
        raise ValueError(
            f"`config.feat_extract_norm` is {config.feat_extract_norm}, but has to be one of ['group', 'layer']"
        )
    self.conv_layers = nn.ModuleList(conv_layers)
    self._requires_grad = True

mindnlp.transformers.models.hubert.modeling_hubert.HubertFeatureEncoder.forward(input_values)

Constructs the hidden states of the HubertFeatureEncoder.

PARAMETER	DESCRIPTION
self	An instance of the HubertFeatureEncoder class. TYPE: HubertFeatureEncoder
input_values	The input values for forwarding the hidden states. It should be a 2-dimensional array with shape (n_samples, n_features). TYPE: array - like

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def forward(self, input_values):
    """
    Constructs the hidden states of the HubertFeatureEncoder.

    Args:
        self (HubertFeatureEncoder): An instance of the HubertFeatureEncoder class.
        input_values (array-like): The input values for forwarding the hidden states.
            It should be a 2-dimensional array with shape (n_samples, n_features).

    Returns:
        None.

    Raises:
        None.
    """
    hidden_states = input_values[:, None]
    for conv_layer in self.conv_layers:
        hidden_states = conv_layer(hidden_states)
    return hidden_states

mindnlp.transformers.models.hubert.modeling_hubert.HubertFeatureProjection

Bases: Module

Represents a feature projection module for the Hubert model.

This class inherits from nn.Module and implements methods for initializing the feature projection layer and performing feature projection on hidden states.

ATTRIBUTE	DESCRIPTION
feat_proj_layer_norm	Indicates whether feature projection layer normalization is enabled. TYPE: bool
layer_norm	If feat_proj_layer_norm is True, this attribute represents the layer normalization module. TYPE: LayerNorm
projection	The dense layer for feature projection. TYPE: Linear
dropout	The dropout layer for feature projection. TYPE: Dropout

METHOD	DESCRIPTION
__init__	Initializes the feature projection layer with the given configuration.
forward	Performs feature projection on the input hidden states and returns the projected hidden states.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

class HubertFeatureProjection(nn.Module):

    '''
    Represents a feature projection module for the Hubert model.

    This class inherits from nn.Module and implements methods for initializing the feature projection layer
    and performing feature projection on hidden states.

    Attributes:
        feat_proj_layer_norm (bool): Indicates whether feature projection layer normalization is enabled.
        layer_norm (nn.LayerNorm): If feat_proj_layer_norm is True, this attribute represents the layer normalization module.
        projection (nn.Linear): The dense layer for feature projection.
        dropout (nn.Dropout): The dropout layer for feature projection.

    Methods:
        __init__: Initializes the feature projection layer with the given configuration.
        forward: Performs feature projection on the input hidden states and returns the projected hidden states.
    '''
    def __init__(self, config: HubertConfig):
        """
        Initializes a new instance of HubertFeatureProjection.

        Args:
            self: The instance of the HubertFeatureProjection class.
            config (HubertConfig):
                An instance of HubertConfig containing configuration parameters for the feature projection.

                - feat_proj_layer_norm (bool): Indicates whether layer normalization should be applied.
                - conv_dim (list): List of dimensions for convolutional layers.
                - layer_norm_eps (float): Epsilon value for layer normalization.
                - hidden_size (int): Size of the hidden layer.
                - feat_proj_dropout (float): Dropout rate for feature projection.

        Returns:
            None.

        Raises:
            TypeError: If the config parameter is not of type HubertConfig.
            AttributeError: If the config object does not contain the required attributes.
            ValueError: If the config attributes are not within the specified range or format.
        """
        super().__init__()
        self.feat_proj_layer_norm = config.feat_proj_layer_norm
        if self.feat_proj_layer_norm:
            self.layer_norm = nn.LayerNorm(config.conv_dim[-1], eps=config.layer_norm_eps)
        self.projection = nn.Linear(config.conv_dim[-1], config.hidden_size)
        self.dropout = nn.Dropout(p=config.feat_proj_dropout)

    def forward(self, hidden_states):
        """
        Constructs the feature projection for the HubertFeatureProjection class.

        Args:
            self: An instance of the HubertFeatureProjection class.
            hidden_states (Tensor): The input hidden states to be projected.
                It should have a shape of (batch_size, seq_length, hidden_size).

        Returns:
            None

        Raises:
            None
        """
        # non-projected hidden states are needed for quantization
        if self.feat_proj_layer_norm:
            hidden_states = self.layer_norm(hidden_states)
        hidden_states = self.projection(hidden_states)
        hidden_states = self.dropout(hidden_states)
        return hidden_states

mindnlp.transformers.models.hubert.modeling_hubert.HubertFeatureProjection.__init__(config)

Initializes a new instance of HubertFeatureProjection.

PARAMETER	DESCRIPTION
self	The instance of the HubertFeatureProjection class.
config	An instance of HubertConfig containing configuration parameters for the feature projection. feat_proj_layer_norm (bool): Indicates whether layer normalization should be applied. conv_dim (list): List of dimensions for convolutional layers. layer_norm_eps (float): Epsilon value for layer normalization. hidden_size (int): Size of the hidden layer. feat_proj_dropout (float): Dropout rate for feature projection. TYPE: HubertConfig

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
TypeError	If the config parameter is not of type HubertConfig.
AttributeError	If the config object does not contain the required attributes.
ValueError	If the config attributes are not within the specified range or format.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def __init__(self, config: HubertConfig):
    """
    Initializes a new instance of HubertFeatureProjection.

    Args:
        self: The instance of the HubertFeatureProjection class.
        config (HubertConfig):
            An instance of HubertConfig containing configuration parameters for the feature projection.

            - feat_proj_layer_norm (bool): Indicates whether layer normalization should be applied.
            - conv_dim (list): List of dimensions for convolutional layers.
            - layer_norm_eps (float): Epsilon value for layer normalization.
            - hidden_size (int): Size of the hidden layer.
            - feat_proj_dropout (float): Dropout rate for feature projection.

    Returns:
        None.

    Raises:
        TypeError: If the config parameter is not of type HubertConfig.
        AttributeError: If the config object does not contain the required attributes.
        ValueError: If the config attributes are not within the specified range or format.
    """
    super().__init__()
    self.feat_proj_layer_norm = config.feat_proj_layer_norm
    if self.feat_proj_layer_norm:
        self.layer_norm = nn.LayerNorm(config.conv_dim[-1], eps=config.layer_norm_eps)
    self.projection = nn.Linear(config.conv_dim[-1], config.hidden_size)
    self.dropout = nn.Dropout(p=config.feat_proj_dropout)

mindnlp.transformers.models.hubert.modeling_hubert.HubertFeatureProjection.forward(hidden_states)

Constructs the feature projection for the HubertFeatureProjection class.

PARAMETER	DESCRIPTION
self	An instance of the HubertFeatureProjection class.
hidden_states	The input hidden states to be projected. It should have a shape of (batch_size, seq_length, hidden_size). TYPE: Tensor

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def forward(self, hidden_states):
    """
    Constructs the feature projection for the HubertFeatureProjection class.

    Args:
        self: An instance of the HubertFeatureProjection class.
        hidden_states (Tensor): The input hidden states to be projected.
            It should have a shape of (batch_size, seq_length, hidden_size).

    Returns:
        None

    Raises:
        None
    """
    # non-projected hidden states are needed for quantization
    if self.feat_proj_layer_norm:
        hidden_states = self.layer_norm(hidden_states)
    hidden_states = self.projection(hidden_states)
    hidden_states = self.dropout(hidden_states)
    return hidden_states

mindnlp.transformers.models.hubert.modeling_hubert.HubertFeedForward

Bases: Module

The HubertFeedForward class represents a feedforward neural network layer for the Hubert model. It inherits from nn.Module and implements the feedforward computation for the hidden states.

ATTRIBUTE	DESCRIPTION
config	The configuration object for the Hubert model. TYPE: HubertConfig

METHOD	DESCRIPTION
__init__	Initializes the HubertFeedForward instance with the provided configuration.
forward	Constructs the feedforward neural network layer using the provided hidden_states.

Example

Instantiate the HubertFeedForward class with a given configuration:
>>> config = HubertConfig(...)
>>> feed_forward_layer = HubertFeedForward(config)

Perform the feedforward computation using the forwarded layer:
>>> hidden_states = ...
>>> output = feed_forward_layer.forward(hidden_states)

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

class HubertFeedForward(nn.Module):

    """
    The HubertFeedForward class represents a feedforward neural network layer for the Hubert model.
    It inherits from nn.Module and implements the feedforward computation for the hidden states.

    Attributes:
        config (HubertConfig): The configuration object for the Hubert model.

    Methods:
        __init__: Initializes the HubertFeedForward instance with the provided configuration.
        forward: Constructs the feedforward neural network layer using the provided hidden_states.

    Example:
        ```python
        Instantiate the HubertFeedForward class with a given configuration:
        >>> config = HubertConfig(...)
        >>> feed_forward_layer = HubertFeedForward(config)

        Perform the feedforward computation using the forwarded layer:
        >>> hidden_states = ...
        >>> output = feed_forward_layer.forward(hidden_states)
        ```
    """
    def __init__(self, config: HubertConfig):
        """
        Initializes the HubertFeedForward class with the specified configuration.

        Args:
            self: The instance of the HubertFeedForward class.
            config (HubertConfig):
                An instance of HubertConfig containing the configuration parameters for the feed-forward layer.
                The config parameter should be of type HubertConfig and is used to set up the feed-forward layer.

        Returns:
            None.

        Raises:
            TypeError: If the config parameter is not of type HubertConfig.
        """
        super().__init__()
        self.intermediate_dropout = nn.Dropout(p=config.activation_dropout)
        self.intermediate_dense = nn.Linear(config.hidden_size, config.intermediate_size)
        if isinstance(config.hidden_act, str):
            self.intermediate_act_fn = ACT2FN[config.hidden_act]
        else:
            self.intermediate_act_fn = config.hidden_act
        self.output_dense = nn.Linear(config.intermediate_size, config.hidden_size)
        self.output_dropout = nn.Dropout(p=config.hidden_dropout)

    def forward(self, hidden_states):
        """
        Constructs the hidden states of the HubertFeedForward model.

        Args:
            self: An instance of the HubertFeedForward class.
            hidden_states (Tensor):
                The hidden states to be processed by the model.

                - Shape: (batch_size, sequence_length, hidden_size).
                - Purpose: Represents the input hidden states for the model.
                - Restrictions: None.

        Returns:
            None.

        Raises:
            None.
        """
        hidden_states = self.intermediate_dense(hidden_states)
        hidden_states = self.intermediate_act_fn(hidden_states)
        hidden_states = self.intermediate_dropout(hidden_states)
        hidden_states = self.output_dense(hidden_states)
        hidden_states = self.output_dropout(hidden_states)
        return hidden_states

mindnlp.transformers.models.hubert.modeling_hubert.HubertFeedForward.__init__(config)

Initializes the HubertFeedForward class with the specified configuration.

PARAMETER	DESCRIPTION
self	The instance of the HubertFeedForward class.
config	An instance of HubertConfig containing the configuration parameters for the feed-forward layer. The config parameter should be of type HubertConfig and is used to set up the feed-forward layer. TYPE: HubertConfig

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
TypeError	If the config parameter is not of type HubertConfig.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def __init__(self, config: HubertConfig):
    """
    Initializes the HubertFeedForward class with the specified configuration.

    Args:
        self: The instance of the HubertFeedForward class.
        config (HubertConfig):
            An instance of HubertConfig containing the configuration parameters for the feed-forward layer.
            The config parameter should be of type HubertConfig and is used to set up the feed-forward layer.

    Returns:
        None.

    Raises:
        TypeError: If the config parameter is not of type HubertConfig.
    """
    super().__init__()
    self.intermediate_dropout = nn.Dropout(p=config.activation_dropout)
    self.intermediate_dense = nn.Linear(config.hidden_size, config.intermediate_size)
    if isinstance(config.hidden_act, str):
        self.intermediate_act_fn = ACT2FN[config.hidden_act]
    else:
        self.intermediate_act_fn = config.hidden_act
    self.output_dense = nn.Linear(config.intermediate_size, config.hidden_size)
    self.output_dropout = nn.Dropout(p=config.hidden_dropout)

mindnlp.transformers.models.hubert.modeling_hubert.HubertFeedForward.forward(hidden_states)

Constructs the hidden states of the HubertFeedForward model.

PARAMETER	DESCRIPTION
self	An instance of the HubertFeedForward class.
hidden_states	The hidden states to be processed by the model. Shape: (batch_size, sequence_length, hidden_size). Purpose: Represents the input hidden states for the model. Restrictions: None. TYPE: Tensor

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def forward(self, hidden_states):
    """
    Constructs the hidden states of the HubertFeedForward model.

    Args:
        self: An instance of the HubertFeedForward class.
        hidden_states (Tensor):
            The hidden states to be processed by the model.

            - Shape: (batch_size, sequence_length, hidden_size).
            - Purpose: Represents the input hidden states for the model.
            - Restrictions: None.

    Returns:
        None.

    Raises:
        None.
    """
    hidden_states = self.intermediate_dense(hidden_states)
    hidden_states = self.intermediate_act_fn(hidden_states)
    hidden_states = self.intermediate_dropout(hidden_states)
    hidden_states = self.output_dense(hidden_states)
    hidden_states = self.output_dropout(hidden_states)
    return hidden_states

mindnlp.transformers.models.hubert.modeling_hubert.HubertForCTC

Bases: HubertPreTrainedModel

A class representing the Hubert model for Connectionist Temporal Classification (CTC).

This class extends the HubertPreTrainedModel class and provides additional methods for freezing the feature encoder and base model, as well as forwarding the model and computing the CTC loss.

ATTRIBUTE	DESCRIPTION
hubert	The Hubert model for feature extraction. TYPE: HubertModel
dropout	Dropout layer for regularization. TYPE: Dropout
target_lang	The target language for the model. TYPE: str
lm_head	Fully connected layer for language modeling. TYPE: Dense

METHOD	DESCRIPTION
__init__	Initializes the HubertForCTC instance with a given configuration and target language.
tie_weights	Overwrites the tie_weights method to correctly load adapter weights when passing target_lang to from_pretrained().
freeze_feature_encoder	Disables gradient computation for the feature encoder to prevent parameter updates during training.
freeze_base_model	Disables gradient computation for the base model to prevent parameter updates during training.
forward	Constructs the model and computes the CTC loss.

Note

• The target_lang parameter is used for loading adapter weights and should not be passed if config.adapter_attn_dim is not defined.
• The forward method computes the CTC loss for connectionist temporal classification tasks.

RAISES	DESCRIPTION
ValueError	If the config.vocab_size is not defined when instantiating the model.
ValueError	If target_lang is passed without config.adapter_attn_dim being defined.

This class is intended to be used as a language model for CTC tasks, where labels are provided for training and the model outputs logits for each input sequence.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

class HubertForCTC(HubertPreTrainedModel):

    """
    A class representing the Hubert model for Connectionist Temporal Classification (CTC).

    This class extends the HubertPreTrainedModel class and provides additional methods for freezing the feature encoder
    and base model, as well as forwarding the model and computing the CTC loss.

    Attributes:
        hubert (HubertModel): The Hubert model for feature extraction.
        dropout (Dropout): Dropout layer for regularization.
        target_lang (str): The target language for the model.
        lm_head (Dense): Fully connected layer for language modeling.

    Methods:
        __init__: Initializes the HubertForCTC instance with a given configuration and target language.
        tie_weights: Overwrites the tie_weights method to correctly load adapter weights when passing target_lang
            to from_pretrained().
        freeze_feature_encoder: Disables gradient computation for the feature encoder to prevent parameter updates
            during training.
        freeze_base_model: Disables gradient computation for the base model to prevent parameter updates during training.
        forward: Constructs the model and computes the CTC loss.

    Note:
        - The target_lang parameter is used for loading adapter weights and should not be passed
        if config.adapter_attn_dim is not defined.
        - The forward method computes the CTC loss for connectionist temporal classification tasks.

    Raises:
        ValueError: If the config.vocab_size is not defined when instantiating the model.
        ValueError: If target_lang is passed without config.adapter_attn_dim being defined.

    This class is intended to be used as a language model for CTC tasks, where labels are provided for training and
    the model outputs logits for each input sequence.
    """
    def __init__(self, config: HubertConfig, target_lang: Optional[str] = None):
        """
        Initializes a new instance of the HubertForCTC class.

        Args:
            self: The object instance.
            config (HubertConfig): The configuration object for the Hubert model.
            target_lang (Optional[str], default=None): The target language for the model.
                If specified, the model will be trained for the specified language.

        Returns:
            None

        Raises:
            ValueError: If the configuration does not define the vocabulary size of the language model head.

        This method initializes the HubertForCTC class by setting up the following components:

        - config: The configuration object for the Hubert model.
        - hubert: The HubertModel instance based on the provided configuration.
        - dropout: A dropout layer with the dropout probability defined in the configuration.
        - target_lang: The target language for the model, if specified.
        - lm_head: A dense layer with the output hidden size and vocabulary size defined in the configuration.

        Note:
            If the configuration has the 'add_adapter' attribute and it is set to True, the output hidden size will be
            the value of 'output_hidden_size'. Otherwise, it will be the value of 'hidden_size'.

        After initializing these components, the 'post_init' method is called to perform any additional setup tasks.
        """
        super().__init__(config)

        self.hubert = HubertModel(config)
        self.dropout = nn.Dropout(p=config.final_dropout)

        self.target_lang = target_lang

        if config.vocab_size is None:
            raise ValueError(
                f"You are trying to instantiate {self.__class__} with a configuration that "
                "does not define the vocabulary size of the language model head. Please "
                "instantiate the model as follows: `HubertForCTC.from_pretrained(..., vocab_size=vocab_size)`. "
                "or define `vocab_size` of your model's configuration."
            )
        output_hidden_size = (
            config.output_hidden_size if hasattr(config, "add_adapter") and config.add_adapter else config.hidden_size
        )
        self.lm_head = nn.Linear(output_hidden_size, config.vocab_size)

        # Initialize weights and apply final processing
        self.post_init()

    def tie_weights(self):
        """
        This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when
        passing `target_lang=...` to `from_pretrained(...)`.

        This method is **not** supposed to be called by the user and is prone to be changed in the future.
        """
        # Note that `tie_weights` is usually used to tie input and output embedding weights. The method is re-purposed to
        # correctly load adapter layers for Hubert so that we do not have to introduce a new API to
        # [`PreTrainedModel`]. While slightly hacky, Hubert never has to tie input and output embeddings, so that it is
        # ok to repurpose this function here.
        target_lang = self.target_lang

        if target_lang is not None and getattr(self.config_class, "adapter_attn_dim", None) is None:
            raise ValueError(f"Cannot pass `target_lang`: {target_lang} if `config.adapter_attn_dim` is not defined.")
        if target_lang is None and getattr(self.config, "adapter_attn_dim", None) is not None:
            logger.info("By default `target_lang` is set to 'eng'.")
        elif target_lang is not None:
            self.load_adapter(target_lang)

    def freeze_feature_encoder(self):
        """
        Calling this function will disable the gradient computation for the feature encoder so that its parameter will
        not be updated during training.
        """
        self.hubert.feature_extractor._freeze_parameters()

    def freeze_base_model(self):
        """
        Calling this function will disable the gradient computation for the base model so that its parameters will not
        be updated during training. Only the classification head will be updated.
        """
        for _, param in self.hubert.parameters_and_names():
            param.requires_grad = False

    def forward(
        self,
        input_values: Optional[Tensor],
        attention_mask: Optional[Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
        labels: Optional[Tensor] = None,
    ) -> Union[Tuple, CausalLMOutput]:
        r"""
        Args:
            labels (`Tensor` of shape `(batch_size, target_length)`, *optional*):
                Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to
                the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`.
                All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ...,
                config.vocab_size - 1]`.
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        outputs = self.hubert(
            input_values,
            attention_mask=attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        hidden_states = outputs[0]
        hidden_states = self.dropout(hidden_states)

        logits = self.lm_head(hidden_states)

        loss = None
        if labels is not None:
            labels = labels.astype(mindspore.int32)
            if labels.max() >= self.config.vocab_size:
                raise ValueError(f"Label values must be <= vocab_size: {self.config.vocab_size}")

            # retrieve loss input_lengths from attention_mask
            attention_mask = (
                attention_mask if attention_mask is not None else ops.ones_like(input_values, dtype=mindspore.int64)
            )
            input_lengths = self._get_feat_extract_output_lengths(attention_mask.sum(-1)).to(mindspore.int64)

            # assuming that padded tokens are filled with -100
            # when not being attended to
            labels_mask = labels >= 0
            target_lengths = labels_mask.sum(-1)
            flattened_targets = labels.masked_select(labels_mask)

            # ctc_loss doesn't support fp16
            log_probs = ops.log_softmax(logits, axis=-1).swapaxes(0, 1)
            loss, log_alpha = ops.ctc_loss(
                log_probs,   # [T, N/B, C/NC]
                labels,      # [N/B, S], replace `flattened_targets`
                input_lengths,
                target_lengths,
                blank=self.config.pad_token_id,
                reduction=self.config.ctc_loss_reduction,
                zero_infinity=self.config.ctc_zero_infinity,
            )

        if not return_dict:
            output = (logits,) + outputs[_HIDDEN_STATES_START_POSITION:]
            return ((loss,) + output) if loss is not None else output

        return CausalLMOutput(
            loss=loss, logits=logits, hidden_states=outputs.hidden_states, attentions=outputs.attentions
        )

mindnlp.transformers.models.hubert.modeling_hubert.HubertForCTC.__init__(config, target_lang=None)

Initializes a new instance of the HubertForCTC class.

PARAMETER	DESCRIPTION
self	The object instance.
config	The configuration object for the Hubert model. TYPE: HubertConfig
target_lang	The target language for the model. If specified, the model will be trained for the specified language. TYPE: Optional[str], default=None DEFAULT: None

RETURNS	DESCRIPTION
	None

RAISES	DESCRIPTION
ValueError	If the configuration does not define the vocabulary size of the language model head.

This method initializes the HubertForCTC class by setting up the following components:

• config: The configuration object for the Hubert model.
• hubert: The HubertModel instance based on the provided configuration.
• dropout: A dropout layer with the dropout probability defined in the configuration.
• target_lang: The target language for the model, if specified.
• lm_head: A dense layer with the output hidden size and vocabulary size defined in the configuration.

Note

If the configuration has the 'add_adapter' attribute and it is set to True, the output hidden size will be the value of 'output_hidden_size'. Otherwise, it will be the value of 'hidden_size'.

After initializing these components, the 'post_init' method is called to perform any additional setup tasks.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def __init__(self, config: HubertConfig, target_lang: Optional[str] = None):
    """
    Initializes a new instance of the HubertForCTC class.

    Args:
        self: The object instance.
        config (HubertConfig): The configuration object for the Hubert model.
        target_lang (Optional[str], default=None): The target language for the model.
            If specified, the model will be trained for the specified language.

    Returns:
        None

    Raises:
        ValueError: If the configuration does not define the vocabulary size of the language model head.

    This method initializes the HubertForCTC class by setting up the following components:

    - config: The configuration object for the Hubert model.
    - hubert: The HubertModel instance based on the provided configuration.
    - dropout: A dropout layer with the dropout probability defined in the configuration.
    - target_lang: The target language for the model, if specified.
    - lm_head: A dense layer with the output hidden size and vocabulary size defined in the configuration.

    Note:
        If the configuration has the 'add_adapter' attribute and it is set to True, the output hidden size will be
        the value of 'output_hidden_size'. Otherwise, it will be the value of 'hidden_size'.

    After initializing these components, the 'post_init' method is called to perform any additional setup tasks.
    """
    super().__init__(config)

    self.hubert = HubertModel(config)
    self.dropout = nn.Dropout(p=config.final_dropout)

    self.target_lang = target_lang

    if config.vocab_size is None:
        raise ValueError(
            f"You are trying to instantiate {self.__class__} with a configuration that "
            "does not define the vocabulary size of the language model head. Please "
            "instantiate the model as follows: `HubertForCTC.from_pretrained(..., vocab_size=vocab_size)`. "
            "or define `vocab_size` of your model's configuration."
        )
    output_hidden_size = (
        config.output_hidden_size if hasattr(config, "add_adapter") and config.add_adapter else config.hidden_size
    )
    self.lm_head = nn.Linear(output_hidden_size, config.vocab_size)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.hubert.modeling_hubert.HubertForCTC.forward(input_values, attention_mask=None, output_attentions=None, output_hidden_states=None, return_dict=None, labels=None)

PARAMETER

DESCRIPTION

labels

Labels for connectionist temporal classification. Note that target_length has to be smaller or equal to the sequence length of the output logits. Indices are selected in [-100, 0, ..., config.vocab_size - 1]. All labels set to -100 are ignored (masked), the loss is only computed for labels in [0, ..., config.vocab_size - 1]. TYPE: `Tensor` of shape `(batch_size, target_length)`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def forward(
    self,
    input_values: Optional[Tensor],
    attention_mask: Optional[Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
    labels: Optional[Tensor] = None,
) -> Union[Tuple, CausalLMOutput]:
    r"""
    Args:
        labels (`Tensor` of shape `(batch_size, target_length)`, *optional*):
            Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to
            the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`.
            All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ...,
            config.vocab_size - 1]`.
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    outputs = self.hubert(
        input_values,
        attention_mask=attention_mask,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    hidden_states = outputs[0]
    hidden_states = self.dropout(hidden_states)

    logits = self.lm_head(hidden_states)

    loss = None
    if labels is not None:
        labels = labels.astype(mindspore.int32)
        if labels.max() >= self.config.vocab_size:
            raise ValueError(f"Label values must be <= vocab_size: {self.config.vocab_size}")

        # retrieve loss input_lengths from attention_mask
        attention_mask = (
            attention_mask if attention_mask is not None else ops.ones_like(input_values, dtype=mindspore.int64)
        )
        input_lengths = self._get_feat_extract_output_lengths(attention_mask.sum(-1)).to(mindspore.int64)

        # assuming that padded tokens are filled with -100
        # when not being attended to
        labels_mask = labels >= 0
        target_lengths = labels_mask.sum(-1)
        flattened_targets = labels.masked_select(labels_mask)

        # ctc_loss doesn't support fp16
        log_probs = ops.log_softmax(logits, axis=-1).swapaxes(0, 1)
        loss, log_alpha = ops.ctc_loss(
            log_probs,   # [T, N/B, C/NC]
            labels,      # [N/B, S], replace `flattened_targets`
            input_lengths,
            target_lengths,
            blank=self.config.pad_token_id,
            reduction=self.config.ctc_loss_reduction,
            zero_infinity=self.config.ctc_zero_infinity,
        )

    if not return_dict:
        output = (logits,) + outputs[_HIDDEN_STATES_START_POSITION:]
        return ((loss,) + output) if loss is not None else output

    return CausalLMOutput(
        loss=loss, logits=logits, hidden_states=outputs.hidden_states, attentions=outputs.attentions
    )

mindnlp.transformers.models.hubert.modeling_hubert.HubertForCTC.freeze_base_model()

Calling this function will disable the gradient computation for the base model so that its parameters will not be updated during training. Only the classification head will be updated.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def freeze_base_model(self):
    """
    Calling this function will disable the gradient computation for the base model so that its parameters will not
    be updated during training. Only the classification head will be updated.
    """
    for _, param in self.hubert.parameters_and_names():
        param.requires_grad = False

mindnlp.transformers.models.hubert.modeling_hubert.HubertForCTC.freeze_feature_encoder()

Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def freeze_feature_encoder(self):
    """
    Calling this function will disable the gradient computation for the feature encoder so that its parameter will
    not be updated during training.
    """
    self.hubert.feature_extractor._freeze_parameters()

mindnlp.transformers.models.hubert.modeling_hubert.HubertForCTC.tie_weights()

This method overwrites [~PreTrainedModel.tie_weights] so that adapter weights can be correctly loaded when passing target_lang=... to from_pretrained(...).

This method is not supposed to be called by the user and is prone to be changed in the future.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def tie_weights(self):
    """
    This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when
    passing `target_lang=...` to `from_pretrained(...)`.

    This method is **not** supposed to be called by the user and is prone to be changed in the future.
    """
    # Note that `tie_weights` is usually used to tie input and output embedding weights. The method is re-purposed to
    # correctly load adapter layers for Hubert so that we do not have to introduce a new API to
    # [`PreTrainedModel`]. While slightly hacky, Hubert never has to tie input and output embeddings, so that it is
    # ok to repurpose this function here.
    target_lang = self.target_lang

    if target_lang is not None and getattr(self.config_class, "adapter_attn_dim", None) is None:
        raise ValueError(f"Cannot pass `target_lang`: {target_lang} if `config.adapter_attn_dim` is not defined.")
    if target_lang is None and getattr(self.config, "adapter_attn_dim", None) is not None:
        logger.info("By default `target_lang` is set to 'eng'.")
    elif target_lang is not None:
        self.load_adapter(target_lang)

mindnlp.transformers.models.hubert.modeling_hubert.HubertForSequenceClassification

Bases: HubertPreTrainedModel

HubertForSequenceClassification is a class that represents a sequence classification model based on the Hubert architecture. This class extends the HubertPreTrainedModel and provides functionality for sequence classification tasks.

METHOD	DESCRIPTION
__init__	Initializes the sequence classification model with the provided configuration.
freeze_feature_encoder	Disables gradient computation for the feature encoder to prevent parameter updates during training.
freeze_base_model	Disables gradient computation for the base model parameters, allowing only the classification head to be updated.
forward	Constructs the sequence classification model and computes the loss based on the provided input values and labels.

ATTRIBUTE	DESCRIPTION
hubert	HubertModel instance for the sequence classification model.
projector	nn.Linear layer for projecting hidden states to the classifier projection size.
classifier	nn.Linear layer for classification predictions.
layer_weights	Parameter for weighted layer sum computation.

Note

• The class assumes a specific structure and functionality based on the provided code snippets.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

class HubertForSequenceClassification(HubertPreTrainedModel):

    """
    HubertForSequenceClassification is a class that represents a sequence classification model based on the Hubert architecture.
    This class extends the HubertPreTrainedModel and provides functionality for sequence classification tasks.

    Methods:
        __init__: Initializes the sequence classification model with the provided configuration.
        freeze_feature_encoder:
            Disables gradient computation for the feature encoder to prevent parameter updates during training.
        freeze_base_model:
            Disables gradient computation for the base model parameters,
            allowing only the classification head to be updated.
        forward:
            Constructs the sequence classification model and computes the loss based on the
            provided input values and labels.

    Attributes:
        hubert: HubertModel instance for the sequence classification model.
        projector: nn.Linear layer for projecting hidden states to the classifier projection size.
        classifier: nn.Linear layer for classification predictions.
        layer_weights: Parameter for weighted layer sum computation.

    Note:
        - The class assumes a specific structure and functionality based on the provided code snippets.
    """
    def __init__(self, config: HubertConfig):
        """
        Initializes a new instance of HubertForSequenceClassification.

        Args:
            self: The instance of the class.
            config (HubertConfig): The configuration object for the Hubert model.

        Returns:
            None.

        Raises:
            ValueError: Raised if the 'config' object has the attribute 'add_adapter' set to True,
            as sequence classification does not support the use of Hubert adapters in this context.
        """
        super().__init__(config)

        if hasattr(config, "add_adapter") and config.add_adapter:
            raise ValueError("Sequence classification does not support the use of Hubert adapters (config.add_adapter=True)")
        self.hubert = HubertModel(config)
        num_layers = config.num_hidden_layers + 1  # transformer layers + input embeddings
        if config.use_weighted_layer_sum:
            self.layer_weights = Parameter(ops.ones(num_layers) / num_layers)
        self.projector = nn.Linear(config.hidden_size, config.classifier_proj_size)
        self.classifier = nn.Linear(config.classifier_proj_size, config.num_labels)

        # Initialize weights and apply final processing
        self.post_init()

    def freeze_feature_encoder(self):
        """
        Calling this function will disable the gradient computation for the feature encoder so that its parameter will
        not be updated during training.
        """
        self.hubert.feature_extractor._freeze_parameters()

    def freeze_base_model(self):
        """
        Calling this function will disable the gradient computation for the base model so that its parameters will not
        be updated during training. Only the classification head will be updated.
        """
        for _, param in self.hubert.parameters_and_names():
            param.requires_grad = False

    def forward(
        self,
        input_values: Optional[Tensor],
        attention_mask: Optional[Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
        labels: Optional[Tensor] = None,
    ) -> Union[Tuple, SequenceClassifierOutput]:
        r"""
        Args:
            labels (`Tensor` of shape `(batch_size,)`, *optional*):
                Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
                config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
                `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
        output_hidden_states = True if self.config.use_weighted_layer_sum else output_hidden_states

        outputs = self.hubert(
            input_values,
            attention_mask=attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        if self.config.use_weighted_layer_sum:
            hidden_states = outputs[_HIDDEN_STATES_START_POSITION]
            hidden_states = ops.stack(hidden_states, axis=1)
            norm_weights = ops.softmax(self.layer_weights, axis=-1)
            hidden_states = (hidden_states * norm_weights.view(-1, 1, 1)).sum(axis=1)
        else:
            hidden_states = outputs[0]

        hidden_states = self.projector(hidden_states)
        if attention_mask is None:
            pooled_output = hidden_states.mean(axis=1)
        else:
            padding_mask = self._get_feature_vector_attention_mask(hidden_states.shape[1], attention_mask)
            hidden_states[~padding_mask] = 0.0
            pooled_output = hidden_states.sum(axis=1) / padding_mask.sum(axis=1).view(-1, 1)

        logits = self.classifier(pooled_output)

        loss = None
        if labels is not None:
            labels = labels.astype(mindspore.int32)
            loss = ops.cross_entropy(logits.view(-1, self.config.num_labels), labels.view(-1))

        if not return_dict:
            output = (logits,) + outputs[_HIDDEN_STATES_START_POSITION:]
            return ((loss,) + output) if loss is not None else output

        return SequenceClassifierOutput(
            loss=loss,
            logits=logits,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )

mindnlp.transformers.models.hubert.modeling_hubert.HubertForSequenceClassification.__init__(config)

Initializes a new instance of HubertForSequenceClassification.

PARAMETER	DESCRIPTION
self	The instance of the class.
config	The configuration object for the Hubert model. TYPE: HubertConfig

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	Raised if the 'config' object has the attribute 'add_adapter' set to True,

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def __init__(self, config: HubertConfig):
    """
    Initializes a new instance of HubertForSequenceClassification.

    Args:
        self: The instance of the class.
        config (HubertConfig): The configuration object for the Hubert model.

    Returns:
        None.

    Raises:
        ValueError: Raised if the 'config' object has the attribute 'add_adapter' set to True,
        as sequence classification does not support the use of Hubert adapters in this context.
    """
    super().__init__(config)

    if hasattr(config, "add_adapter") and config.add_adapter:
        raise ValueError("Sequence classification does not support the use of Hubert adapters (config.add_adapter=True)")
    self.hubert = HubertModel(config)
    num_layers = config.num_hidden_layers + 1  # transformer layers + input embeddings
    if config.use_weighted_layer_sum:
        self.layer_weights = Parameter(ops.ones(num_layers) / num_layers)
    self.projector = nn.Linear(config.hidden_size, config.classifier_proj_size)
    self.classifier = nn.Linear(config.classifier_proj_size, config.num_labels)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.hubert.modeling_hubert.HubertForSequenceClassification.forward(input_values, attention_mask=None, output_attentions=None, output_hidden_states=None, return_dict=None, labels=None)

PARAMETER	DESCRIPTION
labels	Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If config.num_labels > 1 a classification loss is computed (Cross-Entropy). TYPE: `Tensor` of shape `(batch_size,)`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def forward(
    self,
    input_values: Optional[Tensor],
    attention_mask: Optional[Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
    labels: Optional[Tensor] = None,
) -> Union[Tuple, SequenceClassifierOutput]:
    r"""
    Args:
        labels (`Tensor` of shape `(batch_size,)`, *optional*):
            Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
            config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
            `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    output_hidden_states = True if self.config.use_weighted_layer_sum else output_hidden_states

    outputs = self.hubert(
        input_values,
        attention_mask=attention_mask,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    if self.config.use_weighted_layer_sum:
        hidden_states = outputs[_HIDDEN_STATES_START_POSITION]
        hidden_states = ops.stack(hidden_states, axis=1)
        norm_weights = ops.softmax(self.layer_weights, axis=-1)
        hidden_states = (hidden_states * norm_weights.view(-1, 1, 1)).sum(axis=1)
    else:
        hidden_states = outputs[0]

    hidden_states = self.projector(hidden_states)
    if attention_mask is None:
        pooled_output = hidden_states.mean(axis=1)
    else:
        padding_mask = self._get_feature_vector_attention_mask(hidden_states.shape[1], attention_mask)
        hidden_states[~padding_mask] = 0.0
        pooled_output = hidden_states.sum(axis=1) / padding_mask.sum(axis=1).view(-1, 1)

    logits = self.classifier(pooled_output)

    loss = None
    if labels is not None:
        labels = labels.astype(mindspore.int32)
        loss = ops.cross_entropy(logits.view(-1, self.config.num_labels), labels.view(-1))

    if not return_dict:
        output = (logits,) + outputs[_HIDDEN_STATES_START_POSITION:]
        return ((loss,) + output) if loss is not None else output

    return SequenceClassifierOutput(
        loss=loss,
        logits=logits,
        hidden_states=outputs.hidden_states,
        attentions=outputs.attentions,
    )

mindnlp.transformers.models.hubert.modeling_hubert.HubertForSequenceClassification.freeze_base_model()

Calling this function will disable the gradient computation for the base model so that its parameters will not be updated during training. Only the classification head will be updated.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def freeze_base_model(self):
    """
    Calling this function will disable the gradient computation for the base model so that its parameters will not
    be updated during training. Only the classification head will be updated.
    """
    for _, param in self.hubert.parameters_and_names():
        param.requires_grad = False

mindnlp.transformers.models.hubert.modeling_hubert.HubertForSequenceClassification.freeze_feature_encoder()

Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def freeze_feature_encoder(self):
    """
    Calling this function will disable the gradient computation for the feature encoder so that its parameter will
    not be updated during training.
    """
    self.hubert.feature_extractor._freeze_parameters()

mindnlp.transformers.models.hubert.modeling_hubert.HubertGroupNormConvLayer

Bases: Module

A class representing a Group Normalization Convolutional Layer in the Hubert model.

This class inherits from nn.Module and is used to define a single layer of the Hubert model. The layer consists of a 1-dimensional convolutional operation followed by group normalization, an activation function, and returns the output hidden states.

ATTRIBUTE	DESCRIPTION
in_conv_dim	The dimension of the input to the convolutional layer. TYPE: int
out_conv_dim	The dimension of the output from the convolutional layer. TYPE: int
conv	The 1-dimensional convolutional operation. TYPE: Conv1d
activation	The activation function applied to the hidden states. TYPE: function
layer_norm	The group normalization operation. TYPE: GroupNorm

METHOD	DESCRIPTION
forward	Applies the convolutional operation, group normalization, and activation function to the input hidden states and returns the output.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

class HubertGroupNormConvLayer(nn.Module):

    """A class representing a Group Normalization Convolutional Layer in the Hubert model.

    This class inherits from nn.Module and is used to define a single layer of the Hubert model.
    The layer consists of a 1-dimensional convolutional operation followed by group normalization,
    an activation function, and returns the output hidden states.

    Attributes:
        in_conv_dim (int): The dimension of the input to the convolutional layer.
        out_conv_dim (int): The dimension of the output from the convolutional layer.
        conv (nn.Conv1d): The 1-dimensional convolutional operation.
        activation (function): The activation function applied to the hidden states.
        layer_norm (nn.GroupNorm): The group normalization operation.

    Methods:
        forward(hidden_states): Applies the convolutional operation, group normalization,
            and activation function to the input hidden states and returns the output.

    """
    def __init__(self, config: HubertConfig, layer_id=0):
        """
        Initializes a HubertGroupNormConvLayer object.

        Args:
            self (HubertGroupNormConvLayer): The instance of the HubertGroupNormConvLayer class.
            config (HubertConfig): An instance of HubertConfig class containing configuration parameters.
            layer_id (int): The ID of the layer, defaults to 0. Used to access specific convolutional layer configuration.

        Returns:
            None.

        Raises:
            ValueError: If layer_id is less than 0.
            KeyError: If the specified feature extraction activation function is not found in the ACT2FN dictionary.
        """
        super().__init__()
        self.in_conv_dim = config.conv_dim[layer_id - 1] if layer_id > 0 else 1
        self.out_conv_dim = config.conv_dim[layer_id]

        self.conv = nn.Conv1d(
            self.in_conv_dim,
            self.out_conv_dim,
            kernel_size=config.conv_kernel[layer_id],
            stride=config.conv_stride[layer_id],
            bias=config.conv_bias,
            pad_mode="valid",
        )
        self.activation = ACT2FN[config.feat_extract_activation]
        # NOTE: the naming is confusing, but let it be...
        self.layer_norm = nn.GroupNorm(num_groups=self.out_conv_dim, num_channels=self.out_conv_dim, affine=True)

    def forward(self, hidden_states):
        """
        Construct a HubertGroupNormConvLayer by applying a series of operations on the input hidden states.

        Args:
            self (HubertGroupNormConvLayer): An instance of the HubertGroupNormConvLayer class.
            hidden_states (tensor): The input hidden states to be processed.
                Expected shape: (batch_size, channels, height, width).

        Returns:
            None

        Raises:
            None
        """
        hidden_states = self.conv(hidden_states)
        hidden_states = self.layer_norm(hidden_states)
        hidden_states = self.activation(hidden_states)
        return hidden_states

mindnlp.transformers.models.hubert.modeling_hubert.HubertGroupNormConvLayer.__init__(config, layer_id=0)

Initializes a HubertGroupNormConvLayer object.

PARAMETER	DESCRIPTION
self	The instance of the HubertGroupNormConvLayer class. TYPE: HubertGroupNormConvLayer
config	An instance of HubertConfig class containing configuration parameters. TYPE: HubertConfig
layer_id	The ID of the layer, defaults to 0. Used to access specific convolutional layer configuration. TYPE: int DEFAULT: 0

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If layer_id is less than 0.
KeyError	If the specified feature extraction activation function is not found in the ACT2FN dictionary.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def __init__(self, config: HubertConfig, layer_id=0):
    """
    Initializes a HubertGroupNormConvLayer object.

    Args:
        self (HubertGroupNormConvLayer): The instance of the HubertGroupNormConvLayer class.
        config (HubertConfig): An instance of HubertConfig class containing configuration parameters.
        layer_id (int): The ID of the layer, defaults to 0. Used to access specific convolutional layer configuration.

    Returns:
        None.

    Raises:
        ValueError: If layer_id is less than 0.
        KeyError: If the specified feature extraction activation function is not found in the ACT2FN dictionary.
    """
    super().__init__()
    self.in_conv_dim = config.conv_dim[layer_id - 1] if layer_id > 0 else 1
    self.out_conv_dim = config.conv_dim[layer_id]

    self.conv = nn.Conv1d(
        self.in_conv_dim,
        self.out_conv_dim,
        kernel_size=config.conv_kernel[layer_id],
        stride=config.conv_stride[layer_id],
        bias=config.conv_bias,
        pad_mode="valid",
    )
    self.activation = ACT2FN[config.feat_extract_activation]
    # NOTE: the naming is confusing, but let it be...
    self.layer_norm = nn.GroupNorm(num_groups=self.out_conv_dim, num_channels=self.out_conv_dim, affine=True)

mindnlp.transformers.models.hubert.modeling_hubert.HubertGroupNormConvLayer.forward(hidden_states)

Construct a HubertGroupNormConvLayer by applying a series of operations on the input hidden states.

PARAMETER	DESCRIPTION
self	An instance of the HubertGroupNormConvLayer class. TYPE: HubertGroupNormConvLayer
hidden_states	The input hidden states to be processed. Expected shape: (batch_size, channels, height, width). TYPE: tensor

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def forward(self, hidden_states):
    """
    Construct a HubertGroupNormConvLayer by applying a series of operations on the input hidden states.

    Args:
        self (HubertGroupNormConvLayer): An instance of the HubertGroupNormConvLayer class.
        hidden_states (tensor): The input hidden states to be processed.
            Expected shape: (batch_size, channels, height, width).

    Returns:
        None

    Raises:
        None
    """
    hidden_states = self.conv(hidden_states)
    hidden_states = self.layer_norm(hidden_states)
    hidden_states = self.activation(hidden_states)
    return hidden_states

mindnlp.transformers.models.hubert.modeling_hubert.HubertLayerNormConvLayer

Bases: Module

The HubertLayerNormConvLayer class represents a layer with convolution, layer normalization, and activation functions for the HuBERT model. It inherits from nn.Module.

This class initializes with a HubertConfig instance and a layer ID. It defines a convolutional layer with specified input and output dimensions, kernel size, stride, bias, and padding mode. It also applies layer normalization and an activation function to the input hidden states.

The forward method takes hidden states as input, applies the convolution, layer normalization, and activation function, and returns the processed hidden states.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

class HubertLayerNormConvLayer(nn.Module):

    """
    The HubertLayerNormConvLayer class represents a layer with convolution, layer normalization,
    and activation functions for the HuBERT model. It inherits from nn.Module.

    This class initializes with a HubertConfig instance and a layer ID. It defines a convolutional layer with
    specified input and output dimensions, kernel size, stride, bias, and padding mode. It also applies layer
    normalization and an activation function to the input hidden states.

    The forward method takes hidden states as input, applies the convolution, layer normalization, and
    activation function, and returns the processed hidden states.
    """
    def __init__(self, config: HubertConfig, layer_id=0):
        """
        Initializes a new instance of the HubertLayerNormConvLayer class.

        Args:
            self: The object itself.
            config (HubertConfig): The configuration object for the Hubert model.
            layer_id (int, optional): The ID of the layer. Defaults to 0.

        Returns:
            None.

        Raises:
            ValueError: If the provided layer_id is less than 0.
            TypeError: If the provided config is not an instance of HubertConfig.
            KeyError: If the provided config does not contain required attributes.
        """
        super().__init__()
        self.in_conv_dim = config.conv_dim[layer_id - 1] if layer_id > 0 else 1
        self.out_conv_dim = config.conv_dim[layer_id]

        self.conv = nn.Conv1d(
            self.in_conv_dim,
            self.out_conv_dim,
            kernel_size=config.conv_kernel[layer_id],
            stride=config.conv_stride[layer_id],
            bias=config.conv_bias,
            pad_mode="valid",
        )
        self.layer_norm = nn.LayerNorm(self.out_conv_dim)
        self.activation = ACT2FN[config.feat_extract_activation]

    def forward(self, hidden_states):
        """
        This method forwards a HubertLayerNormConvLayer by applying convolution, layer normalization, and
        activation functions to the input hidden states.

        Args:
            self: The instance of the HubertLayerNormConvLayer class.
            hidden_states: A tensor representing the input hidden states that will undergo the transformation.
                It should have the shape (batch_size, sequence_length, hidden_size).

        Returns:
            None: This method does not return any value directly.
                The hidden_states tensor is modified in place and returned after the transformations.

        Raises:
            ValueError: If the hidden_states tensor does not have the expected shape.
            RuntimeError: If any error occurs during the convolution, layer normalization, or activation operations.
        """
        hidden_states = self.conv(hidden_states)
        hidden_states = hidden_states.swapaxes(-2, -1)
        hidden_states = self.layer_norm(hidden_states)
        hidden_states = hidden_states.swapaxes(-2, -1)
        hidden_states = self.activation(hidden_states)
        return hidden_states

mindnlp.transformers.models.hubert.modeling_hubert.HubertLayerNormConvLayer.__init__(config, layer_id=0)

Initializes a new instance of the HubertLayerNormConvLayer class.

PARAMETER	DESCRIPTION
self	The object itself.
config	The configuration object for the Hubert model. TYPE: HubertConfig
layer_id	The ID of the layer. Defaults to 0. TYPE: int DEFAULT: 0

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If the provided layer_id is less than 0.
TypeError	If the provided config is not an instance of HubertConfig.
KeyError	If the provided config does not contain required attributes.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def __init__(self, config: HubertConfig, layer_id=0):
    """
    Initializes a new instance of the HubertLayerNormConvLayer class.

    Args:
        self: The object itself.
        config (HubertConfig): The configuration object for the Hubert model.
        layer_id (int, optional): The ID of the layer. Defaults to 0.

    Returns:
        None.

    Raises:
        ValueError: If the provided layer_id is less than 0.
        TypeError: If the provided config is not an instance of HubertConfig.
        KeyError: If the provided config does not contain required attributes.
    """
    super().__init__()
    self.in_conv_dim = config.conv_dim[layer_id - 1] if layer_id > 0 else 1
    self.out_conv_dim = config.conv_dim[layer_id]

    self.conv = nn.Conv1d(
        self.in_conv_dim,
        self.out_conv_dim,
        kernel_size=config.conv_kernel[layer_id],
        stride=config.conv_stride[layer_id],
        bias=config.conv_bias,
        pad_mode="valid",
    )
    self.layer_norm = nn.LayerNorm(self.out_conv_dim)
    self.activation = ACT2FN[config.feat_extract_activation]

mindnlp.transformers.models.hubert.modeling_hubert.HubertLayerNormConvLayer.forward(hidden_states)

This method forwards a HubertLayerNormConvLayer by applying convolution, layer normalization, and activation functions to the input hidden states.

PARAMETER	DESCRIPTION
self	The instance of the HubertLayerNormConvLayer class.
hidden_states	A tensor representing the input hidden states that will undergo the transformation. It should have the shape (batch_size, sequence_length, hidden_size).

RETURNS	DESCRIPTION
None	This method does not return any value directly. The hidden_states tensor is modified in place and returned after the transformations.

RAISES	DESCRIPTION
ValueError	If the hidden_states tensor does not have the expected shape.
RuntimeError	If any error occurs during the convolution, layer normalization, or activation operations.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def forward(self, hidden_states):
    """
    This method forwards a HubertLayerNormConvLayer by applying convolution, layer normalization, and
    activation functions to the input hidden states.

    Args:
        self: The instance of the HubertLayerNormConvLayer class.
        hidden_states: A tensor representing the input hidden states that will undergo the transformation.
            It should have the shape (batch_size, sequence_length, hidden_size).

    Returns:
        None: This method does not return any value directly.
            The hidden_states tensor is modified in place and returned after the transformations.

    Raises:
        ValueError: If the hidden_states tensor does not have the expected shape.
        RuntimeError: If any error occurs during the convolution, layer normalization, or activation operations.
    """
    hidden_states = self.conv(hidden_states)
    hidden_states = hidden_states.swapaxes(-2, -1)
    hidden_states = self.layer_norm(hidden_states)
    hidden_states = hidden_states.swapaxes(-2, -1)
    hidden_states = self.activation(hidden_states)
    return hidden_states

mindnlp.transformers.models.hubert.modeling_hubert.HubertModel

Bases: HubertPreTrainedModel

A class representing a Hubert model for speech recognition tasks.

This class implements a Hubert model for processing speech input and generating relevant outputs. It includes methods for initializing the model, masking hidden states according to SpecAugment, and forwarding the model's forward pass. The model utilizes a feature extractor, feature projection, and an encoder for processing input data and generating output representations.

ATTRIBUTE	DESCRIPTION
config	HubertConfig
feature_extractor	HubertFeatureEncoder
feature_projection	HubertFeatureProjection
encoder	HubertEncoder or HubertEncoderStableLayerNorm based on configuration

METHOD	DESCRIPTION
__init__	Initializes the HubertModel with the provided configuration.
_mask_hidden_states	Masks hidden states along the time and/or feature axes based on SpecAugment.
forward	Constructs the forward pass of the model, processing input values and returning relevant outputs.

Example

>>> from transformers import AutoProcessor, HubertModel
>>> from datasets import load_dataset
>>> import soundfile as sf
...
>>> processor = AutoProcessor.from_pretrained("facebook/hubert-large-ls960-ft")
>>> model = HubertModel.from_pretrained("facebook/hubert-large-ls960-ft")
...
>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> ds = ds.map(map_to_array)
...
>>> input_values = processor(ds["speech"][0], return_tensors="pt").input_values  # Batch size 1
>>> hidden_states = model(input_values).last_hidden_state

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

class HubertModel(HubertPreTrainedModel):

    """
    A class representing a Hubert model for speech recognition tasks.

    This class implements a Hubert model for processing speech input and generating relevant outputs.
    It includes methods for initializing the model, masking hidden states according to SpecAugment, and
    forwarding the model's forward pass. The model utilizes a feature extractor, feature projection, and an encoder
    for processing input data and generating output representations.

    Attributes:
        config: HubertConfig
        feature_extractor: HubertFeatureEncoder
        feature_projection: HubertFeatureProjection
        encoder: HubertEncoder or HubertEncoderStableLayerNorm based on configuration

    Methods:
        __init__: Initializes the HubertModel with the provided configuration.
        _mask_hidden_states: Masks hidden states along the time and/or feature axes based on SpecAugment.
        forward: Constructs the forward pass of the model, processing input values and returning relevant outputs.

    Example:
        ```python
        >>> from transformers import AutoProcessor, HubertModel
        >>> from datasets import load_dataset
        >>> import soundfile as sf
        ...
        >>> processor = AutoProcessor.from_pretrained("facebook/hubert-large-ls960-ft")
        >>> model = HubertModel.from_pretrained("facebook/hubert-large-ls960-ft")
        ...
        >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
        >>> ds = ds.map(map_to_array)
        ...
        >>> input_values = processor(ds["speech"][0], return_tensors="pt").input_values  # Batch size 1
        >>> hidden_states = model(input_values).last_hidden_state
        ```
    """
    def __init__(self, config: HubertConfig):
        """
        Initializes the HubertModel with the provided configuration.

        Args:
            self: The instance of the HubertModel class.
            config (HubertConfig):
                An instance of the HubertConfig class representing the configuration settings for the model.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(config)
        self.config = config
        self.feature_extractor = HubertFeatureEncoder(config)
        self.feature_projection = HubertFeatureProjection(config)

        if config.mask_time_prob > 0.0 or config.mask_feature_prob > 0.0:
            self.masked_spec_embed = Parameter(initializer(Uniform(), (config.hidden_size,), dtype=mindspore.float32))

        if config.do_stable_layer_norm:
            self.encoder = HubertEncoderStableLayerNorm(config)
        else:
            self.encoder = HubertEncoder(config)

        # Initialize weights and apply final processing
        self.post_init()

    # Copied from transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2Model._mask_hidden_states
    def _mask_hidden_states(
        self,
        hidden_states: Tensor,
        mask_time_indices: Optional[Tensor] = None,
        attention_mask: Optional[Tensor] = None,
    ):
        """
        Masks extracted features along time axis and/or along feature axis according to
        [SpecAugment](https://arxiv.org/abs/1904.08779).
        """
        # `config.apply_spec_augment` can set masking to False
        if not getattr(self.config, "apply_spec_augment", True):
            return hidden_states

        # generate indices & apply SpecAugment along time axis
        batch_size, sequence_length, hidden_size = hidden_states.shape

        if mask_time_indices is not None:
            # apply SpecAugment along time axis with given mask_time_indices
            hidden_states[mask_time_indices] = self.masked_spec_embed.to(hidden_states.dtype)
        elif self.config.mask_time_prob > 0 and self.training:
            mask_time_indices = _compute_mask_indices(
                (batch_size, sequence_length),
                mask_prob=self.config.mask_time_prob,
                mask_length=self.config.mask_time_length,
                attention_mask=attention_mask,
                min_masks=self.config.mask_time_min_masks,
            )
            mask_time_indices = Tensor(mask_time_indices, dtype=mindspore.bool_)
            hidden_states[mask_time_indices] = self.masked_spec_embed.to(hidden_states.dtype)

        if self.config.mask_feature_prob > 0 and self.training:
            # generate indices & apply SpecAugment along feature axis
            mask_feature_indices = _compute_mask_indices(
                (batch_size, hidden_size),
                mask_prob=self.config.mask_feature_prob,
                mask_length=self.config.mask_feature_length,
                min_masks=self.config.mask_feature_min_masks,
            )
            mask_feature_indices = Tensor(mask_feature_indices, dtype=mindspore.bool_)
            mask_feature_indices = mask_feature_indices[:, None].expand(-1, sequence_length, -1)
            hidden_states[mask_feature_indices] = 0

        return hidden_states

    def forward(
        self,
        input_values: Optional[Tensor],
        attention_mask: Optional[Tensor] = None,
        mask_time_indices: Optional[Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, BaseModelOutput]:
        """

        Returns:
            Union[Tuple, BaseModelOutput]

        Example:
            ```python
            >>> from transformers import AutoProcessor, HubertModel
            >>> from datasets import load_dataset
            >>> import soundfile as sf
            ...
            >>> processor = AutoProcessor.from_pretrained("facebook/hubert-large-ls960-ft")
            >>> model = HubertModel.from_pretrained("facebook/hubert-large-ls960-ft")
            ...
            ...
            >>> def map_to_array(batch):
            ...     speech, _ = sf.read(batch["file"])
            ...     batch["speech"] = speech
            ...     return batch
            ...
            ...
            >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
            >>> ds = ds.map(map_to_array)
            ...
            >>> input_values = processor(ds["speech"][0], return_tensors="pt").input_values  # Batch size 1
            >>> hidden_states = model(input_values).last_hidden_state
            ```
        """
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        extract_features = self.feature_extractor(input_values)
        extract_features = extract_features.swapaxes(1, 2)

        if attention_mask is not None:
            # compute reduced attention_mask corresponding to feature vectors
            attention_mask = self._get_feature_vector_attention_mask(extract_features.shape[1], attention_mask)

        hidden_states = self.feature_projection(extract_features)
        hidden_states = self._mask_hidden_states(hidden_states, mask_time_indices=mask_time_indices)

        encoder_outputs = self.encoder(
            hidden_states,
            attention_mask=attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        hidden_states = encoder_outputs[0]

        if not return_dict:
            return (hidden_states,) + encoder_outputs[1:]

        return BaseModelOutput(
            last_hidden_state=hidden_states,
            hidden_states=encoder_outputs.hidden_states,
            attentions=encoder_outputs.attentions,
        )

mindnlp.transformers.models.hubert.modeling_hubert.HubertModel.__init__(config)

Initializes the HubertModel with the provided configuration.

PARAMETER	DESCRIPTION
self	The instance of the HubertModel class.
config	An instance of the HubertConfig class representing the configuration settings for the model. TYPE: HubertConfig

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def __init__(self, config: HubertConfig):
    """
    Initializes the HubertModel with the provided configuration.

    Args:
        self: The instance of the HubertModel class.
        config (HubertConfig):
            An instance of the HubertConfig class representing the configuration settings for the model.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(config)
    self.config = config
    self.feature_extractor = HubertFeatureEncoder(config)
    self.feature_projection = HubertFeatureProjection(config)

    if config.mask_time_prob > 0.0 or config.mask_feature_prob > 0.0:
        self.masked_spec_embed = Parameter(initializer(Uniform(), (config.hidden_size,), dtype=mindspore.float32))

    if config.do_stable_layer_norm:
        self.encoder = HubertEncoderStableLayerNorm(config)
    else:
        self.encoder = HubertEncoder(config)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.hubert.modeling_hubert.HubertModel.forward(input_values, attention_mask=None, mask_time_indices=None, output_attentions=None, output_hidden_states=None, return_dict=None)

RETURNS	DESCRIPTION
Union[Tuple, BaseModelOutput]	Union[Tuple, BaseModelOutput]

Example

>>> from transformers import AutoProcessor, HubertModel
>>> from datasets import load_dataset
>>> import soundfile as sf
...
>>> processor = AutoProcessor.from_pretrained("facebook/hubert-large-ls960-ft")
>>> model = HubertModel.from_pretrained("facebook/hubert-large-ls960-ft")
...
...
>>> def map_to_array(batch):
...     speech, _ = sf.read(batch["file"])
...     batch["speech"] = speech
...     return batch
...
...
>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> ds = ds.map(map_to_array)
...
>>> input_values = processor(ds["speech"][0], return_tensors="pt").input_values  # Batch size 1
>>> hidden_states = model(input_values).last_hidden_state

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def forward(
    self,
    input_values: Optional[Tensor],
    attention_mask: Optional[Tensor] = None,
    mask_time_indices: Optional[Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple, BaseModelOutput]:
    """

    Returns:
        Union[Tuple, BaseModelOutput]

    Example:
        ```python
        >>> from transformers import AutoProcessor, HubertModel
        >>> from datasets import load_dataset
        >>> import soundfile as sf
        ...
        >>> processor = AutoProcessor.from_pretrained("facebook/hubert-large-ls960-ft")
        >>> model = HubertModel.from_pretrained("facebook/hubert-large-ls960-ft")
        ...
        ...
        >>> def map_to_array(batch):
        ...     speech, _ = sf.read(batch["file"])
        ...     batch["speech"] = speech
        ...     return batch
        ...
        ...
        >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
        >>> ds = ds.map(map_to_array)
        ...
        >>> input_values = processor(ds["speech"][0], return_tensors="pt").input_values  # Batch size 1
        >>> hidden_states = model(input_values).last_hidden_state
        ```
    """
    output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
    output_hidden_states = (
        output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
    )
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    extract_features = self.feature_extractor(input_values)
    extract_features = extract_features.swapaxes(1, 2)

    if attention_mask is not None:
        # compute reduced attention_mask corresponding to feature vectors
        attention_mask = self._get_feature_vector_attention_mask(extract_features.shape[1], attention_mask)

    hidden_states = self.feature_projection(extract_features)
    hidden_states = self._mask_hidden_states(hidden_states, mask_time_indices=mask_time_indices)

    encoder_outputs = self.encoder(
        hidden_states,
        attention_mask=attention_mask,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    hidden_states = encoder_outputs[0]

    if not return_dict:
        return (hidden_states,) + encoder_outputs[1:]

    return BaseModelOutput(
        last_hidden_state=hidden_states,
        hidden_states=encoder_outputs.hidden_states,
        attentions=encoder_outputs.attentions,
    )

mindnlp.transformers.models.hubert.modeling_hubert.HubertNoLayerNormConvLayer

Bases: Module

HubertNoLayerNormConvLayer is a Python class representing a convolutional layer without layer normalization. This class inherits from nn.Module.

This class initializes with the following parameters:

• config: A HubertConfig object containing configuration settings.
• layer_id: An integer representing the layer identifier.

The forward method applies a convolutional operation and an activation function to the input hidden states.

ATTRIBUTE	DESCRIPTION
in_conv_dim	Integer representing the input convolutional dimension.
out_conv_dim	Integer representing the output convolutional dimension.
conv	nn.Conv1d object with parameters for the convolutional operation.
activation	Activation function defined in the ACT2FN dictionary based on the config's feat_extract_activation setting.

METHOD	DESCRIPTION
forward	Applies the convolutional operation and activation function to the input hidden_states.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

class HubertNoLayerNormConvLayer(nn.Module):

    """
    HubertNoLayerNormConvLayer is a Python class representing a convolutional layer without layer normalization.
    This class inherits from nn.Module.

    This class initializes with the following parameters:

    - config: A HubertConfig object containing configuration settings.
    - layer_id: An integer representing the layer identifier.

    The forward method applies a convolutional operation and an activation function to the input hidden states.

    Attributes:
        in_conv_dim: Integer representing the input convolutional dimension.
        out_conv_dim: Integer representing the output convolutional dimension.
        conv: nn.Conv1d object with parameters for the convolutional operation.
        activation: Activation function defined in the ACT2FN dictionary based on the config's feat_extract_activation setting.

    Methods:
        forward(hidden_states): Applies the convolutional operation and activation function to the input hidden_states.

    """
    def __init__(self, config: HubertConfig, layer_id=0):
        """
        Initializes a HubertNoLayerNormConvLayer.

        Args:
            self: The object itself.
            config (HubertConfig): The configuration object containing model hyperparameters.
            layer_id (int): The index of the convolution layer.

        Returns:
            None.

        Raises:
            ValueError: If layer_id is less than 0.
            KeyError: If the specified activation function in config is not found in the ACT2FN dictionary.
        """
        super().__init__()
        self.in_conv_dim = config.conv_dim[layer_id - 1] if layer_id > 0 else 1
        self.out_conv_dim = config.conv_dim[layer_id]

        self.conv = nn.Conv1d(
            self.in_conv_dim,
            self.out_conv_dim,
            kernel_size=config.conv_kernel[layer_id],
            stride=config.conv_stride[layer_id],
            bias=config.conv_bias,
            pad_mode="valid",
        )
        self.activation = ACT2FN[config.feat_extract_activation]

    def forward(self, hidden_states):
        """
        Constructs the hidden states of the HubertNoLayerNormConvLayer.

        Args:
            self (HubertNoLayerNormConvLayer): An instance of the HubertNoLayerNormConvLayer class.
            hidden_states (Tensor): The input hidden states to be processed.
                Expected shape is (batch_size, channels, sequence_length).

        Returns:
            Tensor: The processed hidden states after applying the convolutional layer and activation function.
                The shape is (batch_size, channels, sequence_length).

        Raises:
            None.
        """
        hidden_states = self.conv(hidden_states)
        hidden_states = self.activation(hidden_states)
        return hidden_states

mindnlp.transformers.models.hubert.modeling_hubert.HubertNoLayerNormConvLayer.__init__(config, layer_id=0)

Initializes a HubertNoLayerNormConvLayer.

PARAMETER	DESCRIPTION
self	The object itself.
config	The configuration object containing model hyperparameters. TYPE: HubertConfig
layer_id	The index of the convolution layer. TYPE: int DEFAULT: 0

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If layer_id is less than 0.
KeyError	If the specified activation function in config is not found in the ACT2FN dictionary.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def __init__(self, config: HubertConfig, layer_id=0):
    """
    Initializes a HubertNoLayerNormConvLayer.

    Args:
        self: The object itself.
        config (HubertConfig): The configuration object containing model hyperparameters.
        layer_id (int): The index of the convolution layer.

    Returns:
        None.

    Raises:
        ValueError: If layer_id is less than 0.
        KeyError: If the specified activation function in config is not found in the ACT2FN dictionary.
    """
    super().__init__()
    self.in_conv_dim = config.conv_dim[layer_id - 1] if layer_id > 0 else 1
    self.out_conv_dim = config.conv_dim[layer_id]

    self.conv = nn.Conv1d(
        self.in_conv_dim,
        self.out_conv_dim,
        kernel_size=config.conv_kernel[layer_id],
        stride=config.conv_stride[layer_id],
        bias=config.conv_bias,
        pad_mode="valid",
    )
    self.activation = ACT2FN[config.feat_extract_activation]

mindnlp.transformers.models.hubert.modeling_hubert.HubertNoLayerNormConvLayer.forward(hidden_states)

Constructs the hidden states of the HubertNoLayerNormConvLayer.

PARAMETER	DESCRIPTION
self	An instance of the HubertNoLayerNormConvLayer class. TYPE: HubertNoLayerNormConvLayer
hidden_states	The input hidden states to be processed. Expected shape is (batch_size, channels, sequence_length). TYPE: Tensor

RETURNS	DESCRIPTION
Tensor	The processed hidden states after applying the convolutional layer and activation function. The shape is (batch_size, channels, sequence_length).

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def forward(self, hidden_states):
    """
    Constructs the hidden states of the HubertNoLayerNormConvLayer.

    Args:
        self (HubertNoLayerNormConvLayer): An instance of the HubertNoLayerNormConvLayer class.
        hidden_states (Tensor): The input hidden states to be processed.
            Expected shape is (batch_size, channels, sequence_length).

    Returns:
        Tensor: The processed hidden states after applying the convolutional layer and activation function.
            The shape is (batch_size, channels, sequence_length).

    Raises:
        None.
    """
    hidden_states = self.conv(hidden_states)
    hidden_states = self.activation(hidden_states)
    return hidden_states

mindnlp.transformers.models.hubert.modeling_hubert.HubertPositionalConvEmbedding

Bases: Module

Represents a Positional Convolutional Embedding layer for the Hubert model.

This class inherits from nn.Module and is used to apply positional convolutional embeddings to input hidden states. The layer uses a convolutional neural network to process the input hidden states with configurable parameters such as kernel size, padding, activation function, and bias.

ATTRIBUTE	DESCRIPTION
conv	Convolutional layer for processing hidden states. TYPE: Conv1d
padding	Padding layer to ensure input dimensions match convolutional operations. TYPE: HubertSamePadLayer
activation	Activation function to apply after convolution and padding. TYPE: ACT2FN

METHOD	DESCRIPTION
__init__	Initializes the Positional Convolutional Embedding layer with the specified configuration.
forward	Constructs the positional convolutional embedding by applying convolution, padding, and activation functions to the input hidden states.

RETURNS	DESCRIPTION
	The processed hidden states with positional convolutional embeddings applied.

Note

This class is designed specifically for the Hubert model and should be used within the model architecture for optimal performance.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

class HubertPositionalConvEmbedding(nn.Module):

    """
    Represents a Positional Convolutional Embedding layer for the Hubert model.

    This class inherits from nn.Module and is used to apply positional convolutional embeddings to input hidden states.
    The layer uses a convolutional neural network to process the input hidden states with configurable parameters
    such as kernel size, padding, activation function, and bias.

    Attributes:
        conv (nn.Conv1d): Convolutional layer for processing hidden states.
        padding (HubertSamePadLayer): Padding layer to ensure input dimensions match convolutional operations.
        activation (ACT2FN): Activation function to apply after convolution and padding.

    Methods:
        __init__: Initializes the Positional Convolutional Embedding layer with the specified configuration.
        forward: Constructs the positional convolutional embedding by applying convolution, padding,
            and activation functions to the input hidden states.

    Returns:
        The processed hidden states with positional convolutional embeddings applied.

    Note:
        This class is designed specifically for the Hubert model and should be used within the model architecture
        for optimal performance.
    """
    def __init__(self, config: HubertConfig):
        """
        Initializes an instance of the HubertPositionalConvEmbedding class.

        Args:
            self: The instance of the class.
            config (HubertConfig): The configuration object containing the settings for Hubert model.

        Returns:
            None

        Raises:
            None
        """
        super().__init__()
        self.conv = nn.Conv1d(
            config.hidden_size,
            config.hidden_size,
            kernel_size=config.num_conv_pos_embeddings,
            pad_mode='pad',
            padding=config.num_conv_pos_embeddings // 2,
            group=config.num_conv_pos_embedding_groups,
            bias=True,      # TODO: confirm this
        )
        self.conv = weight_norm(self.conv, name='weight', dim=2)
        self.padding = HubertSamePadLayer(config.num_conv_pos_embeddings)
        self.activation = ACT2FN[config.feat_extract_activation]

    def forward(self, hidden_states):
        """
        Constructs the HubertPositionalConvEmbedding.

        Args:
            self (HubertPositionalConvEmbedding): The instance of the HubertPositionalConvEmbedding class.
            hidden_states (numpy.ndarray):
                The input hidden states of shape (batch_size, sequence_length, hidden_size), where batch_size
                represents the number of input samples, sequence_length represents the length of the input sequence,
                and hidden_size represents the dimensionality of the hidden states.
                The hidden states are expected to be in the format (batch_size, hidden_size, sequence_length).

        Returns:
            None.

        Raises:
            None.
        """
        hidden_states = hidden_states.swapaxes(1, 2)
        hidden_states = self.conv(hidden_states)
        hidden_states = self.padding(hidden_states)
        hidden_states = self.activation(hidden_states)
        hidden_states = hidden_states.swapaxes(1, 2)
        return hidden_states

mindnlp.transformers.models.hubert.modeling_hubert.HubertPositionalConvEmbedding.__init__(config)

Initializes an instance of the HubertPositionalConvEmbedding class.

PARAMETER	DESCRIPTION
self	The instance of the class.
config	The configuration object containing the settings for Hubert model. TYPE: HubertConfig

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def __init__(self, config: HubertConfig):
    """
    Initializes an instance of the HubertPositionalConvEmbedding class.

    Args:
        self: The instance of the class.
        config (HubertConfig): The configuration object containing the settings for Hubert model.

    Returns:
        None

    Raises:
        None
    """
    super().__init__()
    self.conv = nn.Conv1d(
        config.hidden_size,
        config.hidden_size,
        kernel_size=config.num_conv_pos_embeddings,
        pad_mode='pad',
        padding=config.num_conv_pos_embeddings // 2,
        group=config.num_conv_pos_embedding_groups,
        bias=True,      # TODO: confirm this
    )
    self.conv = weight_norm(self.conv, name='weight', dim=2)
    self.padding = HubertSamePadLayer(config.num_conv_pos_embeddings)
    self.activation = ACT2FN[config.feat_extract_activation]

mindnlp.transformers.models.hubert.modeling_hubert.HubertPositionalConvEmbedding.forward(hidden_states)

Constructs the HubertPositionalConvEmbedding.

PARAMETER	DESCRIPTION
self	The instance of the HubertPositionalConvEmbedding class. TYPE: HubertPositionalConvEmbedding
hidden_states	The input hidden states of shape (batch_size, sequence_length, hidden_size), where batch_size represents the number of input samples, sequence_length represents the length of the input sequence, and hidden_size represents the dimensionality of the hidden states. The hidden states are expected to be in the format (batch_size, hidden_size, sequence_length). TYPE: ndarray

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def forward(self, hidden_states):
    """
    Constructs the HubertPositionalConvEmbedding.

    Args:
        self (HubertPositionalConvEmbedding): The instance of the HubertPositionalConvEmbedding class.
        hidden_states (numpy.ndarray):
            The input hidden states of shape (batch_size, sequence_length, hidden_size), where batch_size
            represents the number of input samples, sequence_length represents the length of the input sequence,
            and hidden_size represents the dimensionality of the hidden states.
            The hidden states are expected to be in the format (batch_size, hidden_size, sequence_length).

    Returns:
        None.

    Raises:
        None.
    """
    hidden_states = hidden_states.swapaxes(1, 2)
    hidden_states = self.conv(hidden_states)
    hidden_states = self.padding(hidden_states)
    hidden_states = self.activation(hidden_states)
    hidden_states = hidden_states.swapaxes(1, 2)
    return hidden_states

mindnlp.transformers.models.hubert.modeling_hubert.HubertPreTrainedModel

Bases: PreTrainedModel

An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

class HubertPreTrainedModel(PreTrainedModel):
    """
    An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained
    models.
    """
    config_class = HubertConfig
    base_model_prefix = "hubert"
    main_input_name = "input_values"

    def _init_weights(self, cell):
        """Initialize the weights"""
        if isinstance(cell, nn.Linear):
            # Slightly different from the TF version which uses truncated_normal for initialization
            # cf https://github.com/pytorch/pytorch/pull/5617
            cell.weight.set_data(initializer(Normal(self.config.initializer_range), cell.weight.shape, cell.weight.dtype))
        elif isinstance(cell, (nn.LayerNorm, nn.GroupNorm)):
            cell.weight.set_data(initializer('ones', cell.weight.shape, cell.weight.dtype))
            cell.bias.set_data(initializer('zeros', cell.bias.shape, cell.bias.dtype))
        elif isinstance(cell, nn.Conv1d):
            cell.weight.set_data(initializer(HeNormal(), cell.weight.shape, cell.weight.dtype))
        if isinstance(cell, (nn.Linear, nn.Conv1d)) and cell.bias is not None:
            cell.bias.set_data(initializer('zeros', cell.bias.shape, cell.bias.dtype))

    def _get_feat_extract_output_lengths(self, input_lengths: Union[Tensor, int]):
        """
        Computes the output length of the convolutional layers
        """
        def _conv_out_length(input_length, kernel_size, stride):
            # 1D convolutional layer output length formula taken
            # from https://pytorch.org/docs/stable/generated/ops.nn.Conv1d.html
            #return ops.div(input_length - kernel_size, stride, rounding_mode="floor") + 1
            return (input_length - kernel_size) // stride + 1

        for kernel_size, stride in zip(self.config.conv_kernel, self.config.conv_stride):
            input_lengths = _conv_out_length(input_lengths, kernel_size, stride)
        return input_lengths

    def _get_feature_vector_attention_mask(self, feature_vector_length: int, attention_mask: Tensor):
        """
        Method _get_feature_vector_attention_mask in class HubertPreTrainedModel.

        This method calculates the attention mask for feature vectors based on the provided feature_vector_length
        and attention_mask.

        Args:
            self: The instance of the class.
            feature_vector_length (int): The length of the feature vectors.
                This parameter specifies the length of the feature vectors to be used in calculating the attention mask.
                It must be a positive integer.
            attention_mask (Tensor): The attention mask tensor.
                This tensor indicates the positions of the actual tokens in the input sequence.
                It should be a 2D tensor with shape (batch_size, sequence_length) where batch_size is the number of sequences
                in the batch and sequence_length is the length of each sequence.

        Returns:
            None: This method does not return any value. It modifies the input attention_mask tensor in-place to generate the
                feature vector attention mask.

        Raises:
            ValueError: If feature_vector_length is not a positive integer or if attention_mask is not a valid tensor.
            IndexError: If there is an index out of range error during tensor operations.
            TypeError: If the data type of the attention_mask tensor is not supported for the operations in the method.
        """
        output_lengths = self._get_feat_extract_output_lengths(attention_mask.sum(-1)).to(mindspore.int64)
        batch_size = attention_mask.shape[0]

        attention_mask = ops.zeros((batch_size, feature_vector_length), dtype=attention_mask.dtype)
        # these two operations makes sure that all values before the output lengths idxs are attended to
        attention_mask[(ops.arange(attention_mask.shape[0]), output_lengths - 1)] = 1
        attention_mask = attention_mask.flip([-1]).cumsum(-1).flip([-1]).bool()
        return attention_mask

mindnlp.transformers.models.hubert.modeling_hubert.HubertSamePadLayer

Bases: Module

This class represents a layer in the Hubert model that performs same padding on the input hidden states.

The HubertSamePadLayer class is a subclass of the nn.Module class and provides functionality to remove padding from the input hidden states if necessary. It is specifically designed for the Hubert model and is used to ensure that the input hidden states have the same length as the target sequence for further processing.

ATTRIBUTE	DESCRIPTION
num_pad_remove	The number of padding elements to remove from the input hidden states. It is determined based on the number of convolutional positional embeddings. If the number is even, num_pad_remove is set to 1, otherwise it is set to 0. TYPE: int

METHOD	DESCRIPTION
__init__	Initializes a new instance of the HubertSamePadLayer class. Args: num_conv_pos_embeddings (int): The number of convolutional positional embeddings.
forward	Constructs the output hidden states by removing the padding elements if necessary. Args: hidden_states (Tensor): The input hidden states to be processed. Returns: Tensor: The processed hidden states with padding elements removed if necessary.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

class HubertSamePadLayer(nn.Module):

    """
    This class represents a layer in the Hubert model that performs same padding on the input hidden states.

    The HubertSamePadLayer class is a subclass of the nn.Module class and provides functionality to remove padding from
    the input hidden states if necessary. It is specifically designed for the Hubert model and is used to ensure that
    the input hidden states have the same length as the target sequence for further processing.

    Attributes:
        num_pad_remove (int): The number of padding elements to remove from the input hidden states.
            It is determined based on the number of convolutional positional embeddings. If the number is even,
            num_pad_remove is set to 1, otherwise it is set to 0.

    Methods:
        __init__(num_conv_pos_embeddings):
            Initializes a new instance of the HubertSamePadLayer class.

            Args:

            - num_conv_pos_embeddings (int): The number of convolutional positional embeddings.

        forward(hidden_states):
            Constructs the output hidden states by removing the padding elements if necessary.

            Args:

            - hidden_states (Tensor): The input hidden states to be processed.

            Returns:

            - Tensor: The processed hidden states with padding elements removed if necessary.
    """
    def __init__(self, num_conv_pos_embeddings):
        """
        Args:
            self (object): The instance of the class.
            num_conv_pos_embeddings (int): The number of convolutional position embeddings used in the layer.
                It is used to calculate the value of 'num_pad_remove' based on whether it is even or odd.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.num_pad_remove = 1 if num_conv_pos_embeddings % 2 == 0 else 0

    def forward(self, hidden_states):
        """
        Constructs the hidden states for the HubertSamePadLayer.

        Args:
            self (HubertSamePadLayer): An instance of the HubertSamePadLayer class.
            hidden_states (Tensor): The input hidden states to be processed.
                Expected shape is (batch_size, sequence_length, hidden_size).

        Returns:
            None.

        Raises:
            None.
        """
        if self.num_pad_remove > 0:
            hidden_states = hidden_states[:, :, : -self.num_pad_remove]
        return hidden_states

mindnlp.transformers.models.hubert.modeling_hubert.HubertSamePadLayer.__init__(num_conv_pos_embeddings)

PARAMETER	DESCRIPTION
self	The instance of the class. TYPE: object
num_conv_pos_embeddings	The number of convolutional position embeddings used in the layer. It is used to calculate the value of 'num_pad_remove' based on whether it is even or odd. TYPE: int

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def __init__(self, num_conv_pos_embeddings):
    """
    Args:
        self (object): The instance of the class.
        num_conv_pos_embeddings (int): The number of convolutional position embeddings used in the layer.
            It is used to calculate the value of 'num_pad_remove' based on whether it is even or odd.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.num_pad_remove = 1 if num_conv_pos_embeddings % 2 == 0 else 0

mindnlp.transformers.models.hubert.modeling_hubert.HubertSamePadLayer.forward(hidden_states)

Constructs the hidden states for the HubertSamePadLayer.

PARAMETER	DESCRIPTION
self	An instance of the HubertSamePadLayer class. TYPE: HubertSamePadLayer
hidden_states	The input hidden states to be processed. Expected shape is (batch_size, sequence_length, hidden_size). TYPE: Tensor

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/hubert/modeling_hubert.py

def forward(self, hidden_states):
    """
    Constructs the hidden states for the HubertSamePadLayer.

    Args:
        self (HubertSamePadLayer): An instance of the HubertSamePadLayer class.
        hidden_states (Tensor): The input hidden states to be processed.
            Expected shape is (batch_size, sequence_length, hidden_size).

    Returns:
        None.

    Raises:
        None.
    """
    if self.num_pad_remove > 0:
        hidden_states = hidden_states[:, :, : -self.num_pad_remove]
    return hidden_states