wavlm

mindnlp.transformers.models.wavlm.configuration_wavlm

WavLM model configuration

mindnlp.transformers.models.wavlm.configuration_wavlm.WavLMConfig

Bases: PretrainedConfig

This is the configuration class to store the configuration of a [WavLMModel]. It is used to instantiate an WavLM model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the WavLM microsoft/wavlm-base architecture.

Configuration objects inherit from [PretrainedConfig] and can be used to control the model outputs. Read the documentation from [PretrainedConfig] for more information.

PARAMETER	DESCRIPTION
vocab_size	Vocabulary size of the WavLM model. Defines the number of different tokens that can be represented by the inputs_ids passed when calling [WavLMModel]. Vocabulary size of the model. Defines the different tokens that can be represented by the inputs_ids passed to the forward method of [WavLMModel]. TYPE: `int`, *optional*, defaults to 32 DEFAULT: 32
hidden_size	Dimensionality of the encoder layers and the pooler layer. TYPE: `int`, *optional*, defaults to 768 DEFAULT: 768
num_hidden_layers	Number of hidden layers in the Transformer encoder. TYPE: `int`, *optional*, defaults to 12 DEFAULT: 12
num_attention_heads	Number of attention heads for each attention layer in the Transformer encoder. TYPE: `int`, *optional*, defaults to 12 DEFAULT: 12
intermediate_size	Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. TYPE: `int`, *optional*, defaults to 3072 DEFAULT: 3072
hidden_act	The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu", "relu", "selu" and "gelu_new" are supported. TYPE: `str` or `function`, *optional*, defaults to `"gelu"` DEFAULT: 'gelu'
hidden_dropout	The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
activation_dropout	The dropout ratio for activations inside the fully connected layer. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
attention_dropout	The dropout ratio for the attention probabilities. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
final_dropout	The dropout probability for the final projection layer of [WavLMForCTC]. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
layerdrop	The LayerDrop probability. See the LayerDrop paper for more details. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
initializer_range	The standard deviation of the truncated_normal_initializer for initializing all weight matrices. TYPE: `float`, *optional*, defaults to 0.02 DEFAULT: 0.02
layer_norm_eps	The epsilon used by the layer normalization layers. TYPE: `float`, *optional*, defaults to 1e-12 DEFAULT: 1e-05
feat_extract_norm	The norm to be applied to 1D convolutional layers in feature encoder. One of "group" for group normalization of only the first 1D convolutional layer or "layer" for layer normalization of all 1D convolutional layers. TYPE: `str`, *optional*, defaults to `"group"` DEFAULT: 'group'
feat_proj_dropout	The dropout probability for output of the feature encoder. TYPE: `float`, *optional*, defaults to 0.0 DEFAULT: 0.0
feat_extract_activation	The non-linear activation function (function or string) in the 1D convolutional layers of the feature extractor. If string, "gelu", "relu", "selu" and "gelu_new" are supported. TYPE: `str, `optional`, defaults to `"gelu"` DEFAULT: 'gelu'
conv_dim	A tuple of integers defining the number of input and output channels of each 1D convolutional layer in the feature encoder. The length of conv_dim defines the number of 1D convolutional layers. TYPE: `Tuple[int]` or `List[int]`, *optional*, defaults to `(512, 512, 512, 512, 512, 512, 512)` DEFAULT: (512, 512, 512, 512, 512, 512, 512)
conv_stride	A tuple of integers defining the stride of each 1D convolutional layer in the feature encoder. The length of conv_stride defines the number of convolutional layers and has to match the length of conv_dim. TYPE: `Tuple[int]` or `List[int]`, *optional*, defaults to `(5, 2, 2, 2, 2, 2, 2)` DEFAULT: (5, 2, 2, 2, 2, 2, 2)
conv_kernel	A tuple of integers defining the kernel size of each 1D convolutional layer in the feature encoder. The length of conv_kernel defines the number of convolutional layers and has to match the length of conv_dim. TYPE: `Tuple[int]` or `List[int]`, *optional*, defaults to `(10, 3, 3, 3, 3, 3, 3)` DEFAULT: (10, 3, 3, 3, 3, 2, 2)
conv_bias	Whether the 1D convolutional layers have a bias. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
num_conv_pos_embeddings	Number of convolutional positional embeddings. Defines the kernel size of 1D convolutional positional embeddings layer. TYPE: `int`, *optional*, defaults to 128 DEFAULT: 128
num_conv_pos_embedding_groups	Number of groups of 1D convolutional positional embeddings layer. TYPE: `int`, *optional*, defaults to 16 DEFAULT: 16
do_stable_layer_norm	Whether to apply stable layer norm architecture of the Transformer encoder. do_stable_layer_norm is True corresponds to applying layer norm before the attention layer, whereas do_stable_layer_norm is False corresponds to applying layer norm after the attention layer. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
apply_spec_augment	Whether to apply SpecAugment data augmentation to the outputs of the feature encoder. For reference see SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition. TYPE: `bool`, *optional*, defaults to `True` DEFAULT: True
mask_time_prob	Propability of each feature vector along the time axis to be chosen as the start of the vector span to be masked. Approximately mask_time_prob * sequence_length // mask_time_length feature vectors will be masked along the time axis. This is only relevant if apply_spec_augment is True. TYPE: `float`, *optional*, defaults to 0.05 DEFAULT: 0.05
mask_time_length	Length of vector span along the time axis. TYPE: `int`, *optional*, defaults to 10 DEFAULT: 10
mask_time_min_masks	The minimum number of masks of length mask_feature_length generated along the time axis, each time step, irrespectively of mask_feature_prob. Only relevant if ''mask_time_prob*len(time_axis)/mask_time_length < mask_time_min_masks'' TYPE: `int`, *optional*, defaults to 2), DEFAULT: 2
mask_feature_prob	Propability of each feature vector along the feature axis to be chosen as the start of the vector span to be masked. Approximately mask_time_prob * hidden_size // mask_time_length feature vectors will be masked along the time axis. This is only relevant if apply_spec_augment is True. TYPE: `float`, *optional*, defaults to 0.0 DEFAULT: 0.0
mask_feature_length	Length of vector span along the feature axis. TYPE: `int`, *optional*, defaults to 10 DEFAULT: 10
num_codevectors_per_group	Number of entries in each quantization codebook (group). TYPE: `int`, *optional*, defaults to 320 DEFAULT: 320
num_codevector_groups	Number of codevector groups for product codevector quantization. TYPE: `int`, *optional*, defaults to 2 DEFAULT: 2
contrastive_logits_temperature	The temperature kappa in the contrastive loss. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
num_negatives	Number of negative samples for the contrastive loss. TYPE: `int`, *optional*, defaults to 100 DEFAULT: 100
codevector_dim	Dimensionality of the quantized feature vectors. TYPE: `int`, *optional*, defaults to 256 DEFAULT: 256
proj_codevector_dim	Dimensionality of the final projection of both the quantized and the transformer features. TYPE: `int`, *optional*, defaults to 256 DEFAULT: 256
diversity_loss_weight	The weight of the codebook diversity loss component. TYPE: `int`, *optional*, defaults to 0.1 DEFAULT: 0.1
ctc_loss_reduction	Specifies the reduction to apply to the output of torch.nn.CTCLoss. Only relevant when training an instance of [WavLMForCTC]. TYPE: `str`, *optional*, defaults to `"mean"` DEFAULT: 'mean'
ctc_zero_infinity	Whether to zero infinite losses and the associated gradients of torch.nn.CTCLoss. Infinite losses mainly occur when the inputs are too short to be aligned to the targets. Only relevant when training an instance of [WavLMForCTC]. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
use_weighted_layer_sum	Whether to use a weighted average of layer outputs with learned weights. Only relevant when using an instance of [WavLMForSequenceClassification]. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
classifier_proj_size	Dimensionality of the projection before token mean-pooling for classification. TYPE: `int`, *optional*, defaults to 256 DEFAULT: 256
tdnn_dim	A tuple of integers defining the number of output channels of each 1D convolutional layer in the TDNN module of the XVector model. The length of tdnn_dim defines the number of TDNN layers. TYPE: `Tuple[int]` or `List[int]`, *optional*, defaults to `(512, 512, 512, 512, 1500)` DEFAULT: (512, 512, 512, 512, 1500)
tdnn_kernel	A tuple of integers defining the kernel size of each 1D convolutional layer in the TDNN module of the XVector model. The length of tdnn_kernel has to match the length of tdnn_dim. TYPE: `Tuple[int]` or `List[int]`, *optional*, defaults to `(5, 3, 3, 1, 1)` DEFAULT: (5, 3, 3, 1, 1)
tdnn_dilation	A tuple of integers defining the dilation factor of each 1D convolutional layer in TDNN module of the XVector model. The length of tdnn_dilation has to match the length of tdnn_dim. TYPE: `Tuple[int]` or `List[int]`, *optional*, defaults to `(1, 2, 3, 1, 1)` DEFAULT: (1, 2, 3, 1, 1)
xvector_output_dim	Dimensionality of the XVector embedding vectors. TYPE: `int`, *optional*, defaults to 512 DEFAULT: 512
add_adapter	Whether a convolutional network should be stacked on top of the Wav2Vec2 Encoder. Can be very useful for warm-starting Wav2Vec2 for SpeechEncoderDecoder models. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
adapter_kernel_size	Kernel size of the convolutional layers in the adapter network. Only relevant if add_adapter is True. TYPE: `int`, *optional*, defaults to 3 DEFAULT: 3
adapter_stride	Stride of the convolutional layers in the adapter network. Only relevant if add_adapter is True. TYPE: `int`, *optional*, defaults to 2 DEFAULT: 2
num_adapter_layers	Number of convolutional layers that should be used in the adapter network. Only relevant if add_adapter is True. TYPE: `int`, *optional*, defaults to 3 DEFAULT: 3
output_hidden_size	Dimensionality of the encoder output layer. If not defined, this defaults to hidden-size. Only relevant if add_adapter is True. TYPE: `int`, *optional* DEFAULT: None

Example

>>> from transformers import WavLMConfig, WavLMModel
...
>>> # Initializing a WavLM facebook/wavlm-base-960h style configuration
>>> configuration = WavLMConfig()
...
>>> # Initializing a model (with random weights) from the facebook/wavlm-base-960h style configuration
>>> model = WavLMModel(configuration)
...
>>> # Accessing the model configuration
>>> configuration = model.config

Source code in mindnlp/transformers/models/wavlm/configuration_wavlm.py

class WavLMConfig(PretrainedConfig):
    r"""
    This is the configuration class to store the configuration of a [`WavLMModel`]. It is used to instantiate an WavLM
    model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
    defaults will yield a similar configuration to that of the WavLM
    [microsoft/wavlm-base](https://huggingface.co/microsoft/wavlm-base) architecture.

    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
    documentation from [`PretrainedConfig`] for more information.


    Args:
        vocab_size (`int`, *optional*, defaults to 32):
            Vocabulary size of the WavLM model. Defines the number of different tokens that can be represented by the
            `inputs_ids` passed when calling [`WavLMModel`]. Vocabulary size of the model. Defines the different tokens
            that can be represented by the *inputs_ids* passed to the forward method of [`WavLMModel`].
        hidden_size (`int`, *optional*, defaults to 768):
            Dimensionality of the encoder layers and the pooler layer.
        num_hidden_layers (`int`, *optional*, defaults to 12):
            Number of hidden layers in the Transformer encoder.
        num_attention_heads (`int`, *optional*, defaults to 12):
            Number of attention heads for each attention layer in the Transformer encoder.
        intermediate_size (`int`, *optional*, defaults to 3072):
            Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder.
        hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`):
            The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`,
            `"relu"`, `"selu"` and `"gelu_new"` are supported.
        hidden_dropout (`float`, *optional*, defaults to 0.1):
            The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
        activation_dropout (`float`, *optional*, defaults to 0.1):
            The dropout ratio for activations inside the fully connected layer.
        attention_dropout (`float`, *optional*, defaults to 0.1):
            The dropout ratio for the attention probabilities.
        final_dropout (`float`, *optional*, defaults to 0.1):
            The dropout probability for the final projection layer of [`WavLMForCTC`].
        layerdrop (`float`, *optional*, defaults to 0.1):
            The LayerDrop probability. See the [LayerDrop paper](see https://arxiv.org/abs/1909.11556) for more
            details.
        initializer_range (`float`, *optional*, defaults to 0.02):
            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
        layer_norm_eps (`float`, *optional*, defaults to 1e-12):
            The epsilon used by the layer normalization layers.
        feat_extract_norm (`str`, *optional*, defaults to `"group"`):
            The norm to be applied to 1D convolutional layers in feature encoder. One of `"group"` for group
            normalization of only the first 1D convolutional layer or `"layer"` for layer normalization of all 1D
            convolutional layers.
        feat_proj_dropout (`float`, *optional*, defaults to 0.0):
            The dropout probability for output of the feature encoder.
        feat_extract_activation (`str, `optional`, defaults to `"gelu"`):
            The non-linear activation function (function or string) in the 1D convolutional layers of the feature
            extractor. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` are supported.
        conv_dim (`Tuple[int]` or `List[int]`, *optional*, defaults to `(512, 512, 512, 512, 512, 512, 512)`):
            A tuple of integers defining the number of input and output channels of each 1D convolutional layer in the
            feature encoder. The length of *conv_dim* defines the number of 1D convolutional layers.
        conv_stride (`Tuple[int]` or `List[int]`, *optional*, defaults to `(5, 2, 2, 2, 2, 2, 2)`):
            A tuple of integers defining the stride of each 1D convolutional layer in the feature encoder. The length
            of *conv_stride* defines the number of convolutional layers and has to match the length of *conv_dim*.
        conv_kernel (`Tuple[int]` or `List[int]`, *optional*, defaults to `(10, 3, 3, 3, 3, 3, 3)`):
            A tuple of integers defining the kernel size of each 1D convolutional layer in the feature encoder. The
            length of *conv_kernel* defines the number of convolutional layers and has to match the length of
            *conv_dim*.
        conv_bias (`bool`, *optional*, defaults to `False`):
            Whether the 1D convolutional layers have a bias.
        num_conv_pos_embeddings (`int`, *optional*, defaults to 128):
            Number of convolutional positional embeddings. Defines the kernel size of 1D convolutional positional
            embeddings layer.
        num_conv_pos_embedding_groups (`int`, *optional*, defaults to 16):
            Number of groups of 1D convolutional positional embeddings layer.
        do_stable_layer_norm (`bool`, *optional*, defaults to `False`):
            Whether to apply *stable* layer norm architecture of the Transformer encoder. `do_stable_layer_norm is
            True` corresponds to applying layer norm before the attention layer, whereas `do_stable_layer_norm is
            False` corresponds to applying layer norm after the attention layer.
        apply_spec_augment (`bool`, *optional*, defaults to `True`):
            Whether to apply *SpecAugment* data augmentation to the outputs of the feature encoder. For reference see
            [SpecAugment: A Simple Data Augmentation Method for Automatic Speech
            Recognition](https://arxiv.org/abs/1904.08779).
        mask_time_prob (`float`, *optional*, defaults to 0.05):
            Propability of each feature vector along the time axis to be chosen as the start of the vector span to be
            masked. Approximately `mask_time_prob * sequence_length // mask_time_length` feature vectors will be masked
            along the time axis. This is only relevant if `apply_spec_augment is True`.
        mask_time_length (`int`, *optional*, defaults to 10):
            Length of vector span along the time axis.
        mask_time_min_masks (`int`, *optional*, defaults to 2),:
            The minimum number of masks of length `mask_feature_length` generated along the time axis, each time step,
            irrespectively of `mask_feature_prob`. Only relevant if ''mask_time_prob*len(time_axis)/mask_time_length <
            mask_time_min_masks''
        mask_feature_prob (`float`, *optional*, defaults to 0.0):
            Propability of each feature vector along the feature axis to be chosen as the start of the vector span to
            be masked. Approximately `mask_time_prob * hidden_size // mask_time_length` feature vectors will be masked
            along the time axis. This is only relevant if `apply_spec_augment is True`.
        mask_feature_length (`int`, *optional*, defaults to 10):
            Length of vector span along the feature axis.
        num_codevectors_per_group (`int`, *optional*, defaults to 320):
            Number of entries in each quantization codebook (group).
        num_codevector_groups (`int`, *optional*, defaults to 2):
            Number of codevector groups for product codevector quantization.
        contrastive_logits_temperature (`float`, *optional*, defaults to 0.1):
            The temperature *kappa* in the contrastive loss.
        num_negatives (`int`, *optional*, defaults to 100):
            Number of negative samples for the contrastive loss.
        codevector_dim (`int`, *optional*, defaults to 256):
            Dimensionality of the quantized feature vectors.
        proj_codevector_dim (`int`, *optional*, defaults to 256):
            Dimensionality of the final projection of both the quantized and the transformer features.
        diversity_loss_weight (`int`, *optional*, defaults to 0.1):
            The weight of the codebook diversity loss component.
        ctc_loss_reduction (`str`, *optional*, defaults to `"mean"`):
            Specifies the reduction to apply to the output of `torch.nn.CTCLoss`. Only relevant when training an
            instance of [`WavLMForCTC`].
        ctc_zero_infinity (`bool`, *optional*, defaults to `False`):
            Whether to zero infinite losses and the associated gradients of `torch.nn.CTCLoss`. Infinite losses mainly
            occur when the inputs are too short to be aligned to the targets. Only relevant when training an instance
            of [`WavLMForCTC`].
        use_weighted_layer_sum (`bool`, *optional*, defaults to `False`):
            Whether to use a weighted average of layer outputs with learned weights. Only relevant when using an
            instance of [`WavLMForSequenceClassification`].
        classifier_proj_size (`int`, *optional*, defaults to 256):
            Dimensionality of the projection before token mean-pooling for classification.
        tdnn_dim (`Tuple[int]` or `List[int]`, *optional*, defaults to `(512, 512, 512, 512, 1500)`):
            A tuple of integers defining the number of output channels of each 1D convolutional layer in the *TDNN*
            module of the *XVector* model. The length of *tdnn_dim* defines the number of *TDNN* layers.
        tdnn_kernel (`Tuple[int]` or `List[int]`, *optional*, defaults to `(5, 3, 3, 1, 1)`):
            A tuple of integers defining the kernel size of each 1D convolutional layer in the *TDNN* module of the
            *XVector* model. The length of *tdnn_kernel* has to match the length of *tdnn_dim*.
        tdnn_dilation (`Tuple[int]` or `List[int]`, *optional*, defaults to `(1, 2, 3, 1, 1)`):
            A tuple of integers defining the dilation factor of each 1D convolutional layer in *TDNN* module of the
            *XVector* model. The length of *tdnn_dilation* has to match the length of *tdnn_dim*.
        xvector_output_dim (`int`, *optional*, defaults to 512):
            Dimensionality of the *XVector* embedding vectors.
        add_adapter (`bool`, *optional*, defaults to `False`):
            Whether a convolutional network should be stacked on top of the Wav2Vec2 Encoder. Can be very useful for
            warm-starting Wav2Vec2 for SpeechEncoderDecoder models.
        adapter_kernel_size (`int`, *optional*, defaults to 3):
            Kernel size of the convolutional layers in the adapter network. Only relevant if `add_adapter is True`.
        adapter_stride (`int`, *optional*, defaults to 2):
            Stride of the convolutional layers in the adapter network. Only relevant if `add_adapter is True`.
        num_adapter_layers (`int`, *optional*, defaults to 3):
            Number of convolutional layers that should be used in the adapter network. Only relevant if `add_adapter is
            True`.
        output_hidden_size (`int`, *optional*):
            Dimensionality of the encoder output layer. If not defined, this defaults to *hidden-size*. Only relevant
            if `add_adapter is True`.

    Example:
        ```python
        >>> from transformers import WavLMConfig, WavLMModel
        ...
        >>> # Initializing a WavLM facebook/wavlm-base-960h style configuration
        >>> configuration = WavLMConfig()
        ...
        >>> # Initializing a model (with random weights) from the facebook/wavlm-base-960h style configuration
        >>> model = WavLMModel(configuration)
        ...
        >>> # Accessing the model configuration
        >>> configuration = model.config
        ```
    """

    model_type = "wavlm"

    def __init__(
        self,
        vocab_size=32,
        hidden_size=768,
        num_hidden_layers=12,
        num_attention_heads=12,
        intermediate_size=3072,
        hidden_act="gelu",
        hidden_dropout=0.1,
        activation_dropout=0.1,
        attention_dropout=0.1,
        feat_proj_dropout=0.0,
        final_dropout=0.1,
        layerdrop=0.1,
        initializer_range=0.02,
        layer_norm_eps=1e-5,
        feat_extract_norm="group",
        feat_extract_activation="gelu",
        conv_dim=(512, 512, 512, 512, 512, 512, 512),
        conv_stride=(5, 2, 2, 2, 2, 2, 2),
        conv_kernel=(10, 3, 3, 3, 3, 2, 2),
        conv_bias=False,
        num_conv_pos_embeddings=128,
        num_conv_pos_embedding_groups=16,
        num_buckets=320,
        max_bucket_distance=800,
        do_stable_layer_norm=False,
        apply_spec_augment=True,
        mask_time_prob=0.05,
        mask_time_length=10,
        mask_time_min_masks=2,
        mask_feature_prob=0.0,
        mask_feature_length=10,
        num_codevectors_per_group=320,
        num_codevector_groups=2,
        contrastive_logits_temperature=0.1,
        num_negatives=100,
        codevector_dim=256,
        proj_codevector_dim=256,
        diversity_loss_weight=0.1,
        ctc_loss_reduction="mean",
        ctc_zero_infinity=False,
        use_weighted_layer_sum=False,
        classifier_proj_size=256,
        tdnn_dim=(512, 512, 512, 512, 1500),
        tdnn_kernel=(5, 3, 3, 1, 1),
        tdnn_dilation=(1, 2, 3, 1, 1),
        xvector_output_dim=512,
        num_ctc_classes=80,
        pad_token_id=0,
        bos_token_id=1,
        eos_token_id=2,
        add_adapter=False,
        adapter_kernel_size=3,
        adapter_stride=2,
        num_adapter_layers=3,
        output_hidden_size=None,
        **kwargs,
    ):
        super().__init__(**kwargs, pad_token_id=pad_token_id, bos_token_id=bos_token_id, eos_token_id=eos_token_id)
        self.hidden_size = hidden_size
        self.feat_extract_norm = feat_extract_norm
        self.feat_extract_activation = feat_extract_activation
        self.conv_dim = list(conv_dim)
        self.conv_stride = list(conv_stride)
        self.conv_kernel = list(conv_kernel)
        self.conv_bias = conv_bias
        self.num_buckets = num_buckets
        self.max_bucket_distance = max_bucket_distance
        self.num_conv_pos_embeddings = num_conv_pos_embeddings
        self.num_conv_pos_embedding_groups = num_conv_pos_embedding_groups
        self.num_feat_extract_layers = len(self.conv_dim)
        self.num_hidden_layers = num_hidden_layers
        self.intermediate_size = intermediate_size
        self.hidden_act = hidden_act
        self.num_attention_heads = num_attention_heads
        self.hidden_dropout = hidden_dropout
        self.attention_dropout = attention_dropout
        self.activation_dropout = activation_dropout
        self.feat_proj_dropout = feat_proj_dropout
        self.final_dropout = final_dropout
        self.layerdrop = layerdrop
        self.layer_norm_eps = layer_norm_eps
        self.initializer_range = initializer_range
        self.num_ctc_classes = num_ctc_classes
        self.vocab_size = vocab_size
        self.do_stable_layer_norm = do_stable_layer_norm
        self.use_weighted_layer_sum = use_weighted_layer_sum
        self.classifier_proj_size = classifier_proj_size

        if (
            (len(self.conv_stride) != self.num_feat_extract_layers)
            or (len(self.conv_kernel) != self.num_feat_extract_layers)
            or (len(self.conv_dim) != self.num_feat_extract_layers)
        ):
            raise ValueError(
                "Configuration for convolutional layers is incorrect. It is required that `len(config.conv_dim)` =="
                " `len(config.conv_stride)` == `len(config.conv_kernel)`, but is `len(config.conv_dim) ="
                f" {len(self.conv_dim)}`, `len(config.conv_stride) = {len(self.conv_stride)}`,"
                f" `len(config.conv_kernel) = {len(self.conv_kernel)}`."
            )

        # fine-tuning config parameters for SpecAugment: https://arxiv.org/abs/1904.08779
        self.apply_spec_augment = apply_spec_augment
        self.mask_time_prob = mask_time_prob
        self.mask_time_length = mask_time_length
        self.mask_time_min_masks = mask_time_min_masks
        self.mask_feature_prob = mask_feature_prob
        self.mask_feature_length = mask_feature_length

        # parameters for pretraining with codevector quantized representations
        self.num_codevectors_per_group = num_codevectors_per_group
        self.num_codevector_groups = num_codevector_groups
        self.contrastive_logits_temperature = contrastive_logits_temperature
        self.num_negatives = num_negatives
        self.codevector_dim = codevector_dim
        self.proj_codevector_dim = proj_codevector_dim
        self.diversity_loss_weight = diversity_loss_weight

        # ctc loss
        self.ctc_loss_reduction = ctc_loss_reduction
        self.ctc_zero_infinity = ctc_zero_infinity

        # adapter
        self.add_adapter = add_adapter
        self.adapter_kernel_size = adapter_kernel_size
        self.adapter_stride = adapter_stride
        self.num_adapter_layers = num_adapter_layers
        self.output_hidden_size = output_hidden_size or hidden_size

        # SequenceClassification-specific parameter. Feel free to ignore for other classes.
        self.classifier_proj_size = classifier_proj_size

        # XVector-specific parameters. Feel free to ignore for other classes.
        self.tdnn_dim = list(tdnn_dim)
        self.tdnn_kernel = list(tdnn_kernel)
        self.tdnn_dilation = list(tdnn_dilation)
        self.xvector_output_dim = xvector_output_dim

    @property
    def inputs_to_logits_ratio(self):
        return functools.reduce(operator.mul, self.conv_stride, 1)

mindnlp.transformers.models.wavlm.modeling_wavlm

PyTorch WavLM model.

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMAttention

Bases: Module

Multi-headed attention from 'Attention Is All You Need' paper

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

class WavLMAttention(nn.Module):
    """Multi-headed attention from 'Attention Is All You Need' paper"""

    def __init__(
        self,
        embed_dim: int,
        num_heads: int,
        dropout: float = 0.0,
        num_buckets: int = 320,
        max_distance: int = 800,
        has_relative_position_bias: bool = True,
    ):
        super().__init__()
        self.embed_dim = embed_dim
        self.num_heads = num_heads
        self.dropout = dropout
        self.head_dim = embed_dim // num_heads

        if (self.head_dim * num_heads) != self.embed_dim:
            raise ValueError(
                f"embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim}"
                f" and `num_heads`: {num_heads})."
            )
        self.scaling = self.head_dim**-0.5

        self.k_proj = nn.Linear(embed_dim, embed_dim)
        self.v_proj = nn.Linear(embed_dim, embed_dim)
        self.q_proj = nn.Linear(embed_dim, embed_dim)
        self.out_proj = nn.Linear(embed_dim, embed_dim)

        self.num_buckets = num_buckets
        self.max_distance = max_distance

        self.gru_rel_pos_const = mindspore.Parameter(ops.ones(1, self.num_heads, 1, 1))
        self.gru_rel_pos_linear = nn.Linear(self.head_dim, 8)

        if has_relative_position_bias:
            self.rel_attn_embed = nn.Embedding(self.num_buckets, self.num_heads)

    def forward(
        self,
        hidden_states: mindspore.Tensor,
        attention_mask: Optional[mindspore.Tensor] = None,
        position_bias: Optional[mindspore.Tensor] = None,
        output_attentions: bool = False,
        index=0,
    ) -> Tuple[mindspore.Tensor, Optional[mindspore.Tensor], Optional[Tuple[mindspore.Tensor]]]:
        """Attention layer with relative attention"""
        bsz, tgt_len, _ = hidden_states.shape

        # first pass of attention layer creates position bias
        if position_bias is None:
            position_bias = self.compute_bias(tgt_len, tgt_len)
            position_bias = (
                position_bias.unsqueeze(0).repeat(bsz, 1, 1, 1).view(bsz * self.num_heads, tgt_len, tgt_len)
            )

        # Compute relative position bias:
        # 1) get reshape hidden_states
        gated_hidden_states = hidden_states.view(hidden_states.shape[:-1] + (self.num_heads, -1))
        gated_hidden_states = gated_hidden_states.permute(0, 2, 1, 3)

        # 2) project hidden states
        relative_position_proj = self.gru_rel_pos_linear(gated_hidden_states)
        relative_position_proj = relative_position_proj.view(gated_hidden_states.shape[:-1] + (2, 4)).sum(-1)

        # 3) compute gate for position bias from projected hidden states
        gate_a, gate_b = ops.sigmoid(relative_position_proj).chunk(2, axis=-1)
        gate_output = gate_a * (gate_b * self.gru_rel_pos_const - 1.0) + 2.0

        # 4) apply gate to position bias to compute gated position_bias
        gated_position_bias = gate_output.view(bsz * self.num_heads, -1, 1) * position_bias
        gated_position_bias = gated_position_bias.view((-1, tgt_len, tgt_len))

        attn_output, attn_weights = self.torch_multi_head_self_attention(
            hidden_states, attention_mask, gated_position_bias, output_attentions
        )


        return attn_output, attn_weights, position_bias

    def torch_multi_head_self_attention(
        self,
        hidden_states: mindspore.Tensor,
        attention_mask: Union[mindspore.Tensor, mindspore.Tensor],
        gated_position_bias: mindspore.Tensor,
        output_attentions: bool,
    ) -> (mindspore.Tensor, mindspore.Tensor):
        """simple wrapper around torch's multi_head_attention_forward function"""
        # self-attention assumes q = k = v
        query = key = value = hidden_states.swapaxes(0, 1)
        key_padding_mask = attention_mask.ne(1) if attention_mask is not None else None

        # disable bias and add_zero_attn
        bias_k = bias_v = None
        add_zero_attn = False


        def multi_head_attention_forward(
                query: mindspore.Tensor,
                key: mindspore.Tensor,
                value: mindspore.Tensor,
                embed_dim_to_check: int,
                num_heads: int,
                in_proj_weight: Optional[mindspore.Tensor],
                in_proj_bias: Optional[mindspore.Tensor],
                bias_k: Optional[mindspore.Tensor],
                bias_v: Optional[mindspore.Tensor],
                add_zero_attn: bool,
                dropout_p: float,
                out_proj_weight: mindspore.Tensor,
                out_proj_bias: Optional[mindspore.Tensor],
                training: bool = True,
                key_padding_mask: Optional[mindspore.Tensor] = None,
                need_weights: bool = True,
                attn_mask: Optional[mindspore.Tensor] = None,
                use_separate_proj_weight: bool = False,
                q_proj_weight: Optional[mindspore.Tensor] = None,
                k_proj_weight: Optional[mindspore.Tensor] = None,
                v_proj_weight: Optional[mindspore.Tensor] = None,
                static_k: Optional[mindspore.Tensor] = None,
                static_v: Optional[mindspore.Tensor] = None,
                average_attn_weights: bool = True,
                is_causal: bool = False,
        ) -> Tuple[mindspore.Tensor, Optional[mindspore.Tensor]]:
            # tens_ops = (query, key, value, in_proj_weight, in_proj_bias, bias_k, bias_v, out_proj_weight, out_proj_bias)
            # if has_torch_function(tens_ops):
            #     return handle_torch_function(
            #         multi_head_attention_forward,
            #         tens_ops,
            #         query,
            #         key,
            #         value,
            #         embed_dim_to_check,
            #         num_heads,
            #         in_proj_weight,
            #         in_proj_bias,
            #         bias_k,
            #         bias_v,
            #         add_zero_attn,
            #         dropout_p,
            #         out_proj_weight,
            #         out_proj_bias,
            #         training=training,
            #         key_padding_mask=key_padding_mask,
            #         need_weights=need_weights,
            #         attn_mask=attn_mask,
            #         is_causal=is_causal,
            #         use_separate_proj_weight=use_separate_proj_weight,
            #         q_proj_weight=q_proj_weight,
            #         k_proj_weight=k_proj_weight,
            #         v_proj_weight=v_proj_weight,
            #         static_k=static_k,
            #         static_v=static_v,
            #         average_attn_weights=average_attn_weights,
            #     )

            is_batched = _mha_shape_check(query, key, value, key_padding_mask, attn_mask, num_heads)

            # For unbatched input, we unsqueeze at the expected batch-dim to pretend that the input
            # is batched, run the computation and before returning squeeze the
            # batch dimension so that the output doesn't carry this temporary batch dimension.
            if not is_batched:
                # unsqueeze if the input is unbatched
                query = query.unsqueeze(1)
                key = key.unsqueeze(1)
                value = value.unsqueeze(1)
                if key_padding_mask is not None:
                    key_padding_mask = key_padding_mask.unsqueeze(0)

            # set up shape vars
            tgt_len, bsz, embed_dim = query.shape
            src_len, _, _ = key.shape

            key_padding_mask = _canonical_mask(
                mask=key_padding_mask,
                mask_name="key_padding_mask",
                other_type=_none_or_dtype(attn_mask),
                other_name="attn_mask",
                target_type=query.dtype
            )

            if is_causal and attn_mask is None:
                raise RuntimeError(
                    "Need attn_mask if specifying the is_causal hint. "
                    "You may use the Transformer module method "
                    "`generate_square_subsequent_mask` to create this mask."
                )

            if is_causal and key_padding_mask is None and not need_weights:
                # when we have a kpm or need weights, we need attn_mask
                # Otherwise, we use the is_causal hint go as is_causal
                # indicator to SDPA.
                attn_mask = None
            else:
                attn_mask = _canonical_mask(
                    mask=attn_mask,
                    mask_name="attn_mask",
                    other_type=None,
                    other_name="",
                    target_type=query.dtype,
                    check_other=False,
                )

                if key_padding_mask is not None:
                    # We have the attn_mask, and use that to merge kpm into it.
                    # Turn off use of is_causal hint, as the merged mask is no
                    # longer causal.
                    is_causal = False

            assert embed_dim == embed_dim_to_check, \
                f"was expecting embedding dimension of {embed_dim_to_check}, but got {embed_dim}"
            if isinstance(embed_dim, mindspore.Tensor):
                # embed_dim can be a tensor when JIT tracing
                head_dim = embed_dim.div(num_heads, rounding_mode='trunc')
            else:
                head_dim = embed_dim // num_heads
            assert head_dim * num_heads == embed_dim, f"embed_dim {embed_dim} not divisible by num_heads {num_heads}"
            if use_separate_proj_weight:
                # allow MHA to have different embedding dimensions when separate projection weights are used
                assert key.shape[:2] == value.shape[:2], \
                    f"key's sequence and batch dims {key.shape[:2]} do not match value's {value.shape[:2]}"
            else:
                assert key.shape == value.shape, f"key shape {key.shape} does not match value shape {value.shape}"

            #
            # compute in-projection
            #
            if not use_separate_proj_weight:
                assert in_proj_weight is not None, "use_separate_proj_weight is False but in_proj_weight is None"
                q, k, v = _in_projection_packed(query, key, value, in_proj_weight, in_proj_bias)
            else:
                assert q_proj_weight is not None, "use_separate_proj_weight is True but q_proj_weight is None"
                assert k_proj_weight is not None, "use_separate_proj_weight is True but k_proj_weight is None"
                assert v_proj_weight is not None, "use_separate_proj_weight is True but v_proj_weight is None"
                if in_proj_bias is None:
                    b_q = b_k = b_v = None
                else:
                    b_q, b_k, b_v = in_proj_bias.chunk(3)
                q, k, v = _in_projection(query, key, value, q_proj_weight, k_proj_weight, v_proj_weight, b_q, b_k, b_v)

            # prep attention mask

            if attn_mask is not None:
                # ensure attn_mask's dim is 3
                if attn_mask.ndim == 2:
                    correct_2d_size = (tgt_len, src_len)
                    if attn_mask.shape != correct_2d_size:
                        raise RuntimeError(
                            f"The shape of the 2D attn_mask is {attn_mask.shape}, but should be {correct_2d_size}.")
                    attn_mask = attn_mask.unsqueeze(0)
                elif attn_mask.ndim == 3:
                    correct_3d_size = (bsz * num_heads, tgt_len, src_len)
                    if attn_mask.shape != correct_3d_size:
                        raise RuntimeError(
                            f"The shape of the 3D attn_mask is {attn_mask.shape}, but should be {correct_3d_size}.")
                else:
                    raise RuntimeError(f"attn_mask's dimension {attn_mask.ndim} is not supported")

            # add bias along batch dimension (currently second)
            if bias_k is not None and bias_v is not None:
                assert static_k is None, "bias cannot be added to static key."
                assert static_v is None, "bias cannot be added to static value."
                k = ops.cat([k, bias_k.repeat(1, bsz, 1)])
                v = ops.cat([v, bias_v.repeat(1, bsz, 1)])
                if attn_mask is not None:
                    attn_mask = ops.pad(attn_mask, (0, 1))
                if key_padding_mask is not None:
                    key_padding_mask = ops.pad(key_padding_mask, (0, 1))
            else:
                assert bias_k is None
                assert bias_v is None

            #
            # reshape q, k, v for multihead attention and make em batch first
            #
            q = q.view(tgt_len, bsz * num_heads, head_dim).swapaxes(0, 1)
            if static_k is None:
                k = k.view(k.shape[0], bsz * num_heads, head_dim).swapaxes(0, 1)
            else:
                # TODO finish disentangling control flow so we don't do in-projections when statics are passed
                assert static_k.shape[0] == bsz * num_heads, \
                    f"expecting static_k.size(0) of {bsz * num_heads}, but got {static_k.shape[0]}"
                assert static_k.shape[2] == head_dim, \
                    f"expecting static_k.size(2) of {head_dim}, but got {static_k.shape[2]}"
                k = static_k
            if static_v is None:
                v = v.view(v.shape[0], bsz * num_heads, head_dim).swapaxes(0, 1)
            else:
                # TODO finish disentangling control flow so we don't do in-projections when statics are passed
                assert static_v.shape[0] == bsz * num_heads, \
                    f"expecting static_v.size(0) of {bsz * num_heads}, but got {static_v.shape[0]}"
                assert static_v.shape[2] == head_dim, \
                    f"expecting static_v.size(2) of {head_dim}, but got {static_v.shape[2]}"
                v = static_v

            # add zero attention along batch dimension (now first)
            if add_zero_attn:
                zero_attn_shape = (bsz * num_heads, 1, head_dim)
                k = ops.cat([k, ops.zeros(zero_attn_shape, dtype=k.dtype)], axis=1)
                v = ops.cat([v, ops.zeros(zero_attn_shape, dtype=v.dtype)], axis=1)
                if attn_mask is not None:
                    attn_mask = ops.pad(attn_mask, (0, 1))
                if key_padding_mask is not None:
                    key_padding_mask = ops.pad(key_padding_mask, (0, 1))

            # update source sequence length after adjustments
            src_len = k.shape[1]

            # merge key padding and attention masks
            if key_padding_mask is not None:
                assert key_padding_mask.shape == (bsz, src_len), \
                    f"expecting key_padding_mask shape of {(bsz, src_len)}, but got {key_padding_mask.shape}"
                key_padding_mask = key_padding_mask.view(bsz, 1, 1, src_len). \
                    expand(-1, num_heads, -1, -1).reshape(bsz * num_heads, 1, src_len)
                if attn_mask is None:
                    attn_mask = key_padding_mask
                else:
                    attn_mask = attn_mask + key_padding_mask

            # adjust dropout probability
            if not training:
                dropout_p = 0.0

            #
            # (deep breath) calculate attention and out projection
            #

            if need_weights:
                B, Nt, E = q.shape
                q_scaled = q * math.sqrt(1.0 / float(E))

                assert not (is_causal and attn_mask is None), "FIXME: is_causal not implemented for need_weights"

                if attn_mask is not None:
                    attn_output_weights = ops.baddbmm(attn_mask, q_scaled, k.swapaxes(-2, -1))
                else:
                    attn_output_weights = ops.bmm(q_scaled, k.swapaxes(-2, -1))
                attn_output_weights = ops.softmax(attn_output_weights, axis=-1)
                if dropout_p > 0.0:
                    attn_output_weights = ops.dropout(attn_output_weights, p=dropout_p)

                attn_output = ops.bmm(attn_output_weights, v)

                attn_output = attn_output.swapaxes(0, 1).view(tgt_len * bsz, embed_dim)
                attn_output = linear(attn_output, out_proj_weight, out_proj_bias)
                attn_output = attn_output.view(tgt_len, bsz, attn_output.shape[1])

                # optionally average attention weights over heads
                attn_output_weights = attn_output_weights.view(bsz, num_heads, tgt_len, src_len)
                if average_attn_weights:
                    attn_output_weights = attn_output_weights.mean(axis=1)

                if not is_batched:
                    # squeeze the output if input was unbatched
                    attn_output = attn_output.squeeze(1)
                    attn_output_weights = attn_output_weights.squeeze(0)
                return attn_output, attn_output_weights
            else:
                # attn_mask can be either (L,S) or (N*num_heads, L, S)
                # if attn_mask's shape is (1, L, S) we need to unsqueeze to (1, 1, L, S)
                # in order to match the input for SDPA of (N, num_heads, L, S)
                if attn_mask is not None:
                    if attn_mask.shape[0] == 1 and attn_mask.ndim == 3:
                        attn_mask = attn_mask.unsqueeze(0)
                    else:
                        attn_mask = attn_mask.view(bsz, num_heads, -1, src_len)

                q = q.view(bsz, num_heads, tgt_len, head_dim)
                k = k.view(bsz, num_heads, src_len, head_dim)
                v = v.view(bsz, num_heads, src_len, head_dim)

                attn_output = _scaled_dot_product_attention(q, k, v, attn_mask, dropout_p, is_causal, training)
                attn_output = attn_output.permute(2, 0, 1, 3).view(bsz * tgt_len, embed_dim)

                attn_output = linear(attn_output, out_proj_weight, out_proj_bias)
                attn_output = attn_output.view(tgt_len, bsz, attn_output.shape[1])
                if not is_batched:
                    # squeeze the output if input was unbatched
                    attn_output = attn_output.squeeze(1)
                return attn_output, None

        # PyTorch 1.3.0 has F.multi_head_attention_forward defined
        # so no problem with backwards compatibility
        attn_output, attn_weights = multi_head_attention_forward(
            query,
            key,
            value,
            self.embed_dim,
            self.num_heads,
            ops.zeros([0]),
            ops.cat((self.q_proj.bias, self.k_proj.bias, self.v_proj.bias)),
            bias_k,
            bias_v,
            add_zero_attn,
            self.dropout,
            self.out_proj.weight,
            self.out_proj.bias,
            self.training,
            key_padding_mask,
            # attention_mask,
            attn_mask=gated_position_bias,
            use_separate_proj_weight=True,
            q_proj_weight=self.q_proj.weight,
            k_proj_weight=self.k_proj.weight,
            v_proj_weight=self.v_proj.weight,
        )

        # [Seq_Len, Batch Size, ...] -> [Batch Size, Seq_Len, ...]
        attn_output = attn_output.swapaxes(0, 1)

        if attn_weights is not None:
            # IMPORTANT: Attention weights are averaged weights
            # here which should not be the case. This is an open issue
            # on PyTorch: https://github.com/pytorch/pytorch/issues/32590
            attn_weights = attn_weights[:, None].broadcast_to(
                attn_weights.shape[:1] + (self.num_heads,) + attn_weights.shape[1:]
            )

        return attn_output, attn_weights

    def compute_bias(self, query_length: int, key_length: int) -> mindspore.Tensor:
        context_position = ops.arange(query_length, dtype=mindspore.int64)[:, None]
        memory_position = ops.arange(key_length, dtype=mindspore.int64)[None, :]
        relative_position = memory_position - context_position
        relative_position_bucket = self._relative_positions_bucket(relative_position)
        values = self.rel_attn_embed(relative_position_bucket)
        values = values.permute([2, 0, 1])
        return values

    def _relative_positions_bucket(self, relative_positions: mindspore.Tensor) -> mindspore.Tensor:
        num_buckets = self.num_buckets // 2

        relative_buckets = (relative_positions > 0).astype(mindspore.int64) * num_buckets
        relative_positions = ops.abs(relative_positions)

        max_exact = num_buckets // 2
        is_small = relative_positions < max_exact

        relative_positions_if_large = ops.log(relative_positions.float() / max_exact)
        relative_positions_if_large = relative_positions_if_large / math.log(self.max_distance / max_exact)
        relative_positions_if_large = relative_positions_if_large * (num_buckets - max_exact)
        relative_position_if_large = (max_exact + relative_positions_if_large).astype(mindspore.int64)
        # relative_position_if_large = ops.min(
        #     relative_position_if_large, ops.full_like(relative_position_if_large, num_buckets - 1)
        # )
        relative_position_if_large = ops.where(
            relative_position_if_large < ops.full_like(relative_position_if_large, num_buckets - 1),
            relative_position_if_large,
            ops.full_like(relative_position_if_large, num_buckets - 1)
        )


        relative_buckets += ops.where(is_small, relative_positions, relative_position_if_large)
        return relative_buckets

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMAttention.forward(hidden_states, attention_mask=None, position_bias=None, output_attentions=False, index=0)

Attention layer with relative attention

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

def forward(
    self,
    hidden_states: mindspore.Tensor,
    attention_mask: Optional[mindspore.Tensor] = None,
    position_bias: Optional[mindspore.Tensor] = None,
    output_attentions: bool = False,
    index=0,
) -> Tuple[mindspore.Tensor, Optional[mindspore.Tensor], Optional[Tuple[mindspore.Tensor]]]:
    """Attention layer with relative attention"""
    bsz, tgt_len, _ = hidden_states.shape

    # first pass of attention layer creates position bias
    if position_bias is None:
        position_bias = self.compute_bias(tgt_len, tgt_len)
        position_bias = (
            position_bias.unsqueeze(0).repeat(bsz, 1, 1, 1).view(bsz * self.num_heads, tgt_len, tgt_len)
        )

    # Compute relative position bias:
    # 1) get reshape hidden_states
    gated_hidden_states = hidden_states.view(hidden_states.shape[:-1] + (self.num_heads, -1))
    gated_hidden_states = gated_hidden_states.permute(0, 2, 1, 3)

    # 2) project hidden states
    relative_position_proj = self.gru_rel_pos_linear(gated_hidden_states)
    relative_position_proj = relative_position_proj.view(gated_hidden_states.shape[:-1] + (2, 4)).sum(-1)

    # 3) compute gate for position bias from projected hidden states
    gate_a, gate_b = ops.sigmoid(relative_position_proj).chunk(2, axis=-1)
    gate_output = gate_a * (gate_b * self.gru_rel_pos_const - 1.0) + 2.0

    # 4) apply gate to position bias to compute gated position_bias
    gated_position_bias = gate_output.view(bsz * self.num_heads, -1, 1) * position_bias
    gated_position_bias = gated_position_bias.view((-1, tgt_len, tgt_len))

    attn_output, attn_weights = self.torch_multi_head_self_attention(
        hidden_states, attention_mask, gated_position_bias, output_attentions
    )


    return attn_output, attn_weights, position_bias

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMAttention.torch_multi_head_self_attention(hidden_states, attention_mask, gated_position_bias, output_attentions)

simple wrapper around torch's multi_head_attention_forward function

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

def torch_multi_head_self_attention(
    self,
    hidden_states: mindspore.Tensor,
    attention_mask: Union[mindspore.Tensor, mindspore.Tensor],
    gated_position_bias: mindspore.Tensor,
    output_attentions: bool,
) -> (mindspore.Tensor, mindspore.Tensor):
    """simple wrapper around torch's multi_head_attention_forward function"""
    # self-attention assumes q = k = v
    query = key = value = hidden_states.swapaxes(0, 1)
    key_padding_mask = attention_mask.ne(1) if attention_mask is not None else None

    # disable bias and add_zero_attn
    bias_k = bias_v = None
    add_zero_attn = False


    def multi_head_attention_forward(
            query: mindspore.Tensor,
            key: mindspore.Tensor,
            value: mindspore.Tensor,
            embed_dim_to_check: int,
            num_heads: int,
            in_proj_weight: Optional[mindspore.Tensor],
            in_proj_bias: Optional[mindspore.Tensor],
            bias_k: Optional[mindspore.Tensor],
            bias_v: Optional[mindspore.Tensor],
            add_zero_attn: bool,
            dropout_p: float,
            out_proj_weight: mindspore.Tensor,
            out_proj_bias: Optional[mindspore.Tensor],
            training: bool = True,
            key_padding_mask: Optional[mindspore.Tensor] = None,
            need_weights: bool = True,
            attn_mask: Optional[mindspore.Tensor] = None,
            use_separate_proj_weight: bool = False,
            q_proj_weight: Optional[mindspore.Tensor] = None,
            k_proj_weight: Optional[mindspore.Tensor] = None,
            v_proj_weight: Optional[mindspore.Tensor] = None,
            static_k: Optional[mindspore.Tensor] = None,
            static_v: Optional[mindspore.Tensor] = None,
            average_attn_weights: bool = True,
            is_causal: bool = False,
    ) -> Tuple[mindspore.Tensor, Optional[mindspore.Tensor]]:
        # tens_ops = (query, key, value, in_proj_weight, in_proj_bias, bias_k, bias_v, out_proj_weight, out_proj_bias)
        # if has_torch_function(tens_ops):
        #     return handle_torch_function(
        #         multi_head_attention_forward,
        #         tens_ops,
        #         query,
        #         key,
        #         value,
        #         embed_dim_to_check,
        #         num_heads,
        #         in_proj_weight,
        #         in_proj_bias,
        #         bias_k,
        #         bias_v,
        #         add_zero_attn,
        #         dropout_p,
        #         out_proj_weight,
        #         out_proj_bias,
        #         training=training,
        #         key_padding_mask=key_padding_mask,
        #         need_weights=need_weights,
        #         attn_mask=attn_mask,
        #         is_causal=is_causal,
        #         use_separate_proj_weight=use_separate_proj_weight,
        #         q_proj_weight=q_proj_weight,
        #         k_proj_weight=k_proj_weight,
        #         v_proj_weight=v_proj_weight,
        #         static_k=static_k,
        #         static_v=static_v,
        #         average_attn_weights=average_attn_weights,
        #     )

        is_batched = _mha_shape_check(query, key, value, key_padding_mask, attn_mask, num_heads)

        # For unbatched input, we unsqueeze at the expected batch-dim to pretend that the input
        # is batched, run the computation and before returning squeeze the
        # batch dimension so that the output doesn't carry this temporary batch dimension.
        if not is_batched:
            # unsqueeze if the input is unbatched
            query = query.unsqueeze(1)
            key = key.unsqueeze(1)
            value = value.unsqueeze(1)
            if key_padding_mask is not None:
                key_padding_mask = key_padding_mask.unsqueeze(0)

        # set up shape vars
        tgt_len, bsz, embed_dim = query.shape
        src_len, _, _ = key.shape

        key_padding_mask = _canonical_mask(
            mask=key_padding_mask,
            mask_name="key_padding_mask",
            other_type=_none_or_dtype(attn_mask),
            other_name="attn_mask",
            target_type=query.dtype
        )

        if is_causal and attn_mask is None:
            raise RuntimeError(
                "Need attn_mask if specifying the is_causal hint. "
                "You may use the Transformer module method "
                "`generate_square_subsequent_mask` to create this mask."
            )

        if is_causal and key_padding_mask is None and not need_weights:
            # when we have a kpm or need weights, we need attn_mask
            # Otherwise, we use the is_causal hint go as is_causal
            # indicator to SDPA.
            attn_mask = None
        else:
            attn_mask = _canonical_mask(
                mask=attn_mask,
                mask_name="attn_mask",
                other_type=None,
                other_name="",
                target_type=query.dtype,
                check_other=False,
            )

            if key_padding_mask is not None:
                # We have the attn_mask, and use that to merge kpm into it.
                # Turn off use of is_causal hint, as the merged mask is no
                # longer causal.
                is_causal = False

        assert embed_dim == embed_dim_to_check, \
            f"was expecting embedding dimension of {embed_dim_to_check}, but got {embed_dim}"
        if isinstance(embed_dim, mindspore.Tensor):
            # embed_dim can be a tensor when JIT tracing
            head_dim = embed_dim.div(num_heads, rounding_mode='trunc')
        else:
            head_dim = embed_dim // num_heads
        assert head_dim * num_heads == embed_dim, f"embed_dim {embed_dim} not divisible by num_heads {num_heads}"
        if use_separate_proj_weight:
            # allow MHA to have different embedding dimensions when separate projection weights are used
            assert key.shape[:2] == value.shape[:2], \
                f"key's sequence and batch dims {key.shape[:2]} do not match value's {value.shape[:2]}"
        else:
            assert key.shape == value.shape, f"key shape {key.shape} does not match value shape {value.shape}"

        #
        # compute in-projection
        #
        if not use_separate_proj_weight:
            assert in_proj_weight is not None, "use_separate_proj_weight is False but in_proj_weight is None"
            q, k, v = _in_projection_packed(query, key, value, in_proj_weight, in_proj_bias)
        else:
            assert q_proj_weight is not None, "use_separate_proj_weight is True but q_proj_weight is None"
            assert k_proj_weight is not None, "use_separate_proj_weight is True but k_proj_weight is None"
            assert v_proj_weight is not None, "use_separate_proj_weight is True but v_proj_weight is None"
            if in_proj_bias is None:
                b_q = b_k = b_v = None
            else:
                b_q, b_k, b_v = in_proj_bias.chunk(3)
            q, k, v = _in_projection(query, key, value, q_proj_weight, k_proj_weight, v_proj_weight, b_q, b_k, b_v)

        # prep attention mask

        if attn_mask is not None:
            # ensure attn_mask's dim is 3
            if attn_mask.ndim == 2:
                correct_2d_size = (tgt_len, src_len)
                if attn_mask.shape != correct_2d_size:
                    raise RuntimeError(
                        f"The shape of the 2D attn_mask is {attn_mask.shape}, but should be {correct_2d_size}.")
                attn_mask = attn_mask.unsqueeze(0)
            elif attn_mask.ndim == 3:
                correct_3d_size = (bsz * num_heads, tgt_len, src_len)
                if attn_mask.shape != correct_3d_size:
                    raise RuntimeError(
                        f"The shape of the 3D attn_mask is {attn_mask.shape}, but should be {correct_3d_size}.")
            else:
                raise RuntimeError(f"attn_mask's dimension {attn_mask.ndim} is not supported")

        # add bias along batch dimension (currently second)
        if bias_k is not None and bias_v is not None:
            assert static_k is None, "bias cannot be added to static key."
            assert static_v is None, "bias cannot be added to static value."
            k = ops.cat([k, bias_k.repeat(1, bsz, 1)])
            v = ops.cat([v, bias_v.repeat(1, bsz, 1)])
            if attn_mask is not None:
                attn_mask = ops.pad(attn_mask, (0, 1))
            if key_padding_mask is not None:
                key_padding_mask = ops.pad(key_padding_mask, (0, 1))
        else:
            assert bias_k is None
            assert bias_v is None

        #
        # reshape q, k, v for multihead attention and make em batch first
        #
        q = q.view(tgt_len, bsz * num_heads, head_dim).swapaxes(0, 1)
        if static_k is None:
            k = k.view(k.shape[0], bsz * num_heads, head_dim).swapaxes(0, 1)
        else:
            # TODO finish disentangling control flow so we don't do in-projections when statics are passed
            assert static_k.shape[0] == bsz * num_heads, \
                f"expecting static_k.size(0) of {bsz * num_heads}, but got {static_k.shape[0]}"
            assert static_k.shape[2] == head_dim, \
                f"expecting static_k.size(2) of {head_dim}, but got {static_k.shape[2]}"
            k = static_k
        if static_v is None:
            v = v.view(v.shape[0], bsz * num_heads, head_dim).swapaxes(0, 1)
        else:
            # TODO finish disentangling control flow so we don't do in-projections when statics are passed
            assert static_v.shape[0] == bsz * num_heads, \
                f"expecting static_v.size(0) of {bsz * num_heads}, but got {static_v.shape[0]}"
            assert static_v.shape[2] == head_dim, \
                f"expecting static_v.size(2) of {head_dim}, but got {static_v.shape[2]}"
            v = static_v

        # add zero attention along batch dimension (now first)
        if add_zero_attn:
            zero_attn_shape = (bsz * num_heads, 1, head_dim)
            k = ops.cat([k, ops.zeros(zero_attn_shape, dtype=k.dtype)], axis=1)
            v = ops.cat([v, ops.zeros(zero_attn_shape, dtype=v.dtype)], axis=1)
            if attn_mask is not None:
                attn_mask = ops.pad(attn_mask, (0, 1))
            if key_padding_mask is not None:
                key_padding_mask = ops.pad(key_padding_mask, (0, 1))

        # update source sequence length after adjustments
        src_len = k.shape[1]

        # merge key padding and attention masks
        if key_padding_mask is not None:
            assert key_padding_mask.shape == (bsz, src_len), \
                f"expecting key_padding_mask shape of {(bsz, src_len)}, but got {key_padding_mask.shape}"
            key_padding_mask = key_padding_mask.view(bsz, 1, 1, src_len). \
                expand(-1, num_heads, -1, -1).reshape(bsz * num_heads, 1, src_len)
            if attn_mask is None:
                attn_mask = key_padding_mask
            else:
                attn_mask = attn_mask + key_padding_mask

        # adjust dropout probability
        if not training:
            dropout_p = 0.0

        #
        # (deep breath) calculate attention and out projection
        #

        if need_weights:
            B, Nt, E = q.shape
            q_scaled = q * math.sqrt(1.0 / float(E))

            assert not (is_causal and attn_mask is None), "FIXME: is_causal not implemented for need_weights"

            if attn_mask is not None:
                attn_output_weights = ops.baddbmm(attn_mask, q_scaled, k.swapaxes(-2, -1))
            else:
                attn_output_weights = ops.bmm(q_scaled, k.swapaxes(-2, -1))
            attn_output_weights = ops.softmax(attn_output_weights, axis=-1)
            if dropout_p > 0.0:
                attn_output_weights = ops.dropout(attn_output_weights, p=dropout_p)

            attn_output = ops.bmm(attn_output_weights, v)

            attn_output = attn_output.swapaxes(0, 1).view(tgt_len * bsz, embed_dim)
            attn_output = linear(attn_output, out_proj_weight, out_proj_bias)
            attn_output = attn_output.view(tgt_len, bsz, attn_output.shape[1])

            # optionally average attention weights over heads
            attn_output_weights = attn_output_weights.view(bsz, num_heads, tgt_len, src_len)
            if average_attn_weights:
                attn_output_weights = attn_output_weights.mean(axis=1)

            if not is_batched:
                # squeeze the output if input was unbatched
                attn_output = attn_output.squeeze(1)
                attn_output_weights = attn_output_weights.squeeze(0)
            return attn_output, attn_output_weights
        else:
            # attn_mask can be either (L,S) or (N*num_heads, L, S)
            # if attn_mask's shape is (1, L, S) we need to unsqueeze to (1, 1, L, S)
            # in order to match the input for SDPA of (N, num_heads, L, S)
            if attn_mask is not None:
                if attn_mask.shape[0] == 1 and attn_mask.ndim == 3:
                    attn_mask = attn_mask.unsqueeze(0)
                else:
                    attn_mask = attn_mask.view(bsz, num_heads, -1, src_len)

            q = q.view(bsz, num_heads, tgt_len, head_dim)
            k = k.view(bsz, num_heads, src_len, head_dim)
            v = v.view(bsz, num_heads, src_len, head_dim)

            attn_output = _scaled_dot_product_attention(q, k, v, attn_mask, dropout_p, is_causal, training)
            attn_output = attn_output.permute(2, 0, 1, 3).view(bsz * tgt_len, embed_dim)

            attn_output = linear(attn_output, out_proj_weight, out_proj_bias)
            attn_output = attn_output.view(tgt_len, bsz, attn_output.shape[1])
            if not is_batched:
                # squeeze the output if input was unbatched
                attn_output = attn_output.squeeze(1)
            return attn_output, None

    # PyTorch 1.3.0 has F.multi_head_attention_forward defined
    # so no problem with backwards compatibility
    attn_output, attn_weights = multi_head_attention_forward(
        query,
        key,
        value,
        self.embed_dim,
        self.num_heads,
        ops.zeros([0]),
        ops.cat((self.q_proj.bias, self.k_proj.bias, self.v_proj.bias)),
        bias_k,
        bias_v,
        add_zero_attn,
        self.dropout,
        self.out_proj.weight,
        self.out_proj.bias,
        self.training,
        key_padding_mask,
        # attention_mask,
        attn_mask=gated_position_bias,
        use_separate_proj_weight=True,
        q_proj_weight=self.q_proj.weight,
        k_proj_weight=self.k_proj.weight,
        v_proj_weight=self.v_proj.weight,
    )

    # [Seq_Len, Batch Size, ...] -> [Batch Size, Seq_Len, ...]
    attn_output = attn_output.swapaxes(0, 1)

    if attn_weights is not None:
        # IMPORTANT: Attention weights are averaged weights
        # here which should not be the case. This is an open issue
        # on PyTorch: https://github.com/pytorch/pytorch/issues/32590
        attn_weights = attn_weights[:, None].broadcast_to(
            attn_weights.shape[:1] + (self.num_heads,) + attn_weights.shape[1:]
        )

    return attn_output, attn_weights

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMFeatureEncoder

Bases: Module

Construct the features from raw audio waveform

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

class WavLMFeatureEncoder(nn.Module):
    """Construct the features from raw audio waveform"""

    def __init__(self, config):
        super().__init__()

        if config.feat_extract_norm == "group":
            conv_layers = [WavLMGroupNormConvLayer(config, layer_id=0)] + [
                WavLMNoLayerNormConvLayer(config, layer_id=i + 1) for i in range(config.num_feat_extract_layers - 1)
            ]
        elif config.feat_extract_norm == "layer":
            conv_layers = [WavLMLayerNormConvLayer(config, layer_id=i) for i in range(config.num_feat_extract_layers)]
        else:
            raise ValueError(
                f"`config.feat_extract_norm` is {config.feat_extract_norm}, but has to be one of ['group', 'layer']"
            )
        self.conv_layers = nn.ModuleList(conv_layers)
        self.gradient_checkpointing = False
        self._requires_grad = True

    def _freeze_parameters(self):
        for param in self.get_parameters():
            param.requires_grad = False
        self._requires_grad = False

    def forward(self, input_values):
        hidden_states = input_values[:, None]

        # make sure hidden_states require grad for gradient_checkpointing
        if self._requires_grad and self.training:
            hidden_states.requires_grad = True

        for conv_layer in self.conv_layers:
            if self._requires_grad and self.gradient_checkpointing and self.training:
                hidden_states = self._gradient_checkpointing_func(
                    conv_layer.__call__,
                    hidden_states,
                )
            else:
                hidden_states = conv_layer(hidden_states)

        return hidden_states

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMForAudioFrameClassification

Bases: WavLMPreTrainedModel

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

class WavLMForAudioFrameClassification(WavLMPreTrainedModel):
    def __init__(self, config):
        super().__init__(config)

        if hasattr(config, "add_adapter") and config.add_adapter:
            raise ValueError(
                "Audio frame classification does not support the use of WavLM adapters (config.add_adapter=True)"
            )
        self.wavlm = WavLMModel(config)
        num_layers = config.num_hidden_layers + 1  # transformer layers + input embeddings
        if config.use_weighted_layer_sum:
            self.layer_weights = mindspore.Parameter(ops.ones(num_layers) / num_layers)
        self.classifier = nn.Linear(config.hidden_size, config.num_labels)
        self.num_labels = config.num_labels

        self.init_weights()

    def freeze_feature_extractor(self):
        """
        Calling this function will disable the gradient computation for the feature encoder so that its parameter will
        not be updated during training.
        """
        warnings.warn(
            "The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. "
            "Please use the equivalent `freeze_feature_encoder` method instead.",
            FutureWarning,
        )
        self.freeze_feature_encoder()

    def freeze_feature_encoder(self):
        """
        Calling this function will disable the gradient computation for the feature encoder so that its parameter will
        not be updated during training.
        """
        self.wavlm.feature_extractor._freeze_parameters()

    def freeze_base_model(self):
        """
        Calling this function will disable the gradient computation for the base model so that its parameters will not
        be updated during training. Only the classification head will be updated.
        """
        for param in self.wavlm.get_parameters():
            param.requires_grad = False

    def forward(
        self,
        input_values: Optional[mindspore.Tensor],
        attention_mask: Optional[mindspore.Tensor] = None,
        labels: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, TokenClassifierOutput]:
        r"""
        Args:
            labels (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
                Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
                config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
                `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
        """

        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
        output_hidden_states = True if self.config.use_weighted_layer_sum else output_hidden_states

        outputs = self.wavlm(
            input_values,
            attention_mask=attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        if self.config.use_weighted_layer_sum:
            hidden_states = outputs[_HIDDEN_STATES_START_POSITION]
            hidden_states = ops.stack(hidden_states, axis=1)
            norm_weights = ops.softmax(self.layer_weights, axis=-1)
            hidden_states = (hidden_states * norm_weights.view(-1, 1, 1)).sum(axis=1)
        else:
            hidden_states = outputs[0]

        logits = self.classifier(hidden_states)

        loss = None
        if labels is not None:
            loss = ops.cross_entropy(logits.view(-1, self.num_labels), ops.argmax(labels.view(-1, self.num_labels), dim=1))


        if not return_dict:
            output = (logits,) + outputs[_HIDDEN_STATES_START_POSITION:]
            return output

        return TokenClassifierOutput(
            loss=loss,
            logits=logits,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMForAudioFrameClassification.forward(input_values, attention_mask=None, labels=None, output_attentions=None, output_hidden_states=None, return_dict=None)

PARAMETER	DESCRIPTION
labels	Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If config.num_labels > 1 a classification loss is computed (Cross-Entropy). TYPE: `mindspore.Tensor` of shape `(batch_size,)`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

def forward(
    self,
    input_values: Optional[mindspore.Tensor],
    attention_mask: Optional[mindspore.Tensor] = None,
    labels: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple, TokenClassifierOutput]:
    r"""
    Args:
        labels (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
            Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
            config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
            `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
    """

    return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    output_hidden_states = True if self.config.use_weighted_layer_sum else output_hidden_states

    outputs = self.wavlm(
        input_values,
        attention_mask=attention_mask,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    if self.config.use_weighted_layer_sum:
        hidden_states = outputs[_HIDDEN_STATES_START_POSITION]
        hidden_states = ops.stack(hidden_states, axis=1)
        norm_weights = ops.softmax(self.layer_weights, axis=-1)
        hidden_states = (hidden_states * norm_weights.view(-1, 1, 1)).sum(axis=1)
    else:
        hidden_states = outputs[0]

    logits = self.classifier(hidden_states)

    loss = None
    if labels is not None:
        loss = ops.cross_entropy(logits.view(-1, self.num_labels), ops.argmax(labels.view(-1, self.num_labels), dim=1))


    if not return_dict:
        output = (logits,) + outputs[_HIDDEN_STATES_START_POSITION:]
        return output

    return TokenClassifierOutput(
        loss=loss,
        logits=logits,
        hidden_states=outputs.hidden_states,
        attentions=outputs.attentions,
    )

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMForAudioFrameClassification.freeze_base_model()

Calling this function will disable the gradient computation for the base model so that its parameters will not be updated during training. Only the classification head will be updated.

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

def freeze_base_model(self):
    """
    Calling this function will disable the gradient computation for the base model so that its parameters will not
    be updated during training. Only the classification head will be updated.
    """
    for param in self.wavlm.get_parameters():
        param.requires_grad = False

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMForAudioFrameClassification.freeze_feature_encoder()

Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training.

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

def freeze_feature_encoder(self):
    """
    Calling this function will disable the gradient computation for the feature encoder so that its parameter will
    not be updated during training.
    """
    self.wavlm.feature_extractor._freeze_parameters()

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMForAudioFrameClassification.freeze_feature_extractor()

Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training.

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

def freeze_feature_extractor(self):
    """
    Calling this function will disable the gradient computation for the feature encoder so that its parameter will
    not be updated during training.
    """
    warnings.warn(
        "The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. "
        "Please use the equivalent `freeze_feature_encoder` method instead.",
        FutureWarning,
    )
    self.freeze_feature_encoder()

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMForCTC

Bases: WavLMPreTrainedModel

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

class WavLMForCTC(WavLMPreTrainedModel):
    def __init__(self, config, target_lang: Optional[str] = None):
        super().__init__(config)

        self.wavlm = WavLMModel(config)
        self.dropout = nn.Dropout(p=config.final_dropout)

        self.target_lang = target_lang

        if config.vocab_size is None:
            raise ValueError(
                f"You are trying to instantiate {self.__class__} with a configuration that "
                "does not define the vocabulary size of the language model head. Please "
                "instantiate the model as follows: `WavLMForCTC.from_pretrained(..., vocab_size=vocab_size)`. "
                "or define `vocab_size` of your model's configuration."
            )
        output_hidden_size = (
            config.output_hidden_size if hasattr(config, "add_adapter") and config.add_adapter else config.hidden_size
        )
        self.lm_head = nn.Linear(output_hidden_size, config.vocab_size)

        # Initialize weights and apply final processing
        self.post_init()

    def tie_weights(self):
        """
        This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when
        passing `target_lang=...` to `from_pretrained(...)`.

        This method is **not** supposed to be called by the user and is prone to be changed in the future.
        """

        # Note that `tie_weights` is usually used to tie input and output embedding weights. The method is re-purposed to
        # correctly load adapter layers for WavLM so that we do not have to introduce a new API to
        # [`PreTrainedModel`]. While slightly hacky, WavLM never has to tie input and output embeddings, so that it is
        # ok to repurpose this function here.
        target_lang = self.target_lang

        if target_lang is not None and getattr(self.config, "adapter_attn_dim", None) is None:
            raise ValueError(f"Cannot pass `target_lang`: {target_lang} if `config.adapter_attn_dim` is not defined.")
        elif target_lang is None and getattr(self.config, "adapter_attn_dim", None) is not None:
            logger.info("By default `target_lang` is set to 'eng'.")
        elif target_lang is not None:
            self.load_adapter(target_lang)

    def freeze_feature_extractor(self):
        """
        Calling this function will disable the gradient computation for the feature encoder so that its parameter will
        not be updated during training.
        """
        warnings.warn(
            "The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. "
            "Please use the equivalent `freeze_feature_encoder` method instead.",
            FutureWarning,
        )
        self.freeze_feature_encoder()

    def freeze_feature_encoder(self):
        """
        Calling this function will disable the gradient computation for the feature encoder so that its parameter will
        not be updated during training.
        """
        self.wavlm.feature_extractor._freeze_parameters()

    def freeze_base_model(self):
        """
        Calling this function will disable the gradient computation for the base model so that its parameters will not
        be updated during training. Only the classification head will be updated.
        """
        for param in self.wavlm.get_parameters():
            param.requires_grad = False

    def forward(
        self,
        input_values: Optional[mindspore.Tensor],
        attention_mask: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
        labels: Optional[mindspore.Tensor] = None,
    ) -> Union[Tuple, CausalLMOutput]:
        r"""
        Args:
            labels (`mindspore.Tensor` of shape `(batch_size, target_length)`, *optional*):
                Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to
                the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`.
                All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ...,
                config.vocab_size - 1]`.
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
        if labels is not None and labels.max() >= self.config.vocab_size:
            raise ValueError(f"Label values must be <= vocab_size: {self.config.vocab_size}")

        outputs = self.wavlm(
            input_values,
            attention_mask=attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        hidden_states = outputs[0]
        hidden_states = self.dropout(hidden_states)

        logits = self.lm_head(hidden_states)

        loss = None
        if labels is not None:
            # retrieve loss input_lengths from attention_mask
            attention_mask = (
                attention_mask if attention_mask is not None else ops.ones_like(input_values, dtype=mindspore.int64)
            )
            input_lengths = self._get_feat_extract_output_lengths(attention_mask.sum(-1)).astype(mindspore.int64)

            # assuming that padded tokens are filled with -100
            # when not being attended to
            labels_mask = labels >= 0
            target_lengths = labels_mask.sum(-1)
            flattened_targets = labels.masked_select(labels_mask)

            # ctc_loss doesn't support fp16
            log_probs = ops.log_softmax(logits.astype(mindspore.float32), axis=-1).swapaxes(0, 1)

            # with torch.backends.cudnn.flags(enabled=False):
            loss = ops.ctc_loss(
                log_probs,
                labels,
                input_lengths,
                target_lengths,
                blank=self.config.pad_token_id,
                reduction=self.config.ctc_loss_reduction,
                zero_infinity=self.config.ctc_zero_infinity,
            )

        if not return_dict:
            output = (logits,) + outputs[_HIDDEN_STATES_START_POSITION:]
            return ((loss,) + output) if loss is not None else output

        return CausalLMOutput(
            loss=loss, logits=logits, hidden_states=outputs.hidden_states, attentions=outputs.attentions
        )

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMForCTC.forward(input_values, attention_mask=None, output_attentions=None, output_hidden_states=None, return_dict=None, labels=None)

PARAMETER

DESCRIPTION

labels

Labels for connectionist temporal classification. Note that target_length has to be smaller or equal to the sequence length of the output logits. Indices are selected in [-100, 0, ..., config.vocab_size - 1]. All labels set to -100 are ignored (masked), the loss is only computed for labels in [0, ..., config.vocab_size - 1]. TYPE: `mindspore.Tensor` of shape `(batch_size, target_length)`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

def forward(
    self,
    input_values: Optional[mindspore.Tensor],
    attention_mask: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
    labels: Optional[mindspore.Tensor] = None,
) -> Union[Tuple, CausalLMOutput]:
    r"""
    Args:
        labels (`mindspore.Tensor` of shape `(batch_size, target_length)`, *optional*):
            Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to
            the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`.
            All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ...,
            config.vocab_size - 1]`.
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    if labels is not None and labels.max() >= self.config.vocab_size:
        raise ValueError(f"Label values must be <= vocab_size: {self.config.vocab_size}")

    outputs = self.wavlm(
        input_values,
        attention_mask=attention_mask,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    hidden_states = outputs[0]
    hidden_states = self.dropout(hidden_states)

    logits = self.lm_head(hidden_states)

    loss = None
    if labels is not None:
        # retrieve loss input_lengths from attention_mask
        attention_mask = (
            attention_mask if attention_mask is not None else ops.ones_like(input_values, dtype=mindspore.int64)
        )
        input_lengths = self._get_feat_extract_output_lengths(attention_mask.sum(-1)).astype(mindspore.int64)

        # assuming that padded tokens are filled with -100
        # when not being attended to
        labels_mask = labels >= 0
        target_lengths = labels_mask.sum(-1)
        flattened_targets = labels.masked_select(labels_mask)

        # ctc_loss doesn't support fp16
        log_probs = ops.log_softmax(logits.astype(mindspore.float32), axis=-1).swapaxes(0, 1)

        # with torch.backends.cudnn.flags(enabled=False):
        loss = ops.ctc_loss(
            log_probs,
            labels,
            input_lengths,
            target_lengths,
            blank=self.config.pad_token_id,
            reduction=self.config.ctc_loss_reduction,
            zero_infinity=self.config.ctc_zero_infinity,
        )

    if not return_dict:
        output = (logits,) + outputs[_HIDDEN_STATES_START_POSITION:]
        return ((loss,) + output) if loss is not None else output

    return CausalLMOutput(
        loss=loss, logits=logits, hidden_states=outputs.hidden_states, attentions=outputs.attentions
    )

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMForCTC.freeze_base_model()

Calling this function will disable the gradient computation for the base model so that its parameters will not be updated during training. Only the classification head will be updated.

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

def freeze_base_model(self):
    """
    Calling this function will disable the gradient computation for the base model so that its parameters will not
    be updated during training. Only the classification head will be updated.
    """
    for param in self.wavlm.get_parameters():
        param.requires_grad = False

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMForCTC.freeze_feature_encoder()

Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training.

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

def freeze_feature_encoder(self):
    """
    Calling this function will disable the gradient computation for the feature encoder so that its parameter will
    not be updated during training.
    """
    self.wavlm.feature_extractor._freeze_parameters()

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMForCTC.freeze_feature_extractor()

Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training.

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

def freeze_feature_extractor(self):
    """
    Calling this function will disable the gradient computation for the feature encoder so that its parameter will
    not be updated during training.
    """
    warnings.warn(
        "The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. "
        "Please use the equivalent `freeze_feature_encoder` method instead.",
        FutureWarning,
    )
    self.freeze_feature_encoder()

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMForCTC.tie_weights()

This method overwrites [~PreTrainedModel.tie_weights] so that adapter weights can be correctly loaded when passing target_lang=... to from_pretrained(...).

This method is not supposed to be called by the user and is prone to be changed in the future.

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

def tie_weights(self):
    """
    This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when
    passing `target_lang=...` to `from_pretrained(...)`.

    This method is **not** supposed to be called by the user and is prone to be changed in the future.
    """

    # Note that `tie_weights` is usually used to tie input and output embedding weights. The method is re-purposed to
    # correctly load adapter layers for WavLM so that we do not have to introduce a new API to
    # [`PreTrainedModel`]. While slightly hacky, WavLM never has to tie input and output embeddings, so that it is
    # ok to repurpose this function here.
    target_lang = self.target_lang

    if target_lang is not None and getattr(self.config, "adapter_attn_dim", None) is None:
        raise ValueError(f"Cannot pass `target_lang`: {target_lang} if `config.adapter_attn_dim` is not defined.")
    elif target_lang is None and getattr(self.config, "adapter_attn_dim", None) is not None:
        logger.info("By default `target_lang` is set to 'eng'.")
    elif target_lang is not None:
        self.load_adapter(target_lang)

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMForSequenceClassification

Bases: WavLMPreTrainedModel

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

class WavLMForSequenceClassification(WavLMPreTrainedModel):
    def __init__(self, config):
        super().__init__(config)

        if hasattr(config, "add_adapter") and config.add_adapter:
            raise ValueError(
                "Sequence classification does not support the use of WavLM adapters (config.add_adapter=True)"
            )
        self.wavlm = WavLMModel(config)
        num_layers = config.num_hidden_layers + 1  # transformer layers + input embeddings
        if config.use_weighted_layer_sum:
            self.layer_weights = mindspore.Parameter(ops.ones(num_layers) / num_layers)
        self.projector = nn.Linear(config.hidden_size, config.classifier_proj_size)
        self.classifier = nn.Linear(config.classifier_proj_size, config.num_labels)

        # Initialize weights and apply final processing
        self.post_init()

    # Copied from transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForSequenceClassification.freeze_feature_extractor
    def freeze_feature_extractor(self):
        """
        Calling this function will disable the gradient computation for the feature encoder so that its parameters will
        not be updated during training.
        """
        warnings.warn(
            "The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. "
            "Please use the equivalent `freeze_feature_encoder` method instead.",
            FutureWarning,
        )
        self.freeze_feature_encoder()

    # Copied from transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForSequenceClassification.freeze_feature_encoder with wav2vec2->wavlm
    def freeze_feature_encoder(self):
        """
        Calling this function will disable the gradient computation for the feature encoder so that its parameter will
        not be updated during training.
        """
        self.wavlm.feature_extractor._freeze_parameters()

    # Copied from transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForSequenceClassification.freeze_base_model with wav2vec2->wavlm
    def freeze_base_model(self):
        """
        Calling this function will disable the gradient computation for the base model so that its parameters will not
        be updated during training. Only the classification head will be updated.
        """
        for param in self.wavlm.get_parameters():
            param.requires_grad = False

    # Copied from transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForSequenceClassification.forward with Wav2Vec2->WavLM, wav2vec2->wavlm
    def forward(
        self,
        input_values: Optional[mindspore.Tensor],
        attention_mask: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
        labels: Optional[mindspore.Tensor] = None,
    ) -> Union[Tuple, SequenceClassifierOutput]:
        r"""
        Args:
            labels (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
                Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
                config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
                `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
        """

        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
        output_hidden_states = True if self.config.use_weighted_layer_sum else output_hidden_states

        outputs = self.wavlm(
            input_values,
            attention_mask=attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )


        if self.config.use_weighted_layer_sum:
            hidden_states = outputs[_HIDDEN_STATES_START_POSITION]
            hidden_states = ops.stack(hidden_states, axis=1)
            norm_weights = ops.softmax(self.layer_weights, axis=-1)
            hidden_states = (hidden_states * norm_weights.view(-1, 1, 1)).sum(axis=1)
        else:
            hidden_states = outputs[0]

        hidden_states = self.projector(hidden_states)
        if attention_mask is None:
            pooled_output = hidden_states.mean(axis=1)
        else:
            padding_mask = self._get_feature_vector_attention_mask(hidden_states.shape[1], attention_mask)
            hidden_states[~padding_mask] = 0.0
            pooled_output = hidden_states.sum(axis=1) / padding_mask.sum(axis=1).view(-1, 1)

        logits = self.classifier(pooled_output)

        loss = None
        if labels is not None:
            loss = ops.cross_entropy(logits.view(-1, self.config.num_labels), labels.view(-1))

        if not return_dict:
            output = (logits,) + outputs[_HIDDEN_STATES_START_POSITION:]
            return ((loss,) + output) if loss is not None else output

        return SequenceClassifierOutput(
            loss=loss,
            logits=logits,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMForSequenceClassification.forward(input_values, attention_mask=None, output_attentions=None, output_hidden_states=None, return_dict=None, labels=None)

PARAMETER	DESCRIPTION
labels	Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If config.num_labels > 1 a classification loss is computed (Cross-Entropy). TYPE: `mindspore.Tensor` of shape `(batch_size,)`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

def forward(
    self,
    input_values: Optional[mindspore.Tensor],
    attention_mask: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
    labels: Optional[mindspore.Tensor] = None,
) -> Union[Tuple, SequenceClassifierOutput]:
    r"""
    Args:
        labels (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
            Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
            config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
            `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
    """

    return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    output_hidden_states = True if self.config.use_weighted_layer_sum else output_hidden_states

    outputs = self.wavlm(
        input_values,
        attention_mask=attention_mask,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )


    if self.config.use_weighted_layer_sum:
        hidden_states = outputs[_HIDDEN_STATES_START_POSITION]
        hidden_states = ops.stack(hidden_states, axis=1)
        norm_weights = ops.softmax(self.layer_weights, axis=-1)
        hidden_states = (hidden_states * norm_weights.view(-1, 1, 1)).sum(axis=1)
    else:
        hidden_states = outputs[0]

    hidden_states = self.projector(hidden_states)
    if attention_mask is None:
        pooled_output = hidden_states.mean(axis=1)
    else:
        padding_mask = self._get_feature_vector_attention_mask(hidden_states.shape[1], attention_mask)
        hidden_states[~padding_mask] = 0.0
        pooled_output = hidden_states.sum(axis=1) / padding_mask.sum(axis=1).view(-1, 1)

    logits = self.classifier(pooled_output)

    loss = None
    if labels is not None:
        loss = ops.cross_entropy(logits.view(-1, self.config.num_labels), labels.view(-1))

    if not return_dict:
        output = (logits,) + outputs[_HIDDEN_STATES_START_POSITION:]
        return ((loss,) + output) if loss is not None else output

    return SequenceClassifierOutput(
        loss=loss,
        logits=logits,
        hidden_states=outputs.hidden_states,
        attentions=outputs.attentions,
    )

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMForSequenceClassification.freeze_base_model()

Calling this function will disable the gradient computation for the base model so that its parameters will not be updated during training. Only the classification head will be updated.

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

def freeze_base_model(self):
    """
    Calling this function will disable the gradient computation for the base model so that its parameters will not
    be updated during training. Only the classification head will be updated.
    """
    for param in self.wavlm.get_parameters():
        param.requires_grad = False

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMForSequenceClassification.freeze_feature_encoder()

Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training.

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

def freeze_feature_encoder(self):
    """
    Calling this function will disable the gradient computation for the feature encoder so that its parameter will
    not be updated during training.
    """
    self.wavlm.feature_extractor._freeze_parameters()

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMForSequenceClassification.freeze_feature_extractor()

Calling this function will disable the gradient computation for the feature encoder so that its parameters will not be updated during training.

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

def freeze_feature_extractor(self):
    """
    Calling this function will disable the gradient computation for the feature encoder so that its parameters will
    not be updated during training.
    """
    warnings.warn(
        "The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. "
        "Please use the equivalent `freeze_feature_encoder` method instead.",
        FutureWarning,
    )
    self.freeze_feature_encoder()

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMForXVector

Bases: WavLMPreTrainedModel

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

class WavLMForXVector(WavLMPreTrainedModel):
    def __init__(self, config):
        super().__init__(config)

        self.wavlm = WavLMModel(config)
        num_layers = config.num_hidden_layers + 1  # transformer layers + input embeddings
        if config.use_weighted_layer_sum:
            self.layer_weights = mindspore.Parameter(ops.ones(num_layers) / num_layers)
        self.projector = nn.Linear(config.hidden_size, config.tdnn_dim[0])

        tdnn_layers = [TDNNLayer(config, i) for i in range(len(config.tdnn_dim))]
        self.tdnn = nn.ModuleList(tdnn_layers)

        self.feature_extractor = nn.Linear(config.tdnn_dim[-1] * 2, config.xvector_output_dim)
        self.classifier = nn.Linear(config.xvector_output_dim, config.xvector_output_dim)

        self.objective = AMSoftmaxLoss(config.xvector_output_dim, config.num_labels)

        self.init_weights()

    def freeze_feature_extractor(self):
        """
        Calling this function will disable the gradient computation for the feature encoder so that its parameter will
        not be updated during training.
        """
        warnings.warn(
            "The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. "
            "Please use the equivalent `freeze_feature_encoder` method instead.",
            FutureWarning,
        )
        self.freeze_feature_encoder()

    def freeze_feature_encoder(self):
        """
        Calling this function will disable the gradient computation for the feature encoder so that its parameter will
        not be updated during training.
        """
        self.wavlm.feature_extractor._freeze_parameters()

    def freeze_base_model(self):
        """
        Calling this function will disable the gradient computation for the base model so that its parameters will not
        be updated during training. Only the classification head will be updated.
        """
        for param in self.wavlm.get_parameters():
            param.requires_grad = False

    def _get_tdnn_output_lengths(self, input_lengths: Union[mindspore.Tensor, int]):
        """
        Computes the output length of the TDNN layers
        """

        def _conv_out_length(input_length, kernel_size, stride):
            # 1D convolutional layer output length formula taken
            # from https://pytorch.org/docs/stable/generated/torch.nn.Conv1d.html
            return (input_length - kernel_size) // stride + 1

        for kernel_size in self.config.tdnn_kernel:
            input_lengths = _conv_out_length(input_lengths, kernel_size, 1)

        return input_lengths

    def forward(
        self,
        input_values: Optional[mindspore.Tensor],
        attention_mask: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
        labels: Optional[mindspore.Tensor] = None,
    ) -> Union[Tuple, XVectorOutput]:
        r"""
        Args:
            labels (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
                Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
                config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
                `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
        """

        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
        output_hidden_states = True if self.config.use_weighted_layer_sum else output_hidden_states

        outputs = self.wavlm(
            input_values,
            attention_mask=attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        if self.config.use_weighted_layer_sum:
            hidden_states = outputs[_HIDDEN_STATES_START_POSITION]
            hidden_states = ops.stack(hidden_states, axis=1)
            norm_weights = ops.softmax(self.layer_weights, axis=-1)
            hidden_states = (hidden_states * norm_weights.view(-1, 1, 1)).sum(axis=1)
        else:
            hidden_states = outputs[0]

        hidden_states = self.projector(hidden_states)

        for tdnn_layer in self.tdnn:
            hidden_states = tdnn_layer(hidden_states)

        # Statistic Pooling
        if attention_mask is None:
            mean_features = hidden_states.mean(axis=1)
            # std_features = hidden_states.std(axis=1)
            std_features = ops.std(hidden_states, axis=1, keepdims=True)[:,0,:]
        else:
            feat_extract_output_lengths = self._get_feat_extract_output_lengths(attention_mask.sum(axis=1))
            tdnn_output_lengths = self._get_tdnn_output_lengths(feat_extract_output_lengths)
            mean_features = []
            std_features = []
            for i, length in enumerate(tdnn_output_lengths):
                mean_features.append(hidden_states[i, :length].mean(axis=0))
                # std_features.append(hidden_states[i, :length].std(axis=0))
                std_features.append(ops.std(hidden_states[i, :length], axis=0, keepdims=True)[0,:])
            mean_features = ops.stack(mean_features)
            std_features = ops.stack(std_features)
        statistic_pooling = ops.cat([mean_features, std_features], axis=-1)

        output_embeddings = self.feature_extractor(statistic_pooling)
        logits = self.classifier(output_embeddings)

        loss = None
        if labels is not None:
            loss = self.objective(logits, labels)

        if not return_dict:
            output = (logits, output_embeddings) + outputs[_HIDDEN_STATES_START_POSITION:]
            return ((loss,) + output) if loss is not None else output

        return XVectorOutput(
            loss=loss,
            logits=logits,
            embeddings=output_embeddings,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMForXVector.forward(input_values, attention_mask=None, output_attentions=None, output_hidden_states=None, return_dict=None, labels=None)

PARAMETER	DESCRIPTION
labels	Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If config.num_labels > 1 a classification loss is computed (Cross-Entropy). TYPE: `mindspore.Tensor` of shape `(batch_size,)`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

def forward(
    self,
    input_values: Optional[mindspore.Tensor],
    attention_mask: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
    labels: Optional[mindspore.Tensor] = None,
) -> Union[Tuple, XVectorOutput]:
    r"""
    Args:
        labels (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
            Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
            config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
            `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
    """

    return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    output_hidden_states = True if self.config.use_weighted_layer_sum else output_hidden_states

    outputs = self.wavlm(
        input_values,
        attention_mask=attention_mask,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    if self.config.use_weighted_layer_sum:
        hidden_states = outputs[_HIDDEN_STATES_START_POSITION]
        hidden_states = ops.stack(hidden_states, axis=1)
        norm_weights = ops.softmax(self.layer_weights, axis=-1)
        hidden_states = (hidden_states * norm_weights.view(-1, 1, 1)).sum(axis=1)
    else:
        hidden_states = outputs[0]

    hidden_states = self.projector(hidden_states)

    for tdnn_layer in self.tdnn:
        hidden_states = tdnn_layer(hidden_states)

    # Statistic Pooling
    if attention_mask is None:
        mean_features = hidden_states.mean(axis=1)
        # std_features = hidden_states.std(axis=1)
        std_features = ops.std(hidden_states, axis=1, keepdims=True)[:,0,:]
    else:
        feat_extract_output_lengths = self._get_feat_extract_output_lengths(attention_mask.sum(axis=1))
        tdnn_output_lengths = self._get_tdnn_output_lengths(feat_extract_output_lengths)
        mean_features = []
        std_features = []
        for i, length in enumerate(tdnn_output_lengths):
            mean_features.append(hidden_states[i, :length].mean(axis=0))
            # std_features.append(hidden_states[i, :length].std(axis=0))
            std_features.append(ops.std(hidden_states[i, :length], axis=0, keepdims=True)[0,:])
        mean_features = ops.stack(mean_features)
        std_features = ops.stack(std_features)
    statistic_pooling = ops.cat([mean_features, std_features], axis=-1)

    output_embeddings = self.feature_extractor(statistic_pooling)
    logits = self.classifier(output_embeddings)

    loss = None
    if labels is not None:
        loss = self.objective(logits, labels)

    if not return_dict:
        output = (logits, output_embeddings) + outputs[_HIDDEN_STATES_START_POSITION:]
        return ((loss,) + output) if loss is not None else output

    return XVectorOutput(
        loss=loss,
        logits=logits,
        embeddings=output_embeddings,
        hidden_states=outputs.hidden_states,
        attentions=outputs.attentions,
    )

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMForXVector.freeze_base_model()

Calling this function will disable the gradient computation for the base model so that its parameters will not be updated during training. Only the classification head will be updated.

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

def freeze_base_model(self):
    """
    Calling this function will disable the gradient computation for the base model so that its parameters will not
    be updated during training. Only the classification head will be updated.
    """
    for param in self.wavlm.get_parameters():
        param.requires_grad = False

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMForXVector.freeze_feature_encoder()

Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training.

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

def freeze_feature_encoder(self):
    """
    Calling this function will disable the gradient computation for the feature encoder so that its parameter will
    not be updated during training.
    """
    self.wavlm.feature_extractor._freeze_parameters()

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMForXVector.freeze_feature_extractor()

Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training.

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

def freeze_feature_extractor(self):
    """
    Calling this function will disable the gradient computation for the feature encoder so that its parameter will
    not be updated during training.
    """
    warnings.warn(
        "The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. "
        "Please use the equivalent `freeze_feature_encoder` method instead.",
        FutureWarning,
    )
    self.freeze_feature_encoder()

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMGumbelVectorQuantizer

Bases: Module

Vector quantization using gumbel softmax. See CATEGORICAL REPARAMETERIZATION WITH GUMBEL-SOFTMAX for more information.

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

class WavLMGumbelVectorQuantizer(nn.Module):
    """
    Vector quantization using gumbel softmax. See [CATEGORICAL REPARAMETERIZATION WITH
    GUMBEL-SOFTMAX](https://arxiv.org/pdf/1611.01144.pdf) for more information.
    """

    def __init__(self, config):
        super().__init__()
        self.num_groups = config.num_codevector_groups
        self.num_vars = config.num_codevectors_per_group

        if config.codevector_dim % self.num_groups != 0:
            raise ValueError(
                f"`config.codevector_dim {config.codevector_dim} must be divisible"
                f" by `config.num_codevector_groups` {self.num_groups} "
                "for concatenation."
            )

        # storage for codebook variables (codewords)
        self.codevectors = mindspore.Parameter(
            mindspore.Tensor(1, self.num_groups * self.num_vars, config.codevector_dim // self.num_groups)
        )
        self.weight_proj = nn.Linear(config.conv_dim[-1], self.num_groups * self.num_vars)

        # can be decayed for training
        self.temperature = 2

    @staticmethod
    def _compute_perplexity(probs):
        marginal_probs = probs.mean(axis=0)
        perplexity = ops.exp(-ops.sum(marginal_probs * ops.log(marginal_probs + 1e-7), dim=-1)).sum()
        return perplexity

    def forward(self, hidden_states):
        batch_size, sequence_length, hidden_size = hidden_states.shape

        # project to codevector dim
        hidden_states = self.weight_proj(hidden_states)
        hidden_states = hidden_states.view(batch_size * sequence_length * self.num_groups, -1)

        if self.training:
            # sample code vector probs via gumbel in differentiateable way
            codevector_probs = ops.gumbel_softmax(hidden_states.float(), tau=self.temperature, hard=True)
            codevector_probs = codevector_probs.type_as(hidden_states)

            # compute perplexity
            codevector_soft_dist = ops.softmax(
                hidden_states.view(batch_size * sequence_length, self.num_groups, -1).float(), axis=-1
            )
            perplexity = self._compute_perplexity(codevector_soft_dist)
        else:
            # take argmax in non-differentiable way
            # comptute hard codevector distribution (one hot)
            codevector_idx = hidden_states.argmax(dim=-1)
            codevector_probs = hidden_states.new_zeros(*hidden_states.shape).scatter_(
                -1, codevector_idx.view(-1, 1), 1.0
            )
            codevector_probs = codevector_probs.view(batch_size * sequence_length, self.num_groups, -1)

            perplexity = self._compute_perplexity(codevector_probs)

        codevector_probs = codevector_probs.view(batch_size * sequence_length, -1)
        # use probs to retrieve codevectors
        codevectors_per_group = codevector_probs.unsqueeze(-1) * self.codevectors
        codevectors = codevectors_per_group.view(batch_size * sequence_length, self.num_groups, self.num_vars, -1)
        codevectors = codevectors.sum(-2).view(batch_size, sequence_length, -1)

        return codevectors, perplexity

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMModel

Bases: WavLMPreTrainedModel

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

class WavLMModel(WavLMPreTrainedModel):
    def __init__(self, config: WavLMConfig):
        super().__init__(config)
        self.config = config
        self.feature_extractor = WavLMFeatureEncoder(config)
        self.feature_projection = WavLMFeatureProjection(config)

        # model only needs masking vector if mask prob is > 0.0
        if config.mask_time_prob > 0.0 or config.mask_feature_prob > 0.0:

            self.masked_spec_embed_fixed = mindspore.Tensor([0.6690, 0.8174, 0.0483, 0.8542, 0.5385, 0.7270, 0.8509, 0.7227, 0.4435,
                0.9075, 0.5943, 0.5755, 0.2277, 0.5103, 0.1635, 0.6906, 0.3977, 0.9756,
                0.0362, 0.9023, 0.3385, 0.1798, 0.5457, 0.9846, 0.8872, 0.7534, 0.7174,
                0.9129, 0.0361, 0.5914, 0.6458, 0.0551, 0.4543, 0.2475, 0.5665, 0.5622,
                0.7827, 0.2933, 0.4264, 0.2142, 0.8809, 0.7395, 0.8117, 0.8880, 0.9114,
                0.7873, 0.1974, 0.5749, 0.2186, 0.7509, 0.9451, 0.5604, 0.4548, 0.3830,
                0.8748, 0.0481, 0.7892, 0.6930, 0.6757, 0.3346, 0.5754, 0.0830, 0.3630,
                0.3927, 0.4438, 0.3057, 0.2056, 0.6541, 0.8959, 0.3882, 0.3742, 0.6756,
                0.2212, 0.4545, 0.4845, 0.5233, 0.9661, 0.8705, 0.0297, 0.2031, 0.9059,
                0.2570, 0.3765, 0.6301, 0.2756, 0.4591, 0.2101, 0.5576, 0.1532, 0.3753,
                0.6413, 0.1778, 0.5639, 0.7753, 0.4551, 0.7990, 0.1866, 0.0881, 0.5993,
                0.0529, 0.9180, 0.4496, 0.7429, 0.7545, 0.8755, 0.8374, 0.0907, 0.7265,
                0.7455, 0.0652, 0.0794, 0.3860, 0.9730, 0.7865, 0.8821, 0.2630, 0.2690,
                0.6491, 0.0887, 0.4657, 0.8514, 0.0096, 0.6633, 0.7675, 0.9290, 0.9126,
                0.0885, 0.7826, 0.8512, 0.6113, 0.7821, 0.0923, 0.9687, 0.3606, 0.7457,
                0.3216, 0.4239, 0.0411, 0.1968, 0.6589, 0.9997, 0.6803, 0.3238, 0.0318,
                0.3006, 0.0840, 0.3048, 0.7558, 0.5318, 0.0110, 0.6965, 0.9264, 0.8576,
                0.8286, 0.7549, 0.3492, 0.6382, 0.4695, 0.6429, 0.8461, 0.4037, 0.6143,
                0.6750, 0.0130, 0.5454, 0.8819, 0.7204, 0.8509, 0.5713, 0.3463, 0.3251,
                0.1364, 0.9822, 0.1932, 0.4651, 0.8423, 0.0824, 0.0385, 0.6319, 0.4540,
                0.9898, 0.0858, 0.2168, 0.8091, 0.2082, 0.0317, 0.5799, 0.8108, 0.2224,
                0.1679, 0.2297, 0.1149, 0.6511, 0.8530, 0.2673, 0.2593, 0.1479, 0.6914,
                0.1220, 0.2791, 0.2264, 0.3477, 0.0301, 0.4977, 0.9622, 0.9822, 0.1609,
                0.9212, 0.2130, 0.7508, 0.9012, 0.8798, 0.9235, 0.2774, 0.1695, 0.1931,
                0.6583, 0.8880, 0.1824, 0.5290, 0.8476, 0.5914, 0.2393, 0.2043, 0.5509,
                0.4092, 0.5522, 0.1584, 0.1846, 0.5055, 0.3038, 0.2121, 0.1347, 0.8977,
                0.4759, 0.3980, 0.1729, 0.5186, 0.3864, 0.1076, 0.7897, 0.5062, 0.6262,
                0.3445, 0.7281, 0.5154, 0.1098, 0.8532, 0.8998, 0.1109, 0.1660, 0.2890,
                0.3983, 0.9154, 0.2710, 0.6147, 0.1245, 0.2494, 0.1251, 0.6717, 0.4353,
                0.8889, 0.4446, 0.2871, 0.5897, 0.8086, 0.4644, 0.5078, 0.5242, 0.4318,
                0.9208, 0.2187, 0.1061, 0.2322, 0.9779, 0.1891, 0.5374, 0.8748, 0.2969,
                0.9084, 0.4123, 0.2679, 0.1227, 0.2493, 0.0069, 0.4302, 0.7309, 0.6150,
                0.8707, 0.9405, 0.0665, 0.0617, 0.4912, 0.8631, 0.3454, 0.5959, 0.4082,
                0.5628, 0.1539, 0.4820, 0.2230, 0.7901, 0.9863, 0.3853, 0.6251, 0.0294,
                0.5922, 0.4190, 0.1238, 0.9131, 0.7443, 0.7243, 0.2333, 0.5575, 0.9056,
                0.6038, 0.6373, 0.3231, 0.1106, 0.7115, 0.0738, 0.1821, 0.5646, 0.6631,
                0.9203, 0.3644, 0.8854, 0.7089, 0.9513, 0.6969, 0.6221, 0.9998, 0.3835,
                0.1778, 0.8368, 0.4535, 0.0226, 0.7247, 0.3746, 0.3204, 0.0739, 0.5398,
                0.9403, 0.6918, 0.7779, 0.1451, 0.2665, 0.2724, 0.9406, 0.7556, 0.4615,
                0.9865, 0.9019, 0.4024, 0.0430, 0.5586, 0.0194, 0.4044, 0.8839, 0.6115,
                0.9678, 0.0424, 0.1750, 0.1324, 0.3528, 0.0426, 0.4412, 0.0817, 0.5239,
                0.1943, 0.2168, 0.1862, 0.1268, 0.9675, 0.7493, 0.9916, 0.0120, 0.6652,
                0.3382, 0.1434, 0.0340, 0.5746, 0.2504, 0.6652, 0.4948, 0.9776, 0.8149,
                0.8904, 0.6182, 0.5081, 0.9500, 0.6186, 0.7949, 0.9912, 0.0316, 0.5226,
                0.6809, 0.6388, 0.8631, 0.3738, 0.3314, 0.0405, 0.1620, 0.3713, 0.8028,
                0.9732, 0.9597, 0.3242, 0.2495, 0.2347, 0.2002, 0.5536, 0.1284, 0.7263,
                0.5329, 0.3998, 0.5114, 0.9307, 0.3562, 0.7596, 0.7474, 0.5452, 0.6765,
                0.9079, 0.6698, 0.3373, 0.7954, 0.8829, 0.8574, 0.2378, 0.5754, 0.4218,
                0.4776, 0.6210, 0.0870, 0.7172, 0.4000, 0.7223, 0.3835, 0.0187, 0.6055,
                0.2987, 0.1763, 0.9496, 0.0019, 0.6128, 0.2233, 0.6464, 0.6703, 0.3060,
                0.5027, 0.5011, 0.1066, 0.9224, 0.6772, 0.1122, 0.4799, 0.0956, 0.6784,
                0.2987, 0.4378, 0.8626, 0.1457, 0.8810, 0.2955, 0.3982, 0.9872, 0.2424,
                0.4985, 0.9825, 0.8322, 0.6646, 0.5974, 0.9266, 0.7363, 0.8470, 0.3441,
                0.6455, 0.0959, 0.3900, 0.0110, 0.5135, 0.7431, 0.9956, 0.4753, 0.2459,
                0.1745, 0.4280, 0.3137, 0.5803, 0.8807, 0.0013, 0.2719, 0.2735, 0.0174,
                0.5792, 0.2755, 0.7145, 0.6616, 0.7531, 0.0317, 0.1691, 0.2877, 0.9014,
                0.3965, 0.5576, 0.0569, 0.0952, 0.7354, 0.6605, 0.4193, 0.0895, 0.3981,
                0.5928, 0.1463, 0.7944, 0.8587, 0.8905, 0.5828, 0.8698, 0.0869, 0.5440,
                0.0108, 0.9643, 0.2618, 0.0239, 0.5285, 0.9577, 0.5655, 0.6379, 0.2955,
                0.6893, 0.6071, 0.1768, 0.3647, 0.6052, 0.7924, 0.8311, 0.4018, 0.4684,
                0.7488, 0.9257, 0.1174, 0.9175, 0.2108, 0.7104, 0.0650, 0.9683, 0.1456,
                0.3139, 0.9895, 0.4817, 0.3550, 0.3194, 0.2714, 0.3304, 0.3714, 0.6225,
                0.5636, 0.6906, 0.1564, 0.2612, 0.8385, 0.2389, 0.6572, 0.1156, 0.5804,
                0.3947, 0.0016, 0.2312, 0.0136, 0.2436, 0.7072, 0.4118, 0.6912, 0.1629,
                0.0368, 0.5640, 0.7028, 0.0881, 0.9698, 0.7337, 0.0634, 0.7968, 0.0754,
                0.6724, 0.2065, 0.7023, 0.1979, 0.4276, 0.3267, 0.3916, 0.9641, 0.5335,
                0.3355, 0.5741, 0.9364, 0.7964, 0.2325, 0.4632, 0.0586, 0.4343, 0.9153,
                0.3367, 0.3897, 0.8585, 0.4316, 0.3008, 0.4461, 0.3888, 0.4275, 0.2071,
                0.7893, 0.7605, 0.4429, 0.1573, 0.0303, 0.7489, 0.9437, 0.2839, 0.2179,
                0.3195, 0.4809, 0.1952, 0.8383, 0.0198, 0.8895, 0.4406, 0.9321, 0.5931,
                0.3670, 0.9503, 0.5326, 0.9467, 0.2632, 0.4534, 0.7885, 0.7485, 0.9038,
                0.5202, 0.4448, 0.6610, 0.1788, 0.2415, 0.0186, 0.3090, 0.3962, 0.7363,
                0.5319, 0.0024, 0.5918, 0.0702, 0.3051, 0.3310, 0.6551, 0.7465, 0.2650,
                0.3644, 0.8870, 0.9065, 0.9198, 0.6367, 0.5113, 0.1910, 0.8260, 0.4486,
                0.8939, 0.9591, 0.0051, 0.9798, 0.6846, 0.9752, 0.6470, 0.2136, 0.8094,
                0.1351, 0.6637, 0.1317, 0.5875, 0.3815, 0.3004, 0.5598, 0.2138, 0.2395,
                0.7725, 0.4870, 0.2897, 0.5427, 0.7458, 0.4651, 0.7445, 0.5091, 0.5224,
                0.1761, 0.3968, 0.8253, 0.0378, 0.1911, 0.2917, 0.8945, 0.5533, 0.9208,
                0.9452, 0.5043, 0.4790, 0.6593, 0.4681, 0.5305, 0.2849, 0.7655, 0.8555,
                0.2354, 0.5224, 0.2482, 0.6614, 0.4972, 0.8426, 0.3883, 0.1001, 0.4299,
                0.6966, 0.4446, 0.9288, 0.4683, 0.0273, 0.1940, 0.8093, 0.3530, 0.8765,
                0.8774, 0.7397, 0.6672, 0.8504, 0.9556, 0.9929, 0.3112, 0.7945, 0.2682,
                0.4824, 0.1706, 0.8585, 0.9539, 0.1334, 0.0866, 0.8030, 0.8256, 0.1504,
                0.0553, 0.5819, 0.3482, 0.9587, 0.3867, 0.5643, 0.7611, 0.5880, 0.2536,
                0.6834, 0.3636, 0.3593, 0.1886, 0.2166, 0.0668, 0.8122, 0.2461, 0.5877,
                0.0802, 0.4127, 0.1399])
            if config.hidden_size >= 768:
                self.masked_spec_embed=self.masked_spec_embed_fixed
            else:
                self.masked_spec_embed = ops.abs(mindspore.Tensor(shape=(config.hidden_size), dtype=mindspore.float32, init=Uniform(1.0)))



        if config.do_stable_layer_norm:
            self.encoder = WavLMEncoderStableLayerNorm(config)
        else:
            self.encoder = WavLMEncoder(config)

        self.adapter = WavLMAdapter(config) if config.add_adapter else None

        # Initialize weights and apply final processing
        self.post_init()

    def freeze_feature_extractor(self):
        """
        Calling this function will disable the gradient computation for the feature encoder so that its parameters will
        not be updated during training.
        """
        warnings.warn(
            "The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. "
            "Please use the equivalent `freeze_feature_encoder` method instead.",
            FutureWarning,
        )
        self.freeze_feature_encoder()

    def freeze_feature_encoder(self):
        """
        Calling this function will disable the gradient computation for the feature encoder so that its parameter will
        not be updated during training.
        """
        self.feature_extractor._freeze_parameters()

    def _mask_hidden_states(
        self,
        hidden_states: mindspore.Tensor,
        mask_time_indices: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
    ):
        """
        Masks extracted features along time axis and/or along feature axis according to
        [SpecAugment](https://arxiv.org/abs/1904.08779).
        """

        # `config.apply_spec_augment` can set masking to False
        if not getattr(self.config, "apply_spec_augment", True):
            return hidden_states

        # generate indices & apply SpecAugment along time axis
        batch_size, sequence_length, hidden_size = hidden_states.shape

        if mask_time_indices is not None:
            # apply SpecAugment along time axis with given mask_time_indices
            hidden_states[mask_time_indices] = self.masked_spec_embed.astype(hidden_states.dtype)
        elif self.config.mask_time_prob > 0 and self.training:
            mask_time_indices = _compute_mask_indices(
                (batch_size, sequence_length),
                mask_prob=self.config.mask_time_prob,
                mask_length=self.config.mask_time_length,
                attention_mask=attention_mask,
                min_masks=self.config.mask_time_min_masks,
            )
            mask_time_indices = mindspore.Tensor(mask_time_indices, dtype=mindspore.bool_)
            hidden_states[mask_time_indices] = self.masked_spec_embed.astype(hidden_states.dtype)

        if self.config.mask_feature_prob > 0:
            # generate indices & apply SpecAugment along feature axis
            mask_feature_indices = _compute_mask_indices(
                (batch_size, hidden_size),
                mask_prob=self.config.mask_feature_prob,
                mask_length=self.config.mask_feature_length,
                min_masks=self.config.mask_feature_min_masks,
            )
            mask_feature_indices = mindspore.Tensor(mask_feature_indices, dtype=mindspore.bool_)
            mask_feature_indices = mask_feature_indices[:, None].expand(-1, sequence_length, -1)
            hidden_states[mask_feature_indices] = 0


        return hidden_states

    def forward(
        self,
        input_values: Optional[mindspore.Tensor],
        attention_mask: Optional[mindspore.Tensor] = None,
        mask_time_indices: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, Wav2Vec2BaseModelOutput]:
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        extract_features = self.feature_extractor(input_values)
        extract_features = extract_features.swapaxes(1, 2)

        if attention_mask is not None:
            # compute reduced attention_mask corresponding to feature vectors
            attention_mask = self._get_feature_vector_attention_mask(
                extract_features.shape[1], attention_mask, add_adapter=False
            )

        hidden_states, extract_features = self.feature_projection(extract_features)
        hidden_states = self._mask_hidden_states(
            hidden_states, mask_time_indices=mask_time_indices, attention_mask=attention_mask
        )



        encoder_outputs = self.encoder(
            hidden_states,
            attention_mask=attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        hidden_states = encoder_outputs[0]


        if self.adapter is not None:
            hidden_states = self.adapter(hidden_states)

        if not return_dict:
            return (hidden_states, extract_features) + encoder_outputs[1:]

        return Wav2Vec2BaseModelOutput(
            last_hidden_state=hidden_states,
            extract_features=extract_features,
            hidden_states=encoder_outputs.hidden_states,
            attentions=encoder_outputs.attentions,
        )

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMModel.freeze_feature_encoder()

Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training.

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

def freeze_feature_encoder(self):
    """
    Calling this function will disable the gradient computation for the feature encoder so that its parameter will
    not be updated during training.
    """
    self.feature_extractor._freeze_parameters()

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMModel.freeze_feature_extractor()

Calling this function will disable the gradient computation for the feature encoder so that its parameters will not be updated during training.

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

def freeze_feature_extractor(self):
    """
    Calling this function will disable the gradient computation for the feature encoder so that its parameters will
    not be updated during training.
    """
    warnings.warn(
        "The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. "
        "Please use the equivalent `freeze_feature_encoder` method instead.",
        FutureWarning,
    )
    self.freeze_feature_encoder()

mindnlp.transformers.models.wavlm.modeling_wavlm.WavLMPreTrainedModel

Bases: PreTrainedModel

An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models.

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

class WavLMPreTrainedModel(PreTrainedModel):
    """
    An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained
    models.
    """

    config_class = WavLMConfig
    base_model_prefix = "wavlm"
    main_input_name = "input_values"
    supports_gradient_checkpointing = True

    def _init_weights(self, cell):
        """Initialize the weights"""
        # gumbel softmax requires special init
        if isinstance(cell, WavLMGumbelVectorQuantizer):
            # module.weight_proj.weight.data.normal_(mean=0.0, std=1)
            # module.weight_proj.bias.data.zero_()
            # nn.init.uniform_(module.codevectors)
            cell.weight_proj.weight.set_data(initializer(Normal(1),
                                             cell.weight.shape, cell.weight.dtype))
            cell.weight_proj.bias.set_data(initializer('zeros', cell.bias.shape, cell.bias.dtype))
            cell.codevectors.set_data(initializer(TruncatedNormal(sigma=0.2, mean=0.5, a=-2.5, b=2.5),
                                             cell.codevectors.shape, cell.codevectors.dtype))

        elif isinstance(cell, WavLMPositionalConvEmbedding):
            # nn.init.normal_(
            #     module.conv.weight,
            #     mean=0,
            #     std=2 * math.sqrt(1 / (module.conv.kernel_size[0] * module.conv.in_channels)),
            # )
            # nn.init.constant_(module.conv.bias, 0)
            cell.conv.weight.set_data(initializer(Normal(2 * math.sqrt(1 / (cell.conv.kernel_size[0] * cell.conv.in_channels))),
                                                        cell.conv.weight.shape, cell.conv.weight.dtype))
            cell.conv.bias.set_data(initializer('zeros', cell.conv.bias.shape, cell.conv.bias.dtype))



        elif isinstance(cell, WavLMFeatureProjection):
            # k = math.sqrt(1 / module.projection.in_features)
            # nn.init.uniform_(module.projection.weight, a=-k, b=k)
            # nn.init.uniform_(module.projection.bias, a=-k, b=k)
            k = math.sqrt(1 / cell.projection.in_channels)
            cell.projection.weight.set_data(initializer(Uniform(scale=k),
                                                  cell.projection.weight.shape, cell.projection.weight.dtype))
            cell.projection.bias.set_data(initializer(Uniform(scale=k),
                                                        cell.projection.bias.shape, cell.projection.bias.dtype))

        elif isinstance(cell, nn.Linear):
            # module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
            #
            # if module.bias is not None:
            #     module.bias.data.zero_()
            cell.weight.set_data(initializer(Normal(self.config.initializer_range),
                                             cell.weight.shape, cell.weight.dtype))
            if cell.bias:
                cell.bias.set_data(initializer('zeros', cell.bias.shape, cell.bias.dtype))
        elif isinstance(cell, (nn.LayerNorm, nn.GroupNorm)):
            # module.bias.data.zero_()
            # module.weight.data.fill_(1.0)
            cell.weight.set_data(initializer('ones', cell.weight.shape, cell.weight.dtype))
            cell.bias.set_data(initializer('zeros', cell.bias.shape, cell.bias.dtype))
        elif isinstance(cell, nn.Conv1d):
            # nn.init.kaiming_normal_(module.weight)
            #
            # if module.bias is not None:
            #     k = math.sqrt(module.groups / (module.in_channels * module.kernel_size[0]))
            #     nn.init.uniform_(module.bias, a=-k, b=k)
            cell.weight.set_data(
                initializer(HeNormal(),cell.weight.shape, cell.weight.dtype))
            if cell.bias:
                k = math.sqrt(cell.group / (cell.in_channels * cell.kernel_size[0]))
                cell.bias.set_data(initializer(Uniform(scale=k),
                                                    cell.bias.shape, cell.bias.dtype))


    def _get_feat_extract_output_lengths(
        self, input_lengths: Union[mindspore.Tensor, int], add_adapter: Optional[bool] = None
    ):
        """
        Computes the output length of the convolutional layers
        """

        add_adapter = self.config.add_adapter if add_adapter is None else add_adapter

        def _conv_out_length(input_length, kernel_size, stride):
            # 1D convolutional layer output length formula taken
            # from https://pytorch.org/docs/stable/generated/torch.nn.Conv1d.html
            return ops.div(input_length - kernel_size, stride, rounding_mode="floor") + 1

        for kernel_size, stride in zip(self.config.conv_kernel, self.config.conv_stride):
            input_lengths = _conv_out_length(input_lengths, kernel_size, stride)

        if add_adapter:
            for _ in range(self.config.num_adapter_layers):
                input_lengths = _conv_out_length(input_lengths, 1, self.config.adapter_stride)

        return input_lengths

    def _get_feature_vector_attention_mask(
        self, feature_vector_length: int, attention_mask: mindspore.Tensor, add_adapter=None
    ):
        # Effectively attention_mask.sum(-1), but not inplace to be able to run
        # on inference mode.
        non_padded_lengths = attention_mask.cumsum(axis=-1)[:, -1]

        output_lengths = self._get_feat_extract_output_lengths(non_padded_lengths, add_adapter=add_adapter)
        output_lengths = output_lengths.astype(mindspore.int64)

        batch_size = attention_mask.shape[0]

        attention_mask = ops.zeros(
            (batch_size, feature_vector_length), dtype=attention_mask.dtype
        )
        # these two operations makes sure that all values before the output lengths idxs are attended to
        attention_mask[(ops.arange(attention_mask.shape[0]), output_lengths - 1)] = 1
        attention_mask = attention_mask.flip([-1]).cumsum(-1).flip([-1]).bool()
        return attention_mask

mindnlp.transformers.models.wavlm.modeling_wavlm.linear(x, weight, bias)

inner linear

Source code in mindnlp/transformers/models/wavlm/modeling_wavlm.py

def linear(x, weight, bias):
    """inner linear"""
    out = ops.matmul(x, weight.swapaxes(-1, -2))
    if bias is not None:
        out = out + bias
    return out