distilbert

mindnlp.transformers.models.distilbert.configuration_distilbert

DistilBERT model configuration

mindnlp.transformers.models.distilbert.configuration_distilbert.DistilBertConfig

Bases: PretrainedConfig

This is the configuration class to store the configuration of a [DistilBertModel] or a [TFDistilBertModel]. It is used to instantiate a DistilBERT model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the DistilBERT distilbert-base-uncased architecture.

Configuration objects inherit from [PretrainedConfig] and can be used to control the model outputs. Read the documentation from [PretrainedConfig] for more information.

PARAMETER	DESCRIPTION
vocab_size	Vocabulary size of the DistilBERT model. Defines the number of different tokens that can be represented by the inputs_ids passed when calling [DistilBertModel] or [TFDistilBertModel]. TYPE: `int`, *optional*, defaults to 30522 DEFAULT: 30522
max_position_embeddings	The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048). TYPE: `int`, *optional*, defaults to 512 DEFAULT: 512
sinusoidal_pos_embds	Whether to use sinusoidal positional embeddings. TYPE: `boolean`, *optional*, defaults to `False` DEFAULT: False
n_layers	Number of hidden layers in the Transformer encoder. TYPE: `int`, *optional*, defaults to 6 DEFAULT: 6
n_heads	Number of attention heads for each attention layer in the Transformer encoder. TYPE: `int`, *optional*, defaults to 12 DEFAULT: 12
dim	Dimensionality of the encoder layers and the pooler layer. TYPE: `int`, *optional*, defaults to 768 DEFAULT: 768
hidden_dim	The size of the "intermediate" (often named feed-forward) layer in the Transformer encoder. TYPE: `int`, *optional*, defaults to 3072 DEFAULT: 4 * 768
dropout	The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
attention_dropout	The dropout ratio for the attention probabilities. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
activation	The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu", "relu", "silu" and "gelu_new" are supported. TYPE: `str` or `Callable`, *optional*, defaults to `"gelu"` DEFAULT: 'gelu'
initializer_range	The standard deviation of the truncated_normal_initializer for initializing all weight matrices. TYPE: `float`, *optional*, defaults to 0.02 DEFAULT: 0.02
qa_dropout	The dropout probabilities used in the question answering model [DistilBertForQuestionAnswering]. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
seq_classif_dropout	The dropout probabilities used in the sequence classification and the multiple choice model [DistilBertForSequenceClassification]. TYPE: `float`, *optional*, defaults to 0.2 DEFAULT: 0.2

Example

>>> from transformers import DistilBertConfig, DistilBertModel
...
>>> # Initializing a DistilBERT configuration
>>> configuration = DistilBertConfig()
...
>>> # Initializing a model (with random weights) from the configuration
>>> model = DistilBertModel(configuration)
...
>>> # Accessing the model configuration
>>> configuration = model.config

Source code in mindnlp/transformers/models/distilbert/configuration_distilbert.py

class DistilBertConfig(PretrainedConfig):
    r"""
    This is the configuration class to store the configuration of a [`DistilBertModel`] or a [`TFDistilBertModel`]. It
    is used to instantiate a DistilBERT model according to the specified arguments, defining the model architecture.
    Instantiating a configuration with the defaults will yield a similar configuration to that of the DistilBERT
    [distilbert-base-uncased](https://hf-mirror.com/distilbert-base-uncased) architecture.

    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
    documentation from [`PretrainedConfig`] for more information.

    Args:
        vocab_size (`int`, *optional*, defaults to 30522):
            Vocabulary size of the DistilBERT model. Defines the number of different tokens that can be represented by
            the `inputs_ids` passed when calling [`DistilBertModel`] or [`TFDistilBertModel`].
        max_position_embeddings (`int`, *optional*, defaults to 512):
            The maximum sequence length that this model might ever be used with. Typically set this to something large
            just in case (e.g., 512 or 1024 or 2048).
        sinusoidal_pos_embds (`boolean`, *optional*, defaults to `False`):
            Whether to use sinusoidal positional embeddings.
        n_layers (`int`, *optional*, defaults to 6):
            Number of hidden layers in the Transformer encoder.
        n_heads (`int`, *optional*, defaults to 12):
            Number of attention heads for each attention layer in the Transformer encoder.
        dim (`int`, *optional*, defaults to 768):
            Dimensionality of the encoder layers and the pooler layer.
        hidden_dim (`int`, *optional*, defaults to 3072):
            The size of the "intermediate" (often named feed-forward) layer in the Transformer encoder.
        dropout (`float`, *optional*, defaults to 0.1):
            The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
        attention_dropout (`float`, *optional*, defaults to 0.1):
            The dropout ratio for the attention probabilities.
        activation (`str` or `Callable`, *optional*, defaults to `"gelu"`):
            The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`,
            `"relu"`, `"silu"` and `"gelu_new"` are supported.
        initializer_range (`float`, *optional*, defaults to 0.02):
            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
        qa_dropout (`float`, *optional*, defaults to 0.1):
            The dropout probabilities used in the question answering model [`DistilBertForQuestionAnswering`].
        seq_classif_dropout (`float`, *optional*, defaults to 0.2):
            The dropout probabilities used in the sequence classification and the multiple choice model
            [`DistilBertForSequenceClassification`].

    Example:
        ```python
        >>> from transformers import DistilBertConfig, DistilBertModel
        ...
        >>> # Initializing a DistilBERT configuration
        >>> configuration = DistilBertConfig()
        ...
        >>> # Initializing a model (with random weights) from the configuration
        >>> model = DistilBertModel(configuration)
        ...
        >>> # Accessing the model configuration
        >>> configuration = model.config
        ```
    """
    model_type = "distilbert"
    attribute_map = {
        "hidden_size": "dim",
        "num_attention_heads": "n_heads",
        "num_hidden_layers": "n_layers",
    }

    def __init__(
        self,
        vocab_size=30522,
        max_position_embeddings=512,
        sinusoidal_pos_embds=False,
        n_layers=6,
        n_heads=12,
        dim=768,
        hidden_dim=4 * 768,
        dropout=0.1,
        attention_dropout=0.1,
        activation="gelu",
        initializer_range=0.02,
        qa_dropout=0.1,
        seq_classif_dropout=0.2,
        pad_token_id=0,
        **kwargs,
    ):
        """
        Initializes a new instance of the DistilBertConfig class.

        Args:
            self (DistilBertConfig): The instance that the method is called on.
            vocab_size (int, optional): The size of the vocabulary. Defaults to 30522.
            max_position_embeddings (int, optional): The maximum number of tokens in a sequence. Defaults to 512.
            sinusoidal_pos_embds (bool, optional): Whether to use sinusoidal positional embeddings. Defaults to False.
            n_layers (int, optional): The number of layers in the transformer encoder. Defaults to 6.
            n_heads (int, optional): The number of attention heads in each layer. Defaults to 12.
            dim (int, optional): The dimensionality of the encoder layers. Defaults to 768.
            hidden_dim (int, optional): The dimensionality of the hidden layers in the feed-forward network. Defaults to 4 * 768.
            dropout (float, optional): The dropout probability for all fully connected layers. Defaults to 0.1.
            attention_dropout (float, optional): The dropout probability for the attention layers. Defaults to 0.1.
            activation (str, optional): The activation function used in the feed-forward network. Defaults to 'gelu'.
            initializer_range (float, optional): The range of the initializer. Defaults to 0.02.
            qa_dropout (float, optional): The dropout probability for the question answering head. Defaults to 0.1.
            seq_classif_dropout (float, optional): The dropout probability for the sequence classification head. Defaults to 0.2.
            pad_token_id (int, optional): The id of the padding token. Defaults to 0.

        Returns:
            None

        Raises:
            None
        """
        self.vocab_size = vocab_size
        self.max_position_embeddings = max_position_embeddings
        self.sinusoidal_pos_embds = sinusoidal_pos_embds
        self.n_layers = n_layers
        self.n_heads = n_heads
        self.dim = dim
        self.hidden_dim = hidden_dim
        self.dropout = dropout
        self.attention_dropout = attention_dropout
        self.activation = activation
        self.initializer_range = initializer_range
        self.qa_dropout = qa_dropout
        self.seq_classif_dropout = seq_classif_dropout
        super().__init__(**kwargs, pad_token_id=pad_token_id)

mindnlp.transformers.models.distilbert.configuration_distilbert.DistilBertConfig.__init__(vocab_size=30522, max_position_embeddings=512, sinusoidal_pos_embds=False, n_layers=6, n_heads=12, dim=768, hidden_dim=4 * 768, dropout=0.1, attention_dropout=0.1, activation='gelu', initializer_range=0.02, qa_dropout=0.1, seq_classif_dropout=0.2, pad_token_id=0, **kwargs)

Initializes a new instance of the DistilBertConfig class.

PARAMETER	DESCRIPTION
self	The instance that the method is called on. TYPE: DistilBertConfig
vocab_size	The size of the vocabulary. Defaults to 30522. TYPE: int DEFAULT: 30522
max_position_embeddings	The maximum number of tokens in a sequence. Defaults to 512. TYPE: int DEFAULT: 512
sinusoidal_pos_embds	Whether to use sinusoidal positional embeddings. Defaults to False. TYPE: bool DEFAULT: False
n_layers	The number of layers in the transformer encoder. Defaults to 6. TYPE: int DEFAULT: 6
n_heads	The number of attention heads in each layer. Defaults to 12. TYPE: int DEFAULT: 12
dim	The dimensionality of the encoder layers. Defaults to 768. TYPE: int DEFAULT: 768
hidden_dim	The dimensionality of the hidden layers in the feed-forward network. Defaults to 4 * 768. TYPE: int DEFAULT: 4 * 768
dropout	The dropout probability for all fully connected layers. Defaults to 0.1. TYPE: float DEFAULT: 0.1
attention_dropout	The dropout probability for the attention layers. Defaults to 0.1. TYPE: float DEFAULT: 0.1
activation	The activation function used in the feed-forward network. Defaults to 'gelu'. TYPE: str DEFAULT: 'gelu'
initializer_range	The range of the initializer. Defaults to 0.02. TYPE: float DEFAULT: 0.02
qa_dropout	The dropout probability for the question answering head. Defaults to 0.1. TYPE: float DEFAULT: 0.1
seq_classif_dropout	The dropout probability for the sequence classification head. Defaults to 0.2. TYPE: float DEFAULT: 0.2
pad_token_id	The id of the padding token. Defaults to 0. TYPE: int DEFAULT: 0

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/distilbert/configuration_distilbert.py

def __init__(
    self,
    vocab_size=30522,
    max_position_embeddings=512,
    sinusoidal_pos_embds=False,
    n_layers=6,
    n_heads=12,
    dim=768,
    hidden_dim=4 * 768,
    dropout=0.1,
    attention_dropout=0.1,
    activation="gelu",
    initializer_range=0.02,
    qa_dropout=0.1,
    seq_classif_dropout=0.2,
    pad_token_id=0,
    **kwargs,
):
    """
    Initializes a new instance of the DistilBertConfig class.

    Args:
        self (DistilBertConfig): The instance that the method is called on.
        vocab_size (int, optional): The size of the vocabulary. Defaults to 30522.
        max_position_embeddings (int, optional): The maximum number of tokens in a sequence. Defaults to 512.
        sinusoidal_pos_embds (bool, optional): Whether to use sinusoidal positional embeddings. Defaults to False.
        n_layers (int, optional): The number of layers in the transformer encoder. Defaults to 6.
        n_heads (int, optional): The number of attention heads in each layer. Defaults to 12.
        dim (int, optional): The dimensionality of the encoder layers. Defaults to 768.
        hidden_dim (int, optional): The dimensionality of the hidden layers in the feed-forward network. Defaults to 4 * 768.
        dropout (float, optional): The dropout probability for all fully connected layers. Defaults to 0.1.
        attention_dropout (float, optional): The dropout probability for the attention layers. Defaults to 0.1.
        activation (str, optional): The activation function used in the feed-forward network. Defaults to 'gelu'.
        initializer_range (float, optional): The range of the initializer. Defaults to 0.02.
        qa_dropout (float, optional): The dropout probability for the question answering head. Defaults to 0.1.
        seq_classif_dropout (float, optional): The dropout probability for the sequence classification head. Defaults to 0.2.
        pad_token_id (int, optional): The id of the padding token. Defaults to 0.

    Returns:
        None

    Raises:
        None
    """
    self.vocab_size = vocab_size
    self.max_position_embeddings = max_position_embeddings
    self.sinusoidal_pos_embds = sinusoidal_pos_embds
    self.n_layers = n_layers
    self.n_heads = n_heads
    self.dim = dim
    self.hidden_dim = hidden_dim
    self.dropout = dropout
    self.attention_dropout = attention_dropout
    self.activation = activation
    self.initializer_range = initializer_range
    self.qa_dropout = qa_dropout
    self.seq_classif_dropout = seq_classif_dropout
    super().__init__(**kwargs, pad_token_id=pad_token_id)

mindnlp.transformers.models.distilbert.modeling_distilbert

MindSpore DistilBERT model

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForMaskedLM

Bases: DistilBertPreTrainedModel

A class representing a DistilBERT model for Masked Language Modeling (MLM).

This class inherits from the DistilBertPreTrainedModel class and includes methods for initializing the model, resizing position embeddings, getting and setting output embeddings, and forwarding the model.

ATTRIBUTE	DESCRIPTION
activation	Activation function used in the model. TYPE: function
distilbert	DistilBERT model. TYPE: DistilBertModel
vocab_transform	Dense layer for transforming the vocabulary. TYPE: Linear
vocab_layer_norm	Layer normalization for the vocabulary. TYPE: LayerNorm
vocab_projector	Dense layer for projecting the vocabulary. TYPE: Linear
mlm_loss_fct	Cross-entropy loss function for MLM. TYPE: CrossEntropyLoss

METHOD	DESCRIPTION
__init__	Initializes the DistilBertForMaskedLM model.
get_position_embeddings	Returns the position embeddings.
resize_position_embeddings	Resizes position embeddings of the model.
get_output_embeddings	Returns the output embeddings.
set_output_embeddings	Sets the output embeddings.
forward	Constructs the DistilBertForMaskedLM model.

Please see the documentation for the DistilBertPreTrainedModel class for more information on inherited attributes and methods.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

class DistilBertForMaskedLM(DistilBertPreTrainedModel):

    """
    A class representing a DistilBERT model for Masked Language Modeling (MLM).

    This class inherits from the DistilBertPreTrainedModel class and includes methods for initializing the model,
    resizing position embeddings, getting and setting output embeddings, and forwarding the model.

    Attributes:
        activation (function): Activation function used in the model.
        distilbert (DistilBertModel): DistilBERT model.
        vocab_transform (nn.Linear): Dense layer for transforming the vocabulary.
        vocab_layer_norm (nn.LayerNorm): Layer normalization for the vocabulary.
        vocab_projector (nn.Linear): Dense layer for projecting the vocabulary.
        mlm_loss_fct (nn.CrossEntropyLoss): Cross-entropy loss function for MLM.

    Methods:
        __init__: Initializes the DistilBertForMaskedLM model.
        get_position_embeddings: Returns the position embeddings.
        resize_position_embeddings: Resizes position embeddings of the model.
        get_output_embeddings: Returns the output embeddings.
        set_output_embeddings: Sets the output embeddings.
        forward: Constructs the DistilBertForMaskedLM model.

    Please see the documentation for the DistilBertPreTrainedModel class for more information on inherited attributes
    and methods.
    """
    _tied_weights_keys = ["vocab_projector.weight"]

    def __init__(self, config: PretrainedConfig):
        """
        Initializes a new instance of DistilBertForMaskedLM.

        Args:
            self: The object itself.
            config (PretrainedConfig): The configuration for the pretrained model.
                It contains the model's architecture and hyperparameters.

        Returns:
            None.

        Raises:
            TypeError: If the provided 'config' parameter is not of type PretrainedConfig.
            ValueError: If the configuration contains invalid values or is incompatible with the model.
        """
        super().__init__(config)

        self.activation = get_activation(config.activation)

        self.distilbert = DistilBertModel(config)
        self.vocab_transform = nn.Linear(config.dim, config.dim)
        self.vocab_layer_norm = nn.LayerNorm(config.dim, eps=1e-12)
        self.vocab_projector = nn.Linear(config.dim, config.vocab_size)

        # Initialize weights and apply final processing
        self.post_init()

        self.mlm_loss_fct = nn.CrossEntropyLoss()

    def get_position_embeddings(self) -> nn.Embedding:
        """
        Returns the position embeddings
        """
        return self.distilbert.get_position_embeddings()

    def resize_position_embeddings(self, new_num_position_embeddings: int):
        """
        Resizes position embeddings of the model if `new_num_position_embeddings != config.max_position_embeddings`.

        Arguments:
            new_num_position_embeddings (`int`):
                The number of new position embedding matrix. If position embeddings are learned, increasing the size
                will add newly initialized vectors at the end, whereas reducing the size will remove vectors from the
                end. If position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the
                size will add correct vectors at the end following the position encoding algorithm, whereas reducing
                the size will remove vectors from the end.
        """
        self.distilbert.resize_position_embeddings(new_num_position_embeddings)

    def get_output_embeddings(self) -> nn.Module:
        """
        Retrieves the output embeddings of the DistilBertForMaskedLM model.

        Args:
            self (DistilBertForMaskedLM): The instance of the DistilBertForMaskedLM class.

        Returns:
            nn.Module: The output embeddings of the DistilBertForMaskedLM model.
                The embeddings are projected using the vocab_projector.

        Raises:
            None.

        """
        return self.vocab_projector

    def set_output_embeddings(self, new_embeddings: nn.Module):
        """
        This method sets the output embeddings for the DistilBertForMaskedLM model.

        Args:
            self (DistilBertForMaskedLM): The instance of the DistilBertForMaskedLM class.
            new_embeddings (nn.Module): The new embeddings to set as the output embeddings for the model.
                It should be an instance of nn.Module representing the new embeddings to be used.

        Returns:
            None.

        Raises:
            None.
        """
        self.vocab_projector = new_embeddings

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        labels: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[MaskedLMOutput, Tuple[mindspore.Tensor, ...]]:
        r"""
        Args:
            labels (`mindspore.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
                Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ...,
                config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the
                loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        dlbrt_output = self.distilbert(
            input_ids=input_ids,
            attention_mask=attention_mask,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )
        hidden_states = dlbrt_output[0]  # (bs, seq_length, dim)
        prediction_logits = self.vocab_transform(hidden_states)  # (bs, seq_length, dim)
        prediction_logits = self.activation(prediction_logits)  # (bs, seq_length, dim)
        prediction_logits = self.vocab_layer_norm(prediction_logits)  # (bs, seq_length, dim)
        prediction_logits = self.vocab_projector(prediction_logits)  # (bs, seq_length, vocab_size)

        mlm_loss = None
        if labels is not None:
            mlm_loss = self.mlm_loss_fct(prediction_logits.view(-1, prediction_logits.shape[-1]), labels.view(-1))

        if not return_dict:
            output = (prediction_logits,) + dlbrt_output[1:]
            return ((mlm_loss,) + output) if mlm_loss is not None else output

        return MaskedLMOutput(
            loss=mlm_loss,
            logits=prediction_logits,
            hidden_states=dlbrt_output.hidden_states,
            attentions=dlbrt_output.attentions,
        )

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForMaskedLM.__init__(config)

Initializes a new instance of DistilBertForMaskedLM.

PARAMETER	DESCRIPTION
self	The object itself.
config	The configuration for the pretrained model. It contains the model's architecture and hyperparameters. TYPE: PretrainedConfig

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
TypeError	If the provided 'config' parameter is not of type PretrainedConfig.
ValueError	If the configuration contains invalid values or is incompatible with the model.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def __init__(self, config: PretrainedConfig):
    """
    Initializes a new instance of DistilBertForMaskedLM.

    Args:
        self: The object itself.
        config (PretrainedConfig): The configuration for the pretrained model.
            It contains the model's architecture and hyperparameters.

    Returns:
        None.

    Raises:
        TypeError: If the provided 'config' parameter is not of type PretrainedConfig.
        ValueError: If the configuration contains invalid values or is incompatible with the model.
    """
    super().__init__(config)

    self.activation = get_activation(config.activation)

    self.distilbert = DistilBertModel(config)
    self.vocab_transform = nn.Linear(config.dim, config.dim)
    self.vocab_layer_norm = nn.LayerNorm(config.dim, eps=1e-12)
    self.vocab_projector = nn.Linear(config.dim, config.vocab_size)

    # Initialize weights and apply final processing
    self.post_init()

    self.mlm_loss_fct = nn.CrossEntropyLoss()

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForMaskedLM.forward(input_ids=None, attention_mask=None, head_mask=None, inputs_embeds=None, labels=None, output_attentions=None, output_hidden_states=None, return_dict=None)

PARAMETER	DESCRIPTION
labels	Labels for computing the masked language modeling loss. Indices should be in [-100, 0, ..., config.vocab_size] (see input_ids docstring) Tokens with indices set to -100 are ignored (masked), the loss is only computed for the tokens with labels in [0, ..., config.vocab_size]. TYPE: `mindspore.Tensor` of shape `(batch_size, sequence_length)`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    inputs_embeds: Optional[mindspore.Tensor] = None,
    labels: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[MaskedLMOutput, Tuple[mindspore.Tensor, ...]]:
    r"""
    Args:
        labels (`mindspore.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
            Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ...,
            config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the
            loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    dlbrt_output = self.distilbert(
        input_ids=input_ids,
        attention_mask=attention_mask,
        head_mask=head_mask,
        inputs_embeds=inputs_embeds,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )
    hidden_states = dlbrt_output[0]  # (bs, seq_length, dim)
    prediction_logits = self.vocab_transform(hidden_states)  # (bs, seq_length, dim)
    prediction_logits = self.activation(prediction_logits)  # (bs, seq_length, dim)
    prediction_logits = self.vocab_layer_norm(prediction_logits)  # (bs, seq_length, dim)
    prediction_logits = self.vocab_projector(prediction_logits)  # (bs, seq_length, vocab_size)

    mlm_loss = None
    if labels is not None:
        mlm_loss = self.mlm_loss_fct(prediction_logits.view(-1, prediction_logits.shape[-1]), labels.view(-1))

    if not return_dict:
        output = (prediction_logits,) + dlbrt_output[1:]
        return ((mlm_loss,) + output) if mlm_loss is not None else output

    return MaskedLMOutput(
        loss=mlm_loss,
        logits=prediction_logits,
        hidden_states=dlbrt_output.hidden_states,
        attentions=dlbrt_output.attentions,
    )

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForMaskedLM.get_output_embeddings()

Retrieves the output embeddings of the DistilBertForMaskedLM model.

PARAMETER	DESCRIPTION
self	The instance of the DistilBertForMaskedLM class. TYPE: DistilBertForMaskedLM

RETURNS	DESCRIPTION
Module	nn.Module: The output embeddings of the DistilBertForMaskedLM model. The embeddings are projected using the vocab_projector.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def get_output_embeddings(self) -> nn.Module:
    """
    Retrieves the output embeddings of the DistilBertForMaskedLM model.

    Args:
        self (DistilBertForMaskedLM): The instance of the DistilBertForMaskedLM class.

    Returns:
        nn.Module: The output embeddings of the DistilBertForMaskedLM model.
            The embeddings are projected using the vocab_projector.

    Raises:
        None.

    """
    return self.vocab_projector

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForMaskedLM.get_position_embeddings()

Returns the position embeddings

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def get_position_embeddings(self) -> nn.Embedding:
    """
    Returns the position embeddings
    """
    return self.distilbert.get_position_embeddings()

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForMaskedLM.resize_position_embeddings(new_num_position_embeddings)

Resizes position embeddings of the model if new_num_position_embeddings != config.max_position_embeddings.

PARAMETER

DESCRIPTION

new_num_position_embeddings

The number of new position embedding matrix. If position embeddings are learned, increasing the size will add newly initialized vectors at the end, whereas reducing the size will remove vectors from the end. If position embeddings are not learned (e.g. sinusoidal position embeddings), increasing the size will add correct vectors at the end following the position encoding algorithm, whereas reducing the size will remove vectors from the end. TYPE: `int`

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def resize_position_embeddings(self, new_num_position_embeddings: int):
    """
    Resizes position embeddings of the model if `new_num_position_embeddings != config.max_position_embeddings`.

    Arguments:
        new_num_position_embeddings (`int`):
            The number of new position embedding matrix. If position embeddings are learned, increasing the size
            will add newly initialized vectors at the end, whereas reducing the size will remove vectors from the
            end. If position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the
            size will add correct vectors at the end following the position encoding algorithm, whereas reducing
            the size will remove vectors from the end.
    """
    self.distilbert.resize_position_embeddings(new_num_position_embeddings)

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForMaskedLM.set_output_embeddings(new_embeddings)

This method sets the output embeddings for the DistilBertForMaskedLM model.

PARAMETER	DESCRIPTION
self	The instance of the DistilBertForMaskedLM class. TYPE: DistilBertForMaskedLM
new_embeddings	The new embeddings to set as the output embeddings for the model. It should be an instance of nn.Module representing the new embeddings to be used. TYPE: Module

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def set_output_embeddings(self, new_embeddings: nn.Module):
    """
    This method sets the output embeddings for the DistilBertForMaskedLM model.

    Args:
        self (DistilBertForMaskedLM): The instance of the DistilBertForMaskedLM class.
        new_embeddings (nn.Module): The new embeddings to set as the output embeddings for the model.
            It should be an instance of nn.Module representing the new embeddings to be used.

    Returns:
        None.

    Raises:
        None.
    """
    self.vocab_projector = new_embeddings

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForMultipleChoice

Bases: DistilBertPreTrainedModel

DistilBertForMultipleChoice is a class that represents a DistilBERT model for multiple choice tasks. It is a subclass of DistilBertPreTrainedModel.

PARAMETER	DESCRIPTION
config	The configuration class that defines the model architecture and parameters. TYPE: PretrainedConfig

METHOD	DESCRIPTION
get_position_embeddings	Returns the position embeddings.
resize_position_embeddings	Resizes position embeddings of the model if new_num_position_embeddings != config.max_position_embeddings. Args: new_num_position_embeddings (int): The number of new position embeddings. If position embeddings are learned, increasing the size will add newly initialized vectors at the end, whereas reducing the size will remove vectors from the end. If position embeddings are not learned (e.g. sinusoidal position embeddings), increasing the size will add correct vectors at the end following the position encoding algorithm, whereas reducing the size will remove vectors from the end.
forward	This method is used to compute the outputs of the model for multiple choice tasks. Args: input_ids (Optional[mindspore.Tensor]): The input ids of shape (batch_size, num_choices, sequence_length) for multiple choice tasks. attention_mask (Optional[mindspore.Tensor]): The attention mask of shape (batch_size, num_choices, sequence_length). It is used to avoid performing attention on padding token indices. head_mask (Optional[mindspore.Tensor]): The head mask of shape (num_heads,). It is used to mask heads of the attention modules. inputs_embeds (Optional[mindspore.Tensor]): The embedded input of shape (batch_size, num_choices, sequence_length, hidden_size). It is an alternative to input_ids. labels (Optional[mindspore.Tensor]): The labels for computing the multiple choice classification loss. Indices should be in [0, ..., num_choices-1] where num_choices is the size of the second dimension of the input tensors. output_attentions (Optional[bool]): Whether to return attentions tensors or not. output_hidden_states (Optional[bool]): Whether to return hidden states tensors or not. return_dict (Optional[bool]): Whether to return a MultipleChoiceModelOutput instead of a tuple. Returns: If return_dict=True, a :class:~transformers.MultipleChoiceModelOutput containing various elements depending on the configuration (e.g., loss, logits, hidden_states, attentions), otherwise a tuple of objects as follows: logits (:obj:mindspore.Tensor of shape (batch_size, num_choices)): The logits for each choice. hidden_states (:obj:Tuple[mindspore.Tensor], optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True): Tuple of :obj:mindspore.Tensor of shape (batch_size, sequence_length, hidden_size). attentions (:obj:Tuple[mindspore.Tensor], optional, returned when output_attentions=True is passed or when config.output_attentions=True): Tuple of :obj:mindspore.Tensor of shape (batch_size, num_heads, sequence_length, sequence_length).

Example

>>> # importing the required libraries
>>> from transformers import AutoTokenizer, DistilBertForMultipleChoice
>>> import torch
...
>>> # loading the tokenizer and model
>>> tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased")
>>> model = DistilBertForMultipleChoice.from_pretrained("distilbert-base-cased")
...
>>> # input parameters
>>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
>>> choice0 = "It is eaten with a fork and a knife."
>>> choice1 = "It is eaten while held in the hand."
>>> labels = mindspore.Tensor(0).unsqueeze(0)  # choice0 is correct (according to Wikipedia ;)), batch size 1
...
>>> # encoding the prompts and choices
>>> encoding = tokenizer([[prompt, choice0], [prompt, choice1]], return_tensors="pt", padding=True)
>>> outputs = model(**{k: v.unsqueeze(0) for k, v in encoding.items()}, labels=labels)  # batch size is 1
...
>>> # calculating the loss and logits
>>> loss = outputs.loss
>>> logits = outputs.logits

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

class DistilBertForMultipleChoice(DistilBertPreTrainedModel):

    """
    DistilBertForMultipleChoice is a class that represents a DistilBERT model for multiple choice tasks.
    It is a subclass of DistilBertPreTrainedModel.

    Args:
        config (PretrainedConfig): The configuration class that defines the model architecture and parameters.

    Methods:
        get_position_embeddings:
            Returns the position embeddings.

        resize_position_embeddings:
            Resizes position embeddings of the model if `new_num_position_embeddings != config.max_position_embeddings`.

            Args:

            - new_num_position_embeddings (`int`): The number of new position embeddings.

                - If position embeddings are learned, increasing the size will add newly initialized vectors at the end,
                whereas reducing the size will remove vectors from the end.
                - If position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the size
                will add correct vectors at the end following the position encoding algorithm, whereas reducing the size
                will remove vectors from the end.

        forward:
            This method is used to compute the outputs of the model for multiple choice tasks.

            Args:

            - input_ids (Optional[mindspore.Tensor]): The input ids of shape `(batch_size, num_choices, sequence_length)`
            for multiple choice tasks.
            - attention_mask (Optional[mindspore.Tensor]):
            The attention mask of shape `(batch_size, num_choices, sequence_length)`. It is used to avoid performing
            attention on padding token indices.
            - head_mask (Optional[mindspore.Tensor]): The head mask of shape `(num_heads,)`.
            It is used to mask heads of the attention modules.
            - inputs_embeds (Optional[mindspore.Tensor]):
            The embedded input of shape `(batch_size, num_choices, sequence_length, hidden_size)`. It is an alternative
            to input_ids.
            - labels (Optional[mindspore.Tensor]): The labels for computing the multiple choice classification loss.
            Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of
            the input tensors.
            - output_attentions (Optional[bool]): Whether to return attentions tensors or not.
            - output_hidden_states (Optional[bool]): Whether to return hidden states tensors or not.
            - return_dict (Optional[bool]): Whether to return a `MultipleChoiceModelOutput` instead of a tuple.

            Returns:

            If ``return_dict=True``, a :class:`~transformers.MultipleChoiceModelOutput` containing various elements
            depending on the configuration (e.g., ``loss``, ``logits``, ``hidden_states``, ``attentions``),
            otherwise a tuple of objects as follows:

            - **logits** (:obj:`mindspore.Tensor` of shape `(batch_size, num_choices)`): The logits for each choice.
            - **hidden_states** (:obj:`Tuple[mindspore.Tensor]`, optional, returned when ``output_hidden_states=True``
            is passed or when ``config.output_hidden_states=True``): Tuple of :obj:`mindspore.Tensor` of shape
            `(batch_size, sequence_length, hidden_size)`.
            - **attentions** (:obj:`Tuple[mindspore.Tensor]`, optional, returned when ``output_attentions=True``
            is passed or when ``config.output_attentions=True``): Tuple of :obj:`mindspore.Tensor` of shape
            `(batch_size, num_heads, sequence_length, sequence_length)`.

    Example:
        ```python
        >>> # importing the required libraries
        >>> from transformers import AutoTokenizer, DistilBertForMultipleChoice
        >>> import torch
        ...
        >>> # loading the tokenizer and model
        >>> tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased")
        >>> model = DistilBertForMultipleChoice.from_pretrained("distilbert-base-cased")
        ...
        >>> # input parameters
        >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
        >>> choice0 = "It is eaten with a fork and a knife."
        >>> choice1 = "It is eaten while held in the hand."
        >>> labels = mindspore.Tensor(0).unsqueeze(0)  # choice0 is correct (according to Wikipedia ;)), batch size 1
        ...
        >>> # encoding the prompts and choices
        >>> encoding = tokenizer([[prompt, choice0], [prompt, choice1]], return_tensors="pt", padding=True)
        >>> outputs = model(**{k: v.unsqueeze(0) for k, v in encoding.items()}, labels=labels)  # batch size is 1
        ...
        >>> # calculating the loss and logits
        >>> loss = outputs.loss
        >>> logits = outputs.logits
        ```
    """
    def __init__(self, config: PretrainedConfig):
        """
        __init__

        Initializes a new instance of the DistilBertForMultipleChoice class.

        Args:
            self: The object instance.
            config (PretrainedConfig):
                An instance of PretrainedConfig class containing the configuration parameters for the model.

        Returns:
            None.

        Raises:
            TypeError: If the provided config parameter is not an instance of PretrainedConfig.
            ValueError: If the config parameter does not contain valid configuration parameters.
            RuntimeError: If an error occurs during the initialization process.
        """
        super().__init__(config)

        self.distilbert = DistilBertModel(config)
        self.pre_classifier = nn.Linear(config.dim, config.dim)
        self.classifier = nn.Linear(config.dim, 1)
        self.dropout = nn.Dropout(p=config.seq_classif_dropout)

        # Initialize weights and apply final processing
        self.post_init()

    def get_position_embeddings(self) -> nn.Embedding:
        """
        Returns the position embeddings
        """
        return self.distilbert.get_position_embeddings()

    def resize_position_embeddings(self, new_num_position_embeddings: int):
        """
        Resizes position embeddings of the model if `new_num_position_embeddings != config.max_position_embeddings`.

        Arguments:
            new_num_position_embeddings (`int`)
                The number of new position embeddings. If position embeddings are learned, increasing the size will add
                newly initialized vectors at the end, whereas reducing the size will remove vectors from the end. If
                position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the size will
                add correct vectors at the end following the position encoding algorithm, whereas reducing the size
                will remove vectors from the end.
        """
        self.distilbert.resize_position_embeddings(new_num_position_embeddings)

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        labels: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[MultipleChoiceModelOutput, Tuple[mindspore.Tensor, ...]]:
        r"""
        Args:
            labels (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
                Labels for computing the multiple choice classification loss. Indices should be in `[0, ...,
                num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See
                `input_ids` above)

        Returns:
            Union[MultipleChoiceModelOutput, Tuple[mindspore.Tensor, ...]]

        Example:
            ```python
            >>> from transformers import AutoTokenizer, DistilBertForMultipleChoice
            >>> import torch
            ...
            >>> tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased")
            >>> model = DistilBertForMultipleChoice.from_pretrained("distilbert-base-cased")
            ...
            >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
            >>> choice0 = "It is eaten with a fork and a knife."
            >>> choice1 = "It is eaten while held in the hand."
            >>> labels = mindspore.Tensor(0).unsqueeze(0)  # choice0 is correct (according to Wikipedia ;)), batch size 1
            ...
            >>> encoding = tokenizer([[prompt, choice0], [prompt, choice1]], return_tensors="pt", padding=True)
            >>> outputs = model(**{k: v.unsqueeze(0) for k, v in encoding.items()}, labels=labels)  # batch size is 1
            ...
            >>> # the linear classifier still needs to be trained
            >>> loss = outputs.loss
            >>> logits = outputs.logits
            ```
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
        num_choices = input_ids.shape[1] if input_ids is not None else inputs_embeds.shape[1]

        input_ids = input_ids.view(-1, input_ids.shape[-1]) if input_ids is not None else None
        attention_mask = attention_mask.view(-1, attention_mask.shape[-1]) if attention_mask is not None else None
        inputs_embeds = (
            inputs_embeds.view(-1, inputs_embeds.shape[-2], inputs_embeds.shape[-1])
            if inputs_embeds is not None
            else None
        )

        outputs = self.distilbert(
            input_ids,
            attention_mask=attention_mask,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        hidden_state = outputs[0]  # (bs * num_choices, seq_len, dim)
        pooled_output = hidden_state[:, 0]  # (bs * num_choices, dim)
        pooled_output = self.pre_classifier(pooled_output)  # (bs * num_choices, dim)
        pooled_output = nn.ReLU()(pooled_output)  # (bs * num_choices, dim)
        pooled_output = self.dropout(pooled_output)  # (bs * num_choices, dim)
        logits = self.classifier(pooled_output)  # (bs * num_choices, 1)

        reshaped_logits = logits.view(-1, num_choices)  # (bs, num_choices)

        loss = None
        if labels is not None:
            loss = ops.cross_entropy(reshaped_logits, labels)

        if not return_dict:
            output = (reshaped_logits,) + outputs[1:]
            return ((loss,) + output) if loss is not None else output

        return MultipleChoiceModelOutput(
            loss=loss,
            logits=reshaped_logits,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForMultipleChoice.__init__(config)

init

Initializes a new instance of the DistilBertForMultipleChoice class.

PARAMETER	DESCRIPTION
self	The object instance.
config	An instance of PretrainedConfig class containing the configuration parameters for the model. TYPE: PretrainedConfig

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
TypeError	If the provided config parameter is not an instance of PretrainedConfig.
ValueError	If the config parameter does not contain valid configuration parameters.
RuntimeError	If an error occurs during the initialization process.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def __init__(self, config: PretrainedConfig):
    """
    __init__

    Initializes a new instance of the DistilBertForMultipleChoice class.

    Args:
        self: The object instance.
        config (PretrainedConfig):
            An instance of PretrainedConfig class containing the configuration parameters for the model.

    Returns:
        None.

    Raises:
        TypeError: If the provided config parameter is not an instance of PretrainedConfig.
        ValueError: If the config parameter does not contain valid configuration parameters.
        RuntimeError: If an error occurs during the initialization process.
    """
    super().__init__(config)

    self.distilbert = DistilBertModel(config)
    self.pre_classifier = nn.Linear(config.dim, config.dim)
    self.classifier = nn.Linear(config.dim, 1)
    self.dropout = nn.Dropout(p=config.seq_classif_dropout)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForMultipleChoice.forward(input_ids=None, attention_mask=None, head_mask=None, inputs_embeds=None, labels=None, output_attentions=None, output_hidden_states=None, return_dict=None)

PARAMETER	DESCRIPTION
labels	Labels for computing the multiple choice classification loss. Indices should be in [0, ..., num_choices-1] where num_choices is the size of the second dimension of the input tensors. (See input_ids above) TYPE: `mindspore.Tensor` of shape `(batch_size,)`, *optional* DEFAULT: None

RETURNS	DESCRIPTION
Union[MultipleChoiceModelOutput, Tuple[Tensor, ...]]	Union[MultipleChoiceModelOutput, Tuple[mindspore.Tensor, ...]]

Example

>>> from transformers import AutoTokenizer, DistilBertForMultipleChoice
>>> import torch
...
>>> tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased")
>>> model = DistilBertForMultipleChoice.from_pretrained("distilbert-base-cased")
...
>>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
>>> choice0 = "It is eaten with a fork and a knife."
>>> choice1 = "It is eaten while held in the hand."
>>> labels = mindspore.Tensor(0).unsqueeze(0)  # choice0 is correct (according to Wikipedia ;)), batch size 1
...
>>> encoding = tokenizer([[prompt, choice0], [prompt, choice1]], return_tensors="pt", padding=True)
>>> outputs = model(**{k: v.unsqueeze(0) for k, v in encoding.items()}, labels=labels)  # batch size is 1
...
>>> # the linear classifier still needs to be trained
>>> loss = outputs.loss
>>> logits = outputs.logits

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    inputs_embeds: Optional[mindspore.Tensor] = None,
    labels: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[MultipleChoiceModelOutput, Tuple[mindspore.Tensor, ...]]:
    r"""
    Args:
        labels (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
            Labels for computing the multiple choice classification loss. Indices should be in `[0, ...,
            num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See
            `input_ids` above)

    Returns:
        Union[MultipleChoiceModelOutput, Tuple[mindspore.Tensor, ...]]

    Example:
        ```python
        >>> from transformers import AutoTokenizer, DistilBertForMultipleChoice
        >>> import torch
        ...
        >>> tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased")
        >>> model = DistilBertForMultipleChoice.from_pretrained("distilbert-base-cased")
        ...
        >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
        >>> choice0 = "It is eaten with a fork and a knife."
        >>> choice1 = "It is eaten while held in the hand."
        >>> labels = mindspore.Tensor(0).unsqueeze(0)  # choice0 is correct (according to Wikipedia ;)), batch size 1
        ...
        >>> encoding = tokenizer([[prompt, choice0], [prompt, choice1]], return_tensors="pt", padding=True)
        >>> outputs = model(**{k: v.unsqueeze(0) for k, v in encoding.items()}, labels=labels)  # batch size is 1
        ...
        >>> # the linear classifier still needs to be trained
        >>> loss = outputs.loss
        >>> logits = outputs.logits
        ```
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    num_choices = input_ids.shape[1] if input_ids is not None else inputs_embeds.shape[1]

    input_ids = input_ids.view(-1, input_ids.shape[-1]) if input_ids is not None else None
    attention_mask = attention_mask.view(-1, attention_mask.shape[-1]) if attention_mask is not None else None
    inputs_embeds = (
        inputs_embeds.view(-1, inputs_embeds.shape[-2], inputs_embeds.shape[-1])
        if inputs_embeds is not None
        else None
    )

    outputs = self.distilbert(
        input_ids,
        attention_mask=attention_mask,
        head_mask=head_mask,
        inputs_embeds=inputs_embeds,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    hidden_state = outputs[0]  # (bs * num_choices, seq_len, dim)
    pooled_output = hidden_state[:, 0]  # (bs * num_choices, dim)
    pooled_output = self.pre_classifier(pooled_output)  # (bs * num_choices, dim)
    pooled_output = nn.ReLU()(pooled_output)  # (bs * num_choices, dim)
    pooled_output = self.dropout(pooled_output)  # (bs * num_choices, dim)
    logits = self.classifier(pooled_output)  # (bs * num_choices, 1)

    reshaped_logits = logits.view(-1, num_choices)  # (bs, num_choices)

    loss = None
    if labels is not None:
        loss = ops.cross_entropy(reshaped_logits, labels)

    if not return_dict:
        output = (reshaped_logits,) + outputs[1:]
        return ((loss,) + output) if loss is not None else output

    return MultipleChoiceModelOutput(
        loss=loss,
        logits=reshaped_logits,
        hidden_states=outputs.hidden_states,
        attentions=outputs.attentions,
    )

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForMultipleChoice.get_position_embeddings()

Returns the position embeddings

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def get_position_embeddings(self) -> nn.Embedding:
    """
    Returns the position embeddings
    """
    return self.distilbert.get_position_embeddings()

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForMultipleChoice.resize_position_embeddings(new_num_position_embeddings)

Resizes position embeddings of the model if new_num_position_embeddings != config.max_position_embeddings.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def resize_position_embeddings(self, new_num_position_embeddings: int):
    """
    Resizes position embeddings of the model if `new_num_position_embeddings != config.max_position_embeddings`.

    Arguments:
        new_num_position_embeddings (`int`)
            The number of new position embeddings. If position embeddings are learned, increasing the size will add
            newly initialized vectors at the end, whereas reducing the size will remove vectors from the end. If
            position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the size will
            add correct vectors at the end following the position encoding algorithm, whereas reducing the size
            will remove vectors from the end.
    """
    self.distilbert.resize_position_embeddings(new_num_position_embeddings)

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForQuestionAnswering

Bases: DistilBertPreTrainedModel

DistilBertForQuestionAnswering is a class that represents a fine-tuned DistilBERT model for question answering tasks. It is designed to provide predictions for the start and end positions of the answer span given a question and a context.

Please note that this class assumes the existence of a DistilBERT model and a dense layer for question answering outputs (qa_outputs), which are initialized in the init method.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

class DistilBertForQuestionAnswering(DistilBertPreTrainedModel):

    """
    DistilBertForQuestionAnswering is a class that represents a fine-tuned DistilBERT model for question answering tasks.
    It is designed to provide predictions for the start and end positions of the answer span given a question and a context.

    Please note that this class assumes the existence of a DistilBERT model and a dense layer for question answering
    outputs (qa_outputs), which are initialized in the __init__ method.
    """
    def __init__(self, config: PretrainedConfig):
        """
        Initializes a new instance of the DistilBertForQuestionAnswering class.

        Args:
            self: The instance of the class.
            config (PretrainedConfig): The configuration object for the pretrained model.

        Returns:
            None.

        Raises:
            ValueError: If the number of labels in the configuration is not equal to 2, a ValueError is raised.
        """
        super().__init__(config)

        self.distilbert = DistilBertModel(config)
        self.qa_outputs = nn.Linear(config.dim, config.num_labels)
        if config.num_labels != 2:
            raise ValueError(f"config.num_labels should be 2, but it is {config.num_labels}")

        self.dropout = nn.Dropout(p=config.qa_dropout)

        # Initialize weights and apply final processing
        self.post_init()

    def get_position_embeddings(self) -> nn.Embedding:
        """
        Returns the position embeddings
        """
        return self.distilbert.get_position_embeddings()

    def resize_position_embeddings(self, new_num_position_embeddings: int):
        """
        Resizes position embeddings of the model if `new_num_position_embeddings != config.max_position_embeddings`.

        Arguments:
            new_num_position_embeddings (`int`):
                The number of new position embedding matrix.

                - If position embeddings are learned, increasing the size will add newly initialized vectors at the end,
                whereas reducing the size will remove vectors from the end.
                - If position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the
                size will add correct vectors at the end following the position encoding algorithm, whereas reducing
                the size will remove vectors from the end.
        """
        self.distilbert.resize_position_embeddings(new_num_position_embeddings)

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        start_positions: Optional[mindspore.Tensor] = None,
        end_positions: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[QuestionAnsweringModelOutput, Tuple[mindspore.Tensor, ...]]:
        r"""
        Args:
            start_positions (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
                Labels for position (index) of the start of the labelled span for computing the token classification loss.
                Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
                are not taken into account for computing the loss.
            end_positions (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
                Labels for position (index) of the end of the labelled span for computing the token classification loss.
                Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
                are not taken into account for computing the loss.
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        distilbert_output = self.distilbert(
            input_ids=input_ids,
            attention_mask=attention_mask,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )
        hidden_states = distilbert_output[0]  # (bs, max_query_len, dim)

        hidden_states = self.dropout(hidden_states)  # (bs, max_query_len, dim)
        logits = self.qa_outputs(hidden_states)  # (bs, max_query_len, 2)
        start_logits, end_logits = logits.split(1, axis=-1)
        start_logits = start_logits.squeeze(-1)  # (bs, max_query_len)
        end_logits = end_logits.squeeze(-1)  # (bs, max_query_len)

        total_loss = None
        if start_positions is not None and end_positions is not None:
            # If we are on multi-GPU, split add a dimension
            if len(start_positions.shape) > 1:
                start_positions = start_positions.squeeze(-1)
            if len(end_positions.shape) > 1:
                end_positions = end_positions.squeeze(-1)
            # sometimes the start/end positions are outside our model inputs, we ignore these terms
            ignored_index = start_logits.shape[1]
            start_positions = start_positions.clamp(0, ignored_index)
            end_positions = end_positions.clamp(0, ignored_index)

            loss_fct = nn.CrossEntropyLoss(ignore_index=ignored_index)
            start_loss = loss_fct(start_logits, start_positions)
            end_loss = loss_fct(end_logits, end_positions)
            total_loss = (start_loss + end_loss) / 2

        if not return_dict:
            output = (start_logits, end_logits) + distilbert_output[1:]
            return ((total_loss,) + output) if total_loss is not None else output

        return QuestionAnsweringModelOutput(
            loss=total_loss,
            start_logits=start_logits,
            end_logits=end_logits,
            hidden_states=distilbert_output.hidden_states,
            attentions=distilbert_output.attentions,
        )

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForQuestionAnswering.__init__(config)

Initializes a new instance of the DistilBertForQuestionAnswering class.

PARAMETER	DESCRIPTION
self	The instance of the class.
config	The configuration object for the pretrained model. TYPE: PretrainedConfig

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If the number of labels in the configuration is not equal to 2, a ValueError is raised.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def __init__(self, config: PretrainedConfig):
    """
    Initializes a new instance of the DistilBertForQuestionAnswering class.

    Args:
        self: The instance of the class.
        config (PretrainedConfig): The configuration object for the pretrained model.

    Returns:
        None.

    Raises:
        ValueError: If the number of labels in the configuration is not equal to 2, a ValueError is raised.
    """
    super().__init__(config)

    self.distilbert = DistilBertModel(config)
    self.qa_outputs = nn.Linear(config.dim, config.num_labels)
    if config.num_labels != 2:
        raise ValueError(f"config.num_labels should be 2, but it is {config.num_labels}")

    self.dropout = nn.Dropout(p=config.qa_dropout)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForQuestionAnswering.forward(input_ids=None, attention_mask=None, head_mask=None, inputs_embeds=None, start_positions=None, end_positions=None, output_attentions=None, output_hidden_states=None, return_dict=None)

PARAMETER	DESCRIPTION
start_positions	Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence are not taken into account for computing the loss. TYPE: `mindspore.Tensor` of shape `(batch_size,)`, *optional* DEFAULT: None
end_positions	Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence are not taken into account for computing the loss. TYPE: `mindspore.Tensor` of shape `(batch_size,)`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    inputs_embeds: Optional[mindspore.Tensor] = None,
    start_positions: Optional[mindspore.Tensor] = None,
    end_positions: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[QuestionAnsweringModelOutput, Tuple[mindspore.Tensor, ...]]:
    r"""
    Args:
        start_positions (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
            Labels for position (index) of the start of the labelled span for computing the token classification loss.
            Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
            are not taken into account for computing the loss.
        end_positions (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
            Labels for position (index) of the end of the labelled span for computing the token classification loss.
            Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
            are not taken into account for computing the loss.
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    distilbert_output = self.distilbert(
        input_ids=input_ids,
        attention_mask=attention_mask,
        head_mask=head_mask,
        inputs_embeds=inputs_embeds,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )
    hidden_states = distilbert_output[0]  # (bs, max_query_len, dim)

    hidden_states = self.dropout(hidden_states)  # (bs, max_query_len, dim)
    logits = self.qa_outputs(hidden_states)  # (bs, max_query_len, 2)
    start_logits, end_logits = logits.split(1, axis=-1)
    start_logits = start_logits.squeeze(-1)  # (bs, max_query_len)
    end_logits = end_logits.squeeze(-1)  # (bs, max_query_len)

    total_loss = None
    if start_positions is not None and end_positions is not None:
        # If we are on multi-GPU, split add a dimension
        if len(start_positions.shape) > 1:
            start_positions = start_positions.squeeze(-1)
        if len(end_positions.shape) > 1:
            end_positions = end_positions.squeeze(-1)
        # sometimes the start/end positions are outside our model inputs, we ignore these terms
        ignored_index = start_logits.shape[1]
        start_positions = start_positions.clamp(0, ignored_index)
        end_positions = end_positions.clamp(0, ignored_index)

        loss_fct = nn.CrossEntropyLoss(ignore_index=ignored_index)
        start_loss = loss_fct(start_logits, start_positions)
        end_loss = loss_fct(end_logits, end_positions)
        total_loss = (start_loss + end_loss) / 2

    if not return_dict:
        output = (start_logits, end_logits) + distilbert_output[1:]
        return ((total_loss,) + output) if total_loss is not None else output

    return QuestionAnsweringModelOutput(
        loss=total_loss,
        start_logits=start_logits,
        end_logits=end_logits,
        hidden_states=distilbert_output.hidden_states,
        attentions=distilbert_output.attentions,
    )

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForQuestionAnswering.get_position_embeddings()

Returns the position embeddings

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def get_position_embeddings(self) -> nn.Embedding:
    """
    Returns the position embeddings
    """
    return self.distilbert.get_position_embeddings()

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForQuestionAnswering.resize_position_embeddings(new_num_position_embeddings)

Resizes position embeddings of the model if new_num_position_embeddings != config.max_position_embeddings.

PARAMETER

DESCRIPTION

new_num_position_embeddings

The number of new position embedding matrix. If position embeddings are learned, increasing the size will add newly initialized vectors at the end, whereas reducing the size will remove vectors from the end. If position embeddings are not learned (e.g. sinusoidal position embeddings), increasing the size will add correct vectors at the end following the position encoding algorithm, whereas reducing the size will remove vectors from the end. TYPE: `int`

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def resize_position_embeddings(self, new_num_position_embeddings: int):
    """
    Resizes position embeddings of the model if `new_num_position_embeddings != config.max_position_embeddings`.

    Arguments:
        new_num_position_embeddings (`int`):
            The number of new position embedding matrix.

            - If position embeddings are learned, increasing the size will add newly initialized vectors at the end,
            whereas reducing the size will remove vectors from the end.
            - If position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the
            size will add correct vectors at the end following the position encoding algorithm, whereas reducing
            the size will remove vectors from the end.
    """
    self.distilbert.resize_position_embeddings(new_num_position_embeddings)

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForSequenceClassification

Bases: DistilBertPreTrainedModel

DistilBertForSequenceClassification is a class for sequence classification tasks based on the DistilBert architecture. This class inherits from the DistilBertPreTrainedModel class and provides a sequence classification layer on top of the DistilBert encoder.

ATTRIBUTE	DESCRIPTION
num_labels	The number of labels for the sequence classification task. TYPE: int
config	The configuration object for the model. TYPE: PretrainedConfig
distilbert	The DistilBert encoder model. TYPE: DistilBertModel
pre_classifier	A fully connected layer for the classification task. TYPE: Linear
classifier	A fully connected layer for the classification task. TYPE: Linear
dropout	A dropout layer to prevent overfitting. TYPE: Dropout

METHOD	DESCRIPTION
get_position_embeddings	Returns the position embeddings.
resize_position_embeddings	Resizes the position embeddings of the model.
forward	Constructs the model for sequence classification. Args: input_ids (Optional[mindspore.Tensor]): The input sequence tensor of shape (batch_size, sequence_length). attention_mask (Optional[mindspore.Tensor]): The attention mask tensor of shape (batch_size, sequence_length). head_mask (Optional[mindspore.Tensor]): The head mask tensor of shape (num_heads,). inputs_embeds (Optional[mindspore.Tensor]): The input embeddings tensor of shape (batch_size, sequence_length, hidden_size). labels (Optional[mindspore.Tensor]): The labels tensor of shape (batch_size,). output_attentions (Optional[bool]): Whether to return the attentions tensor or not. output_hidden_states (Optional[bool]): Whether to return the hidden states tensor or not. return_dict (Optional[bool]): Whether to return a dictionary of outputs or a tuple. Returns: If return_dict is False, returns a tuple of (loss, logits, hidden_states, attentions). If return_dict is True, returns a dictionary of outputs with keys loss, logits, hidden_states, and attentions.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

class DistilBertForSequenceClassification(DistilBertPreTrainedModel):

    """
    DistilBertForSequenceClassification is a class for sequence classification tasks based on the DistilBert architecture.
    This class inherits from the DistilBertPreTrainedModel class and provides a sequence classification layer on top of
    the DistilBert encoder.

    Attributes:
        num_labels (int): The number of labels for the sequence classification task.
        config (PretrainedConfig): The configuration object for the model.
        distilbert (DistilBertModel): The DistilBert encoder model.
        pre_classifier (nn.Linear): A fully connected layer for the classification task.
        classifier (nn.Linear): A fully connected layer for the classification task.
        dropout (nn.Dropout): A dropout layer to prevent overfitting.

    Methods:
        get_position_embeddings: Returns the position embeddings.
        resize_position_embeddings: Resizes the position embeddings of the model.
        forward:
            Constructs the model for sequence classification.

            Args:

            - input_ids (Optional[mindspore.Tensor]): The input sequence tensor of shape `(batch_size, sequence_length)`.
            - attention_mask (Optional[mindspore.Tensor]): The attention mask tensor of shape `(batch_size, sequence_length)`.
            - head_mask (Optional[mindspore.Tensor]): The head mask tensor of shape `(num_heads,)`.
            - inputs_embeds (Optional[mindspore.Tensor]): The input embeddings tensor of shape `(batch_size, sequence_length, hidden_size)`.
            - labels (Optional[mindspore.Tensor]): The labels tensor of shape `(batch_size,)`.
            - output_attentions (Optional[bool]): Whether to return the attentions tensor or not.
            - output_hidden_states (Optional[bool]): Whether to return the hidden states tensor or not.
            - return_dict (Optional[bool]): Whether to return a dictionary of outputs or a tuple.

            Returns:

            - If `return_dict` is `False`, returns a tuple of `(loss, logits, hidden_states, attentions)`.
            - If `return_dict` is `True`, returns a dictionary of outputs with keys `loss`, `logits`, `hidden_states`, and `attentions`.
    """
    def __init__(self, config: PretrainedConfig):
        """Initialize a DistilBertForSequenceClassification model.

        Args:
            self: The object instance itself.
            config (PretrainedConfig): The configuration object containing various parameters for the model.
                It specifies the model architecture, hyperparameters, and other settings.
                Must be an instance of PretrainedConfig.

        Returns:
            None.

        Raises:
            TypeError: If the provided config parameter is not an instance of PretrainedConfig.
            ValueError: If any of the required attributes in the config object are missing or invalid.
            RuntimeError: If there are issues during model initialization or attribute assignment.
        """
        super().__init__(config)
        self.num_labels = config.num_labels
        self.config = config

        self.distilbert = DistilBertModel(config)
        self.pre_classifier = nn.Linear(config.dim, config.dim)
        self.classifier = nn.Linear(config.dim, config.num_labels)
        self.dropout = nn.Dropout(p=config.seq_classif_dropout)

        # Initialize weights and apply final processing
        self.post_init()

    def get_position_embeddings(self) -> nn.Embedding:
        """
        Returns the position embeddings
        """
        return self.distilbert.get_position_embeddings()

    def resize_position_embeddings(self, new_num_position_embeddings: int):
        """
        Resizes position embeddings of the model if `new_num_position_embeddings != config.max_position_embeddings`.

        Arguments:
            new_num_position_embeddings (`int`):
                The number of new position embedding matrix.

                - If position embeddings are learned, increasing the size will add newly initialized vectors at the end,
                whereas reducing the size will remove vectors from the end.
                - If position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the
                size will add correct vectors at the end following the position encoding algorithm, whereas reducing
                the size will remove vectors from the end.
        """
        self.distilbert.resize_position_embeddings(new_num_position_embeddings)

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        labels: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[SequenceClassifierOutput, Tuple[mindspore.Tensor, ...]]:
        r"""
        Args:
            labels (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
                Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
                config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
                `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        distilbert_output = self.distilbert(
            input_ids=input_ids,
            attention_mask=attention_mask,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )
        hidden_state = distilbert_output[0]  # (bs, seq_len, dim)
        pooled_output = hidden_state[:, 0]  # (bs, dim)
        pooled_output = self.pre_classifier(pooled_output)  # (bs, dim)
        pooled_output = nn.ReLU()(pooled_output)  # (bs, dim)
        pooled_output = self.dropout(pooled_output)  # (bs, dim)
        logits = self.classifier(pooled_output)  # (bs, num_labels)

        loss = None
        if labels is not None:
            if self.config.problem_type is None:
                if self.num_labels == 1:
                    self.config.problem_type = "regression"
                elif self.num_labels > 1 and labels.dtype in (mindspore.int64, mindspore.int32):
                    self.config.problem_type = "single_label_classification"
                else:
                    self.config.problem_type = "multi_label_classification"

            if self.config.problem_type == "regression":
                if self.num_labels == 1:
                    loss = ops.mse_loss(logits.squeeze(), labels.squeeze())
                else:
                    loss = ops.mse_loss(logits, labels)
            elif self.config.problem_type == "single_label_classification":
                loss = ops.cross_entropy(logits.view(-1, self.num_labels), labels.view(-1))
            elif self.config.problem_type == "multi_label_classification":
                loss = ops.binary_cross_entropy_with_logits(logits, labels)

        if not return_dict:
            output = (logits,) + distilbert_output[1:]
            return ((loss,) + output) if loss is not None else output

        return SequenceClassifierOutput(
            loss=loss,
            logits=logits,
            hidden_states=distilbert_output.hidden_states,
            attentions=distilbert_output.attentions,
        )

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForSequenceClassification.__init__(config)

Initialize a DistilBertForSequenceClassification model.

PARAMETER	DESCRIPTION
self	The object instance itself.
config	The configuration object containing various parameters for the model. It specifies the model architecture, hyperparameters, and other settings. Must be an instance of PretrainedConfig. TYPE: PretrainedConfig

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
TypeError	If the provided config parameter is not an instance of PretrainedConfig.
ValueError	If any of the required attributes in the config object are missing or invalid.
RuntimeError	If there are issues during model initialization or attribute assignment.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def __init__(self, config: PretrainedConfig):
    """Initialize a DistilBertForSequenceClassification model.

    Args:
        self: The object instance itself.
        config (PretrainedConfig): The configuration object containing various parameters for the model.
            It specifies the model architecture, hyperparameters, and other settings.
            Must be an instance of PretrainedConfig.

    Returns:
        None.

    Raises:
        TypeError: If the provided config parameter is not an instance of PretrainedConfig.
        ValueError: If any of the required attributes in the config object are missing or invalid.
        RuntimeError: If there are issues during model initialization or attribute assignment.
    """
    super().__init__(config)
    self.num_labels = config.num_labels
    self.config = config

    self.distilbert = DistilBertModel(config)
    self.pre_classifier = nn.Linear(config.dim, config.dim)
    self.classifier = nn.Linear(config.dim, config.num_labels)
    self.dropout = nn.Dropout(p=config.seq_classif_dropout)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForSequenceClassification.forward(input_ids=None, attention_mask=None, head_mask=None, inputs_embeds=None, labels=None, output_attentions=None, output_hidden_states=None, return_dict=None)

PARAMETER	DESCRIPTION
labels	Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If config.num_labels > 1 a classification loss is computed (Cross-Entropy). TYPE: `mindspore.Tensor` of shape `(batch_size,)`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    inputs_embeds: Optional[mindspore.Tensor] = None,
    labels: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[SequenceClassifierOutput, Tuple[mindspore.Tensor, ...]]:
    r"""
    Args:
        labels (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
            Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
            config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
            `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    distilbert_output = self.distilbert(
        input_ids=input_ids,
        attention_mask=attention_mask,
        head_mask=head_mask,
        inputs_embeds=inputs_embeds,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )
    hidden_state = distilbert_output[0]  # (bs, seq_len, dim)
    pooled_output = hidden_state[:, 0]  # (bs, dim)
    pooled_output = self.pre_classifier(pooled_output)  # (bs, dim)
    pooled_output = nn.ReLU()(pooled_output)  # (bs, dim)
    pooled_output = self.dropout(pooled_output)  # (bs, dim)
    logits = self.classifier(pooled_output)  # (bs, num_labels)

    loss = None
    if labels is not None:
        if self.config.problem_type is None:
            if self.num_labels == 1:
                self.config.problem_type = "regression"
            elif self.num_labels > 1 and labels.dtype in (mindspore.int64, mindspore.int32):
                self.config.problem_type = "single_label_classification"
            else:
                self.config.problem_type = "multi_label_classification"

        if self.config.problem_type == "regression":
            if self.num_labels == 1:
                loss = ops.mse_loss(logits.squeeze(), labels.squeeze())
            else:
                loss = ops.mse_loss(logits, labels)
        elif self.config.problem_type == "single_label_classification":
            loss = ops.cross_entropy(logits.view(-1, self.num_labels), labels.view(-1))
        elif self.config.problem_type == "multi_label_classification":
            loss = ops.binary_cross_entropy_with_logits(logits, labels)

    if not return_dict:
        output = (logits,) + distilbert_output[1:]
        return ((loss,) + output) if loss is not None else output

    return SequenceClassifierOutput(
        loss=loss,
        logits=logits,
        hidden_states=distilbert_output.hidden_states,
        attentions=distilbert_output.attentions,
    )

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForSequenceClassification.get_position_embeddings()

Returns the position embeddings

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def get_position_embeddings(self) -> nn.Embedding:
    """
    Returns the position embeddings
    """
    return self.distilbert.get_position_embeddings()

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForSequenceClassification.resize_position_embeddings(new_num_position_embeddings)

Resizes position embeddings of the model if new_num_position_embeddings != config.max_position_embeddings.

PARAMETER

DESCRIPTION

new_num_position_embeddings

The number of new position embedding matrix. If position embeddings are learned, increasing the size will add newly initialized vectors at the end, whereas reducing the size will remove vectors from the end. If position embeddings are not learned (e.g. sinusoidal position embeddings), increasing the size will add correct vectors at the end following the position encoding algorithm, whereas reducing the size will remove vectors from the end. TYPE: `int`

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def resize_position_embeddings(self, new_num_position_embeddings: int):
    """
    Resizes position embeddings of the model if `new_num_position_embeddings != config.max_position_embeddings`.

    Arguments:
        new_num_position_embeddings (`int`):
            The number of new position embedding matrix.

            - If position embeddings are learned, increasing the size will add newly initialized vectors at the end,
            whereas reducing the size will remove vectors from the end.
            - If position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the
            size will add correct vectors at the end following the position encoding algorithm, whereas reducing
            the size will remove vectors from the end.
    """
    self.distilbert.resize_position_embeddings(new_num_position_embeddings)

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForTokenClassification

Bases: DistilBertPreTrainedModel

Represents a DistilBERT model for token classification.

This class inherits from DistilBertPreTrainedModel and implements methods for initializing the model, retrieving position embeddings, resizing position embeddings, and forwarding the model for token classification tasks.

ATTRIBUTE	DESCRIPTION
config	The configuration for the pretrained model. TYPE: PretrainedConfig

METHOD	DESCRIPTION
__init__	Initializes the DistilBertForTokenClassification model.
get_position_embeddings	Returns the position embeddings.
resize_position_embeddings	Resizes position embeddings of the model.
forward	Constructs the model for token classification tasks.

RAISES	DESCRIPTION
NotImplementedError	If the method is not implemented.

Note

This class is intended to be subclassed when implementing a custom DistilBERT model for token classification tasks.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

class DistilBertForTokenClassification(DistilBertPreTrainedModel):

    """
    Represents a DistilBERT model for token classification.

    This class inherits from DistilBertPreTrainedModel and implements methods for initializing the model,
    retrieving position embeddings, resizing position embeddings, and forwarding the model for token
    classification tasks.

    Attributes:
        config (PretrainedConfig): The configuration for the pretrained model.

    Methods:
        __init__: Initializes the DistilBertForTokenClassification model.
        get_position_embeddings: Returns the position embeddings.
        resize_position_embeddings: Resizes position embeddings of the model.
        forward: Constructs the model for token classification tasks.

    Raises:
        NotImplementedError: If the method is not implemented.

    Note:
        This class is intended to be subclassed when implementing a custom DistilBERT model for token classification tasks.
    """
    def __init__(self, config: PretrainedConfig):
        """
        Initializes a new instance of the `DistilBertForTokenClassification` class.

        Args:
            self: The object itself.
            config (PretrainedConfig): The configuration for the model.

        Returns:
            None

        Raises:
            None
        """
        super().__init__(config)
        self.num_labels = config.num_labels

        self.distilbert = DistilBertModel(config)
        self.dropout = nn.Dropout(p=config.dropout)
        self.classifier = nn.Linear(config.hidden_size, config.num_labels)

        # Initialize weights and apply final processing
        self.post_init()

    def get_position_embeddings(self) -> nn.Embedding:
        """
        Returns the position embeddings
        """
        return self.distilbert.get_position_embeddings()

    def resize_position_embeddings(self, new_num_position_embeddings: int):
        """
        Resizes position embeddings of the model if `new_num_position_embeddings != config.max_position_embeddings`.

        Arguments:
            new_num_position_embeddings (`int`):
                The number of new position embedding matrix.

                - If position embeddings are learned, increasing the size will add newly initialized vectors at the end,
                whereas reducing the size will remove vectors from the end.
                - If position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the
                size will add correct vectors at the end following the position encoding algorithm, whereas reducing
                the size will remove vectors from the end.
        """
        self.distilbert.resize_position_embeddings(new_num_position_embeddings)

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        labels: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[TokenClassifierOutput, Tuple[mindspore.Tensor, ...]]:
        r"""
        Args:
            labels (`mindspore.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
                Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`.
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        outputs = self.distilbert(
            input_ids,
            attention_mask=attention_mask,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        sequence_output = outputs[0]

        sequence_output = self.dropout(sequence_output)
        logits = self.classifier(sequence_output)

        loss = None
        if labels is not None:
            loss = ops.cross_entropy(logits.view(-1, self.num_labels), labels.view(-1))

        if not return_dict:
            output = (logits,) + outputs[1:]
            return ((loss,) + output) if loss is not None else output

        return TokenClassifierOutput(
            loss=loss,
            logits=logits,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForTokenClassification.__init__(config)

Initializes a new instance of the DistilBertForTokenClassification class.

PARAMETER	DESCRIPTION
self	The object itself.
config	The configuration for the model. TYPE: PretrainedConfig

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def __init__(self, config: PretrainedConfig):
    """
    Initializes a new instance of the `DistilBertForTokenClassification` class.

    Args:
        self: The object itself.
        config (PretrainedConfig): The configuration for the model.

    Returns:
        None

    Raises:
        None
    """
    super().__init__(config)
    self.num_labels = config.num_labels

    self.distilbert = DistilBertModel(config)
    self.dropout = nn.Dropout(p=config.dropout)
    self.classifier = nn.Linear(config.hidden_size, config.num_labels)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForTokenClassification.forward(input_ids=None, attention_mask=None, head_mask=None, inputs_embeds=None, labels=None, output_attentions=None, output_hidden_states=None, return_dict=None)

PARAMETER	DESCRIPTION
labels	Labels for computing the token classification loss. Indices should be in [0, ..., config.num_labels - 1]. TYPE: `mindspore.Tensor` of shape `(batch_size, sequence_length)`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    inputs_embeds: Optional[mindspore.Tensor] = None,
    labels: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[TokenClassifierOutput, Tuple[mindspore.Tensor, ...]]:
    r"""
    Args:
        labels (`mindspore.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
            Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`.
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    outputs = self.distilbert(
        input_ids,
        attention_mask=attention_mask,
        head_mask=head_mask,
        inputs_embeds=inputs_embeds,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    sequence_output = outputs[0]

    sequence_output = self.dropout(sequence_output)
    logits = self.classifier(sequence_output)

    loss = None
    if labels is not None:
        loss = ops.cross_entropy(logits.view(-1, self.num_labels), labels.view(-1))

    if not return_dict:
        output = (logits,) + outputs[1:]
        return ((loss,) + output) if loss is not None else output

    return TokenClassifierOutput(
        loss=loss,
        logits=logits,
        hidden_states=outputs.hidden_states,
        attentions=outputs.attentions,
    )

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForTokenClassification.get_position_embeddings()

Returns the position embeddings

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def get_position_embeddings(self) -> nn.Embedding:
    """
    Returns the position embeddings
    """
    return self.distilbert.get_position_embeddings()

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertForTokenClassification.resize_position_embeddings(new_num_position_embeddings)

Resizes position embeddings of the model if new_num_position_embeddings != config.max_position_embeddings.

PARAMETER

DESCRIPTION

new_num_position_embeddings

The number of new position embedding matrix. If position embeddings are learned, increasing the size will add newly initialized vectors at the end, whereas reducing the size will remove vectors from the end. If position embeddings are not learned (e.g. sinusoidal position embeddings), increasing the size will add correct vectors at the end following the position encoding algorithm, whereas reducing the size will remove vectors from the end. TYPE: `int`

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def resize_position_embeddings(self, new_num_position_embeddings: int):
    """
    Resizes position embeddings of the model if `new_num_position_embeddings != config.max_position_embeddings`.

    Arguments:
        new_num_position_embeddings (`int`):
            The number of new position embedding matrix.

            - If position embeddings are learned, increasing the size will add newly initialized vectors at the end,
            whereas reducing the size will remove vectors from the end.
            - If position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the
            size will add correct vectors at the end following the position encoding algorithm, whereas reducing
            the size will remove vectors from the end.
    """
    self.distilbert.resize_position_embeddings(new_num_position_embeddings)

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertModel

Bases: DistilBertPreTrainedModel

Represents a DistilBERT model for natural language processing tasks.

This class inherits from DistilBertPreTrainedModel and implements various methods for handling position embeddings, input embeddings, head pruning, and model forwardion. The class provides functionality to initialize the model, resize position embeddings, get position embeddings, resize input embeddings, set input embeddings, prune heads, and forward the model for inference.

METHOD	DESCRIPTION
get_position_embeddings	Returns the position embeddings.
resize_position_embeddings	Resizes position embeddings of the model.
get_input_embeddings	Returns the input embeddings.
set_input_embeddings	Sets new input embeddings.
_prune_heads	Prunes heads of the model based on the provided dictionary.
forward	Constructs the model for inference with optional parameters.

Note

This class assumes familiarity with the DistilBERT model architecture and its specific components.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

class DistilBertModel(DistilBertPreTrainedModel):

    """
    Represents a DistilBERT model for natural language processing tasks.

    This class inherits from DistilBertPreTrainedModel and implements various methods for handling position embeddings,
    input embeddings, head pruning, and model forwardion.
    The class provides functionality to initialize the model, resize position embeddings, get position embeddings,
    resize input embeddings, set input embeddings, prune heads, and forward the model for inference.

    Methods:
        get_position_embeddings: Returns the position embeddings.
        resize_position_embeddings: Resizes position embeddings of the model.
        get_input_embeddings: Returns the input embeddings.
        set_input_embeddings: Sets new input embeddings.
        _prune_heads: Prunes heads of the model based on the provided dictionary.
        forward: Constructs the model for inference with optional parameters.

    Note:
        This class assumes familiarity with the DistilBERT model architecture and its specific components.
    """
    def __init__(self, config: PretrainedConfig):
        """
        Initializes a new instance of the DistilBertModel class.

        Args:
            self: The instance of the DistilBertModel class.
            config (PretrainedConfig):
                An instance of the PretrainedConfig class containing the configuration settings for the model.
                This parameter is required to configure the model's embeddings and transformer components.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(config)

        self.embeddings = Embeddings(config)  # Embeddings
        self.transformer = Transformer(config)  # Encoder

        # Initialize weights and apply final processing
        self.post_init()

    def get_position_embeddings(self) -> nn.Embedding:
        """
        Returns the position embeddings
        """
        return self.embeddings.position_embeddings

    def resize_position_embeddings(self, new_num_position_embeddings: int):
        """
        Resizes position embeddings of the model if `new_num_position_embeddings != config.max_position_embeddings`.

        Arguments:
            new_num_position_embeddings (`int`):
                The number of new position embedding matrix.

                - If position embeddings are learned, increasing the size will add newly initialized vectors at the end,
                whereas reducing the size will remove vectors from the end.
                - If position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the
                size will add correct vectors at the end following the position encoding algorithm, whereas reducing
                the size will remove vectors from the end.
        """
        num_position_embeds_diff = new_num_position_embeddings - self.config.max_position_embeddings

        # no resizing needs to be done if the length stays the same
        if num_position_embeds_diff == 0:
            return

        logger.info(f"Setting `config.max_position_embeddings={new_num_position_embeddings}`...")
        self.config.max_position_embeddings = new_num_position_embeddings

        old_position_embeddings_weight = self.embeddings.position_embeddings.weight.clone()

        self.embeddings.position_embeddings = nn.Embedding(self.config.max_position_embeddings, self.config.dim)

        if self.config.sinusoidal_pos_embds:
            create_sinusoidal_embeddings(
                n_pos=self.config.max_position_embeddings, dim=self.config.dim, out=self.position_embeddings.weight
            )
        else:
            if num_position_embeds_diff > 0:
                self.embeddings.position_embeddings.weight[:-num_position_embeds_diff] = nn.Parameter(
                    old_position_embeddings_weight
                )
            else:
                self.embeddings.position_embeddings.weight = nn.Parameter(
                    old_position_embeddings_weight[:num_position_embeds_diff]
                )

    def get_input_embeddings(self) -> nn.Embedding:
        """
        Retrieve the input embeddings for the DistilBertModel.

        Args:
            self (DistilBertModel): An instance of the DistilBertModel class.
                Represents the current instance of the DistilBertModel.
                This parameter is required for accessing the embeddings.

        Returns:
            nn.Embedding: An instance of the nn.Embedding class.
                Represents the word embeddings used for input to the DistilBertModel.
                These embeddings are used to convert input tokens to dense vectors for processing.

        Raises:
            None.
        """
        return self.embeddings.word_embeddings

    def set_input_embeddings(self, new_embeddings: nn.Embedding):
        """
        Sets the input embeddings for the DistilBertModel.

        Args:
            self (DistilBertModel): The instance of the DistilBertModel class.
            new_embeddings (nn.Embedding): The new embeddings to be set for the input.

        Returns:
            None.

        Raises:
            None.

        This method allows setting new embeddings for the input in the DistilBertModel.
        The 'self' parameter refers to the instance of the DistilBertModel class on which the method is being called.
        The 'new_embeddings' parameter is of type 'nn.Embedding' and represents the new embeddings to be set for the input.

        Example:
            ```python
            >>> model = DistilBertModel()
            >>> embeddings = nn.Embedding(vocab_size, embedding_dim)
            >>> model.set_input_embeddings(embeddings)
            ```
        """
        self.embeddings.word_embeddings = new_embeddings

    def _prune_heads(self, heads_to_prune: Dict[int, List[List[int]]]):
        """
        Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base
        class PreTrainedModel
        """
        for layer, heads in heads_to_prune.items():
            self.transformer.layer[layer].attention.prune_heads(heads)

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[BaseModelOutput, Tuple[mindspore.Tensor, ...]]:
        """
        Constructs a DistilBertModel.

        Args:
            self (DistilBertModel): The instance of the DistilBertModel class.
            input_ids (Optional[mindspore.Tensor]): Input tensor containing the indices of input sequence tokens.
                Default is None.
            attention_mask (Optional[mindspore.Tensor]): Mask tensor indicating which tokens should be attended to.
                Default is None.
            head_mask (Optional[mindspore.Tensor]): Mask tensor indicating which heads should be masked out.
                Default is None.
            inputs_embeds (Optional[mindspore.Tensor]):
                Input tensor containing the embedded representation of input sequence tokens. Default is None.
            output_attentions (Optional[bool]): Whether to return attention weights. Default is None.
            output_hidden_states (Optional[bool]): Whether to return hidden states. Default is None.
            return_dict (Optional[bool]): Whether to return a BaseModelOutput instead of a tuple. Default is None.

        Returns:
            Union[BaseModelOutput, Tuple[mindspore.Tensor, ...]]:
                The output of the DistilBertModel.

                - If `return_dict` is set to True, a BaseModelOutput object is returned.
                - Otherwise, a tuple containing a tensor and optionally, attention weights and hidden states is returned.

        Raises:
            ValueError: If both `input_ids` and `inputs_embeds` are specified simultaneously.
            ValueError: If neither `input_ids` nor `inputs_embeds` are specified.
            Exception: Any other exception that may occur during execution.
        """
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        if input_ids is not None and inputs_embeds is not None:
            raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
        if input_ids is not None:
            self.warn_if_padding_and_no_attention_mask(input_ids, attention_mask)
            input_shape = input_ids.shape
        elif inputs_embeds is not None:
            input_shape = inputs_embeds.shape[:-1]
        else:
            raise ValueError("You have to specify either input_ids or inputs_embeds")

        # Prepare head mask if needed
        head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)

        embeddings = self.embeddings(input_ids, inputs_embeds)  # (bs, seq_length, dim)

        if attention_mask is None:
            attention_mask = ops.ones(input_shape)  # (bs, seq_length)

        return self.transformer(
            x=embeddings,
            attn_mask=attention_mask,
            head_mask=head_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertModel.__init__(config)

Initializes a new instance of the DistilBertModel class.

PARAMETER	DESCRIPTION
self	The instance of the DistilBertModel class.
config	An instance of the PretrainedConfig class containing the configuration settings for the model. This parameter is required to configure the model's embeddings and transformer components. TYPE: PretrainedConfig

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def __init__(self, config: PretrainedConfig):
    """
    Initializes a new instance of the DistilBertModel class.

    Args:
        self: The instance of the DistilBertModel class.
        config (PretrainedConfig):
            An instance of the PretrainedConfig class containing the configuration settings for the model.
            This parameter is required to configure the model's embeddings and transformer components.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(config)

    self.embeddings = Embeddings(config)  # Embeddings
    self.transformer = Transformer(config)  # Encoder

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertModel.forward(input_ids=None, attention_mask=None, head_mask=None, inputs_embeds=None, output_attentions=None, output_hidden_states=None, return_dict=None)

Constructs a DistilBertModel.

PARAMETER	DESCRIPTION
self	The instance of the DistilBertModel class. TYPE: DistilBertModel
input_ids	Input tensor containing the indices of input sequence tokens. Default is None. TYPE: Optional[Tensor] DEFAULT: None
attention_mask	Mask tensor indicating which tokens should be attended to. Default is None. TYPE: Optional[Tensor] DEFAULT: None
head_mask	Mask tensor indicating which heads should be masked out. Default is None. TYPE: Optional[Tensor] DEFAULT: None
inputs_embeds	Input tensor containing the embedded representation of input sequence tokens. Default is None. TYPE: Optional[Tensor] DEFAULT: None
output_attentions	Whether to return attention weights. Default is None. TYPE: Optional[bool] DEFAULT: None
output_hidden_states	Whether to return hidden states. Default is None. TYPE: Optional[bool] DEFAULT: None
return_dict	Whether to return a BaseModelOutput instead of a tuple. Default is None. TYPE: Optional[bool] DEFAULT: None

RETURNS	DESCRIPTION
Union[BaseModelOutput, Tuple[Tensor, ...]]	Union[BaseModelOutput, Tuple[mindspore.Tensor, ...]]: The output of the DistilBertModel. If return_dict is set to True, a BaseModelOutput object is returned. Otherwise, a tuple containing a tensor and optionally, attention weights and hidden states is returned.

RAISES	DESCRIPTION
ValueError	If both input_ids and inputs_embeds are specified simultaneously.
ValueError	If neither input_ids nor inputs_embeds are specified.
Exception	Any other exception that may occur during execution.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    inputs_embeds: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[BaseModelOutput, Tuple[mindspore.Tensor, ...]]:
    """
    Constructs a DistilBertModel.

    Args:
        self (DistilBertModel): The instance of the DistilBertModel class.
        input_ids (Optional[mindspore.Tensor]): Input tensor containing the indices of input sequence tokens.
            Default is None.
        attention_mask (Optional[mindspore.Tensor]): Mask tensor indicating which tokens should be attended to.
            Default is None.
        head_mask (Optional[mindspore.Tensor]): Mask tensor indicating which heads should be masked out.
            Default is None.
        inputs_embeds (Optional[mindspore.Tensor]):
            Input tensor containing the embedded representation of input sequence tokens. Default is None.
        output_attentions (Optional[bool]): Whether to return attention weights. Default is None.
        output_hidden_states (Optional[bool]): Whether to return hidden states. Default is None.
        return_dict (Optional[bool]): Whether to return a BaseModelOutput instead of a tuple. Default is None.

    Returns:
        Union[BaseModelOutput, Tuple[mindspore.Tensor, ...]]:
            The output of the DistilBertModel.

            - If `return_dict` is set to True, a BaseModelOutput object is returned.
            - Otherwise, a tuple containing a tensor and optionally, attention weights and hidden states is returned.

    Raises:
        ValueError: If both `input_ids` and `inputs_embeds` are specified simultaneously.
        ValueError: If neither `input_ids` nor `inputs_embeds` are specified.
        Exception: Any other exception that may occur during execution.
    """
    output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
    output_hidden_states = (
        output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
    )
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    if input_ids is not None and inputs_embeds is not None:
        raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
    if input_ids is not None:
        self.warn_if_padding_and_no_attention_mask(input_ids, attention_mask)
        input_shape = input_ids.shape
    elif inputs_embeds is not None:
        input_shape = inputs_embeds.shape[:-1]
    else:
        raise ValueError("You have to specify either input_ids or inputs_embeds")

    # Prepare head mask if needed
    head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)

    embeddings = self.embeddings(input_ids, inputs_embeds)  # (bs, seq_length, dim)

    if attention_mask is None:
        attention_mask = ops.ones(input_shape)  # (bs, seq_length)

    return self.transformer(
        x=embeddings,
        attn_mask=attention_mask,
        head_mask=head_mask,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertModel.get_input_embeddings()

Retrieve the input embeddings for the DistilBertModel.

PARAMETER	DESCRIPTION
self	An instance of the DistilBertModel class. Represents the current instance of the DistilBertModel. This parameter is required for accessing the embeddings. TYPE: DistilBertModel

RETURNS	DESCRIPTION
Embedding	nn.Embedding: An instance of the nn.Embedding class. Represents the word embeddings used for input to the DistilBertModel. These embeddings are used to convert input tokens to dense vectors for processing.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def get_input_embeddings(self) -> nn.Embedding:
    """
    Retrieve the input embeddings for the DistilBertModel.

    Args:
        self (DistilBertModel): An instance of the DistilBertModel class.
            Represents the current instance of the DistilBertModel.
            This parameter is required for accessing the embeddings.

    Returns:
        nn.Embedding: An instance of the nn.Embedding class.
            Represents the word embeddings used for input to the DistilBertModel.
            These embeddings are used to convert input tokens to dense vectors for processing.

    Raises:
        None.
    """
    return self.embeddings.word_embeddings

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertModel.get_position_embeddings()

Returns the position embeddings

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def get_position_embeddings(self) -> nn.Embedding:
    """
    Returns the position embeddings
    """
    return self.embeddings.position_embeddings

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertModel.resize_position_embeddings(new_num_position_embeddings)

Resizes position embeddings of the model if new_num_position_embeddings != config.max_position_embeddings.

PARAMETER

DESCRIPTION

new_num_position_embeddings

The number of new position embedding matrix. If position embeddings are learned, increasing the size will add newly initialized vectors at the end, whereas reducing the size will remove vectors from the end. If position embeddings are not learned (e.g. sinusoidal position embeddings), increasing the size will add correct vectors at the end following the position encoding algorithm, whereas reducing the size will remove vectors from the end. TYPE: `int`

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def resize_position_embeddings(self, new_num_position_embeddings: int):
    """
    Resizes position embeddings of the model if `new_num_position_embeddings != config.max_position_embeddings`.

    Arguments:
        new_num_position_embeddings (`int`):
            The number of new position embedding matrix.

            - If position embeddings are learned, increasing the size will add newly initialized vectors at the end,
            whereas reducing the size will remove vectors from the end.
            - If position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the
            size will add correct vectors at the end following the position encoding algorithm, whereas reducing
            the size will remove vectors from the end.
    """
    num_position_embeds_diff = new_num_position_embeddings - self.config.max_position_embeddings

    # no resizing needs to be done if the length stays the same
    if num_position_embeds_diff == 0:
        return

    logger.info(f"Setting `config.max_position_embeddings={new_num_position_embeddings}`...")
    self.config.max_position_embeddings = new_num_position_embeddings

    old_position_embeddings_weight = self.embeddings.position_embeddings.weight.clone()

    self.embeddings.position_embeddings = nn.Embedding(self.config.max_position_embeddings, self.config.dim)

    if self.config.sinusoidal_pos_embds:
        create_sinusoidal_embeddings(
            n_pos=self.config.max_position_embeddings, dim=self.config.dim, out=self.position_embeddings.weight
        )
    else:
        if num_position_embeds_diff > 0:
            self.embeddings.position_embeddings.weight[:-num_position_embeds_diff] = nn.Parameter(
                old_position_embeddings_weight
            )
        else:
            self.embeddings.position_embeddings.weight = nn.Parameter(
                old_position_embeddings_weight[:num_position_embeds_diff]
            )

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertModel.set_input_embeddings(new_embeddings)

Sets the input embeddings for the DistilBertModel.

PARAMETER	DESCRIPTION
self	The instance of the DistilBertModel class. TYPE: DistilBertModel
new_embeddings	The new embeddings to be set for the input. TYPE: Embedding

RETURNS	DESCRIPTION
	None.

This method allows setting new embeddings for the input in the DistilBertModel. The 'self' parameter refers to the instance of the DistilBertModel class on which the method is being called. The 'new_embeddings' parameter is of type 'nn.Embedding' and represents the new embeddings to be set for the input.

Example

>>> model = DistilBertModel()
>>> embeddings = nn.Embedding(vocab_size, embedding_dim)
>>> model.set_input_embeddings(embeddings)

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def set_input_embeddings(self, new_embeddings: nn.Embedding):
    """
    Sets the input embeddings for the DistilBertModel.

    Args:
        self (DistilBertModel): The instance of the DistilBertModel class.
        new_embeddings (nn.Embedding): The new embeddings to be set for the input.

    Returns:
        None.

    Raises:
        None.

    This method allows setting new embeddings for the input in the DistilBertModel.
    The 'self' parameter refers to the instance of the DistilBertModel class on which the method is being called.
    The 'new_embeddings' parameter is of type 'nn.Embedding' and represents the new embeddings to be set for the input.

    Example:
        ```python
        >>> model = DistilBertModel()
        >>> embeddings = nn.Embedding(vocab_size, embedding_dim)
        >>> model.set_input_embeddings(embeddings)
        ```
    """
    self.embeddings.word_embeddings = new_embeddings

mindnlp.transformers.models.distilbert.modeling_distilbert.DistilBertPreTrainedModel

Bases: PreTrainedModel

An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

class DistilBertPreTrainedModel(PreTrainedModel):
    """
    An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained
    models.
    """
    config_class = DistilBertConfig
    load_tf_weights = None
    base_model_prefix = "distilbert"

    def _init_weights(self, cell):
        """Initialize the weights"""
        if isinstance(cell, nn.Linear):
            # Slightly different from the TF version which uses truncated_normal for initialization
            # cf https://github.com/pytorch/pytorch/pull/5617
            cell.weight.set_data(initializer(Normal(self.config.initializer_range),
                                                    cell.weight.shape, cell.weight.dtype))
            if cell.bias:
                cell.bias.set_data(initializer('zeros', cell.bias.shape, cell.bias.dtype))
        elif isinstance(cell, nn.Embedding):
            weight = np.random.normal(0.0, self.config.initializer_range, cell.weight.shape)
            if cell.padding_idx:
                weight[cell.padding_idx] = 0

            cell.weight.set_data(Tensor(weight, cell.weight.dtype))
        elif isinstance(cell, nn.LayerNorm):
            cell.weight.set_data(initializer('ones', cell.weight.shape, cell.weight.dtype))
            cell.bias.set_data(initializer('zeros', cell.bias.shape, cell.bias.dtype))

mindnlp.transformers.models.distilbert.modeling_distilbert.Embeddings

Bases: Module

Represents a neural network cell for embedding tokens with position embeddings.

This class inherits from the nn.Module module and provides methods for initializing and forwarding embeddings for token sequences. The embedding process includes the use of word embeddings and optional pre-computed word embeddings, along with position embeddings.

The 'Embeddings' class initializes the word embeddings, position embeddings, LayerNorm, and dropout, and forwards the embedded tokens by combining input embeddings with position embeddings. Additionally, it provides a method for forwarding embedded tokens with the option to pass pre-computed word embeddings.

The 'forward' method takes input token ids and, if provided, pre-computed word embeddings to return the embedded tokens with position embeddings. The resulting embedded tokens have the shape (bs, max_seq_length, dim), where 'bs' represents the batch size, 'max_seq_length' represents the maximum sequence length, and 'dim' represents the dimensionality of the embeddings. No token_type embeddings are included in the output.

PARAMETER	DESCRIPTION
input_ids	The token ids to embed. It should be a mindspore.Tensor with the shape (bs, max_seq_length). TYPE: Tensor
input_embeds	The pre-computed word embeddings. It can only be passed if the input ids are None. TYPE: *optional*, mindspore.Tensor

RETURNS	DESCRIPTION
	mindspore.Tensor: The embedded tokens (plus position embeddings, no token_type embeddings) with the shape (bs, max_seq_length, dim).

Note

The 'Embeddings' class requires the 'config' parameter of type 'PretrainedConfig' during initialization to configure the embeddings.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

class Embeddings(nn.Module):

    """
    Represents a neural network cell for embedding tokens with position embeddings.

    This class inherits from the nn.Module module and provides methods for initializing and forwarding embeddings
    for token sequences. The embedding process includes the use of word embeddings and optional pre-computed word
    embeddings, along with position embeddings.

    The 'Embeddings' class initializes the word embeddings, position embeddings, LayerNorm, and dropout, and forwards
    the embedded tokens by combining input embeddings with position embeddings.
    Additionally, it provides a method for forwarding embedded tokens with the option to pass pre-computed word embeddings.

    The 'forward' method takes input token ids and, if provided, pre-computed word embeddings to return the embedded
    tokens with position embeddings.
    The resulting embedded tokens have the shape (bs, max_seq_length, dim), where 'bs' represents the batch size,
    'max_seq_length' represents the maximum sequence length, and 'dim' represents the dimensionality of the embeddings.
    No token_type embeddings are included in the output.

    Parameters:
        input_ids (mindspore.Tensor):
            The token ids to embed. It should be a mindspore.Tensor with the shape (bs, max_seq_length).
        input_embeds (*optional*, mindspore.Tensor):
            The pre-computed word embeddings. It can only be passed if the input ids are `None`.

    Returns:
        mindspore.Tensor:
            The embedded tokens (plus position embeddings, no token_type embeddings) with the shape (bs, max_seq_length, dim).

    Note:
        The 'Embeddings' class requires the 'config' parameter of type 'PretrainedConfig' during initialization to configure the embeddings.

    """
    def __init__(self, config: PretrainedConfig):
        """
        Initializes an instance of the Embeddings class.

        Args:
            self: The instance of the class.
            config (PretrainedConfig):
                The configuration object containing the parameters for the embeddings.

                 - vocab_size (int): The size of the vocabulary.
                 - dim (int): The dimensionality of the embeddings.
                 - pad_token_id (int): The ID of the padding token.
                 - max_position_embeddings (int): The maximum number of positions for the position embeddings.
                 - sinusoidal_pos_embds (bool): Determines whether to use sinusoidal position embeddings.
                 - dropout (float): The dropout rate for the embeddings.

        Returns:
            None

        Raises:
            None
        """
        super().__init__()
        self.word_embeddings = nn.Embedding(config.vocab_size, config.dim, padding_idx=config.pad_token_id)
        self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.dim)
        if config.sinusoidal_pos_embds:
            create_sinusoidal_embeddings(
                n_pos=config.max_position_embeddings, dim=config.dim, out=self.position_embeddings.weight
            )

        self.LayerNorm = nn.LayerNorm(config.dim, eps=1e-12)
        self.dropout = nn.Dropout(p=config.dropout)
        self.position_ids = ops.arange(config.max_position_embeddings).expand((1, -1))

    def forward(self, input_ids: mindspore.Tensor, input_embeds: Optional[mindspore.Tensor] = None) -> mindspore.Tensor:
        """
        Parameters:
            input_ids (mindspore.Tensor):
                mindspore.Tensor(bs, max_seq_length) The token ids to embed.
            input_embeds (*optional*, mindspore.Tensor):
                The pre-computed word embeddings. Can only be passed if the input ids are `None`.


        Returns:
            mindspore.Tensor(bs, max_seq_length, dim):
                The embedded tokens (plus position embeddings, no token_type embeddings)
        """
        if input_ids is not None:
            input_embeds = self.word_embeddings(input_ids)  # (bs, max_seq_length, dim)

        seq_length = input_embeds.shape[1]

        # Setting the position-ids to the registered buffer in forwardor, it helps
        # when tracing the model without passing position-ids, solves
        # isues similar to issue #5664
        if hasattr(self, "position_ids"):
            position_ids = self.position_ids[:, :seq_length]
        else:
            position_ids = ops.arange(seq_length, dtype=mindspore.int64)  # (max_seq_length)
            position_ids = position_ids.unsqueeze(0).expand_as(input_ids)  # (bs, max_seq_length)

        position_embeddings = self.position_embeddings(position_ids)  # (bs, max_seq_length, dim)

        embeddings = input_embeds + position_embeddings  # (bs, max_seq_length, dim)
        embeddings = self.LayerNorm(embeddings)  # (bs, max_seq_length, dim)
        embeddings = self.dropout(embeddings)  # (bs, max_seq_length, dim)
        return embeddings

mindnlp.transformers.models.distilbert.modeling_distilbert.Embeddings.__init__(config)

Initializes an instance of the Embeddings class.

PARAMETER	DESCRIPTION
self	The instance of the class.
config	The configuration object containing the parameters for the embeddings. vocab_size (int): The size of the vocabulary. dim (int): The dimensionality of the embeddings. pad_token_id (int): The ID of the padding token. max_position_embeddings (int): The maximum number of positions for the position embeddings. sinusoidal_pos_embds (bool): Determines whether to use sinusoidal position embeddings. dropout (float): The dropout rate for the embeddings. TYPE: PretrainedConfig

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def __init__(self, config: PretrainedConfig):
    """
    Initializes an instance of the Embeddings class.

    Args:
        self: The instance of the class.
        config (PretrainedConfig):
            The configuration object containing the parameters for the embeddings.

             - vocab_size (int): The size of the vocabulary.
             - dim (int): The dimensionality of the embeddings.
             - pad_token_id (int): The ID of the padding token.
             - max_position_embeddings (int): The maximum number of positions for the position embeddings.
             - sinusoidal_pos_embds (bool): Determines whether to use sinusoidal position embeddings.
             - dropout (float): The dropout rate for the embeddings.

    Returns:
        None

    Raises:
        None
    """
    super().__init__()
    self.word_embeddings = nn.Embedding(config.vocab_size, config.dim, padding_idx=config.pad_token_id)
    self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.dim)
    if config.sinusoidal_pos_embds:
        create_sinusoidal_embeddings(
            n_pos=config.max_position_embeddings, dim=config.dim, out=self.position_embeddings.weight
        )

    self.LayerNorm = nn.LayerNorm(config.dim, eps=1e-12)
    self.dropout = nn.Dropout(p=config.dropout)
    self.position_ids = ops.arange(config.max_position_embeddings).expand((1, -1))

mindnlp.transformers.models.distilbert.modeling_distilbert.Embeddings.forward(input_ids, input_embeds=None)

PARAMETER	DESCRIPTION
input_ids	mindspore.Tensor(bs, max_seq_length) The token ids to embed. TYPE: Tensor
input_embeds	The pre-computed word embeddings. Can only be passed if the input ids are None. TYPE: *optional*, mindspore.Tensor DEFAULT: None

RETURNS	DESCRIPTION
Tensor	mindspore.Tensor(bs, max_seq_length, dim): The embedded tokens (plus position embeddings, no token_type embeddings)

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def forward(self, input_ids: mindspore.Tensor, input_embeds: Optional[mindspore.Tensor] = None) -> mindspore.Tensor:
    """
    Parameters:
        input_ids (mindspore.Tensor):
            mindspore.Tensor(bs, max_seq_length) The token ids to embed.
        input_embeds (*optional*, mindspore.Tensor):
            The pre-computed word embeddings. Can only be passed if the input ids are `None`.


    Returns:
        mindspore.Tensor(bs, max_seq_length, dim):
            The embedded tokens (plus position embeddings, no token_type embeddings)
    """
    if input_ids is not None:
        input_embeds = self.word_embeddings(input_ids)  # (bs, max_seq_length, dim)

    seq_length = input_embeds.shape[1]

    # Setting the position-ids to the registered buffer in forwardor, it helps
    # when tracing the model without passing position-ids, solves
    # isues similar to issue #5664
    if hasattr(self, "position_ids"):
        position_ids = self.position_ids[:, :seq_length]
    else:
        position_ids = ops.arange(seq_length, dtype=mindspore.int64)  # (max_seq_length)
        position_ids = position_ids.unsqueeze(0).expand_as(input_ids)  # (bs, max_seq_length)

    position_embeddings = self.position_embeddings(position_ids)  # (bs, max_seq_length, dim)

    embeddings = input_embeds + position_embeddings  # (bs, max_seq_length, dim)
    embeddings = self.LayerNorm(embeddings)  # (bs, max_seq_length, dim)
    embeddings = self.dropout(embeddings)  # (bs, max_seq_length, dim)
    return embeddings

mindnlp.transformers.models.distilbert.modeling_distilbert.FFN

Bases: Module

A class that represents a Feed-Forward Network (FFN) implemented as a neural network cell.

The FFN class inherits from the nn.Module class of the MindSpore framework. It is designed to process input tensors using a two-layer fully connected feed-forward network. The class implements the forward pass of the network, applying linear transformations, activation functions, and dropout regularization.

ATTRIBUTE	DESCRIPTION
dropout	The dropout regularization layer. TYPE: Dropout
chunk_size_feed_forward	The size of chunks used in the forward pass. TYPE: int
seq_len_dim	The dimension corresponding to the sequence length in the input tensor. TYPE: int
lin1	The first fully connected layer of the FFN. TYPE: Linear
lin2	The second fully connected layer of the FFN. TYPE: Linear
activation	The activation function used in the network. TYPE: Callable

METHOD	DESCRIPTION
forward	Applies the forward pass of the FFN to the input tensor.
ff_chunk	Performs a chunk of the forward pass on the input tensor.

Note

This class assumes that the input tensor has shape (batch_size, sequence_length, dim).

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

class FFN(nn.Module):

    """A class that represents a Feed-Forward Network (FFN) implemented as a neural network cell.

    The FFN class inherits from the nn.Module class of the MindSpore framework. It is designed to process input tensors using a
    two-layer fully connected feed-forward network. The class implements the forward pass of the network, applying linear
    transformations, activation functions, and dropout regularization.

    Attributes:
        dropout (mindspore.nn.Dropout): The dropout regularization layer.
        chunk_size_feed_forward (int): The size of chunks used in the forward pass.
        seq_len_dim (int): The dimension corresponding to the sequence length in the input tensor.
        lin1 (mindspore.nn.Linear): The first fully connected layer of the FFN.
        lin2 (mindspore.nn.Linear): The second fully connected layer of the FFN.
        activation (Callable): The activation function used in the network.

    Methods:
        forward:
            Applies the forward pass of the FFN to the input tensor.

        ff_chunk:
            Performs a chunk of the forward pass on the input tensor.

    Note:
        This class assumes that the input tensor has shape (batch_size, sequence_length, dim).

    """
    def __init__(self, config: PretrainedConfig):
        """
        Initializes an instance of the FFN (Feed Forward Network) class.

        Args:
            self: The object itself.
            config (PretrainedConfig): The configuration object containing various settings for the FFN.
                The object should have the following attributes:

                 - dropout (float): The dropout probability used in the FFN.
                 - chunk_size_feed_forward (int): The chunk size for feed-forward operations.
                 - dim (int): The input dimension of the FFN.
                 - hidden_dim (int): The hidden dimension of the FFN.
                 - activation (str): The activation function used in the FFN.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.dropout = nn.Dropout(p=config.dropout)
        self.chunk_size_feed_forward = config.chunk_size_feed_forward
        self.seq_len_dim = 1
        self.lin1 = nn.Linear(config.dim, config.hidden_dim)
        self.lin2 = nn.Linear(config.hidden_dim, config.dim)
        self.activation = get_activation(config.activation)

    def forward(self, input: mindspore.Tensor) -> mindspore.Tensor:
        """Constructs the feedforward network.

        Args:
            self (FFN): An instance of the FFN class.
            input (mindspore.Tensor): The input tensor of shape [batch_size, sequence_length, hidden_size].

        Returns:
            mindspore.Tensor: The output tensor of shape [batch_size, sequence_length, hidden_size].

        Raises:
            TypeError: If input is not an instance of the mindspore.Tensor class.
        """
        return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)

    def ff_chunk(self, input: mindspore.Tensor) -> mindspore.Tensor:
        """
        Performs a forward pass through the feedforward network (FFN) chunk.

        Args:
            self (FFN): The instance of the FFN class.
            input (mindspore.Tensor): The input tensor to the FFN chunk.

        Returns:
            mindspore.Tensor: The output tensor after passing through the FFN chunk.

        Raises:
            None.

        This method applies a series of operations to the input tensor to perform a forward pass through the FFN chunk.
        The operations include linear transformation, activation function, linear transformation, and dropout.
        The resulting tensor is then returned as the output of the FFN chunk.
        """
        x = self.lin1(input)
        x = self.activation(x)
        x = self.lin2(x)
        x = self.dropout(x)
        return x

mindnlp.transformers.models.distilbert.modeling_distilbert.FFN.__init__(config)

Initializes an instance of the FFN (Feed Forward Network) class.

PARAMETER	DESCRIPTION
self	The object itself.
config	The configuration object containing various settings for the FFN. The object should have the following attributes: dropout (float): The dropout probability used in the FFN. chunk_size_feed_forward (int): The chunk size for feed-forward operations. dim (int): The input dimension of the FFN. hidden_dim (int): The hidden dimension of the FFN. activation (str): The activation function used in the FFN. TYPE: PretrainedConfig

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def __init__(self, config: PretrainedConfig):
    """
    Initializes an instance of the FFN (Feed Forward Network) class.

    Args:
        self: The object itself.
        config (PretrainedConfig): The configuration object containing various settings for the FFN.
            The object should have the following attributes:

             - dropout (float): The dropout probability used in the FFN.
             - chunk_size_feed_forward (int): The chunk size for feed-forward operations.
             - dim (int): The input dimension of the FFN.
             - hidden_dim (int): The hidden dimension of the FFN.
             - activation (str): The activation function used in the FFN.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.dropout = nn.Dropout(p=config.dropout)
    self.chunk_size_feed_forward = config.chunk_size_feed_forward
    self.seq_len_dim = 1
    self.lin1 = nn.Linear(config.dim, config.hidden_dim)
    self.lin2 = nn.Linear(config.hidden_dim, config.dim)
    self.activation = get_activation(config.activation)

mindnlp.transformers.models.distilbert.modeling_distilbert.FFN.ff_chunk(input)

Performs a forward pass through the feedforward network (FFN) chunk.

PARAMETER	DESCRIPTION
self	The instance of the FFN class. TYPE: FFN
input	The input tensor to the FFN chunk. TYPE: Tensor

RETURNS	DESCRIPTION
Tensor	mindspore.Tensor: The output tensor after passing through the FFN chunk.

This method applies a series of operations to the input tensor to perform a forward pass through the FFN chunk. The operations include linear transformation, activation function, linear transformation, and dropout. The resulting tensor is then returned as the output of the FFN chunk.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def ff_chunk(self, input: mindspore.Tensor) -> mindspore.Tensor:
    """
    Performs a forward pass through the feedforward network (FFN) chunk.

    Args:
        self (FFN): The instance of the FFN class.
        input (mindspore.Tensor): The input tensor to the FFN chunk.

    Returns:
        mindspore.Tensor: The output tensor after passing through the FFN chunk.

    Raises:
        None.

    This method applies a series of operations to the input tensor to perform a forward pass through the FFN chunk.
    The operations include linear transformation, activation function, linear transformation, and dropout.
    The resulting tensor is then returned as the output of the FFN chunk.
    """
    x = self.lin1(input)
    x = self.activation(x)
    x = self.lin2(x)
    x = self.dropout(x)
    return x

mindnlp.transformers.models.distilbert.modeling_distilbert.FFN.forward(input)

Constructs the feedforward network.

PARAMETER	DESCRIPTION
self	An instance of the FFN class. TYPE: FFN
input	The input tensor of shape [batch_size, sequence_length, hidden_size]. TYPE: Tensor

RETURNS	DESCRIPTION
Tensor	mindspore.Tensor: The output tensor of shape [batch_size, sequence_length, hidden_size].

RAISES	DESCRIPTION
TypeError	If input is not an instance of the mindspore.Tensor class.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def forward(self, input: mindspore.Tensor) -> mindspore.Tensor:
    """Constructs the feedforward network.

    Args:
        self (FFN): An instance of the FFN class.
        input (mindspore.Tensor): The input tensor of shape [batch_size, sequence_length, hidden_size].

    Returns:
        mindspore.Tensor: The output tensor of shape [batch_size, sequence_length, hidden_size].

    Raises:
        TypeError: If input is not an instance of the mindspore.Tensor class.
    """
    return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)

mindnlp.transformers.models.distilbert.modeling_distilbert.MultiHeadSelfAttention

Bases: Module

This class represents a multi-head self-attention mechanism used in neural networks. It is designed to be used as a building block for Transformer-based models. The class inherits from the nn.Module class and implements methods for initializing the attention mechanism, pruning heads, and forwarding the attention weights and contextualized layer.

ATTRIBUTE	DESCRIPTION
config	The configuration object containing the parameters for the attention mechanism. TYPE: PretrainedConfig
n_heads	The number of attention heads. TYPE: int
dim	The dimensionality of the attention mechanism. TYPE: int
dropout	The dropout layer applied to the attention weights. TYPE: Dropout
is_causal	Indicates whether the attention mechanism is causal or not. TYPE: bool
q_lin	Linear layer for query projection. TYPE: Linear
k_lin	Linear layer for key projection. TYPE: Linear
v_lin	Linear layer for value projection. TYPE: Linear
out_lin	Linear layer for the output projection. TYPE: Linear
pruned_heads	Set of pruned attention heads. TYPE: Set[int]
attention_head_size	The size of each attention head. TYPE: int

METHOD	DESCRIPTION
__init__	Initializes the MultiHeadSelfAttention object with the provided configuration.
prune_heads	Prunes the specified attention heads from the attention mechanism.
forward	Constructs the attention weights and contextualized layer.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

class MultiHeadSelfAttention(nn.Module):

    """
    This class represents a multi-head self-attention mechanism used in neural networks.
    It is designed to be used as a building block for Transformer-based models.
    The class inherits from the nn.Module class and implements methods for initializing the attention mechanism,
    pruning heads, and forwarding the attention weights and contextualized layer.

    Attributes:
        config (PretrainedConfig): The configuration object containing the parameters for the attention mechanism.
        n_heads (int): The number of attention heads.
        dim (int): The dimensionality of the attention mechanism.
        dropout (nn.Dropout): The dropout layer applied to the attention weights.
        is_causal (bool): Indicates whether the attention mechanism is causal or not.
        q_lin (nn.Linear): Linear layer for query projection.
        k_lin (nn.Linear): Linear layer for key projection.
        v_lin (nn.Linear): Linear layer for value projection.
        out_lin (nn.Linear): Linear layer for the output projection.
        pruned_heads (Set[int]): Set of pruned attention heads.
        attention_head_size (int): The size of each attention head.

    Methods:
        __init__: Initializes the MultiHeadSelfAttention object with the provided configuration.
        prune_heads: Prunes the specified attention heads from the attention mechanism.
        forward:
          Constructs the attention weights and contextualized layer.
    """
    def __init__(self, config: PretrainedConfig):
        """
        Initializes the MultiHeadSelfAttention class.

        Args:
            self: The instance of the MultiHeadSelfAttention class.
            config (PretrainedConfig): The configuration object used for initializing the MultiHeadSelfAttention.
                It contains parameters such as number of heads (n_heads), dimension (dim), and
                attention dropout (attention_dropout).

        Returns:
            None.

        Raises:
            ValueError: If the number of heads (n_heads) specified in the config does not evenly divide
                the dimension (dim).

        """
        super().__init__()
        self.config = config

        self.n_heads = config.n_heads
        self.dim = config.dim
        self.dropout = nn.Dropout(p=config.attention_dropout)
        self.is_causal = False

        # Have an even number of multi heads that divide the dimensions
        if self.dim % self.n_heads != 0:
            # Raise value errors for even multi-head attention nodes
            raise ValueError(f"self.n_heads: {self.n_heads} must divide self.dim: {self.dim} evenly")

        self.q_lin = nn.Linear(config.dim, config.dim)
        self.k_lin = nn.Linear(config.dim, config.dim)
        self.v_lin = nn.Linear(config.dim, config.dim)
        self.out_lin = nn.Linear(config.dim, config.dim)

        self.pruned_heads: Set[int] = set()
        self.attention_head_size = self.dim // self.n_heads

    def prune_heads(self, heads: List[int]):
        """
        This method 'prune_heads' is defined within the 'MultiHeadSelfAttention' class and is used to prune specific 
        attention heads in the multi-head self-attention mechanism.

        Args:
            self: The instance of the MultiHeadSelfAttention class.
            heads (List[int]): A list of integers representing the attention heads to be pruned. 
                It identifies the specific attention heads to be removed from the attention mechanism.

        Returns:
            None: This method does not return a value as it operates directly on the instance attributes.

        Raises:
            None.
        """
        if len(heads) == 0:
            return
        heads, index = find_pruneable_heads_and_indices(
            heads, self.n_heads, self.attention_head_size, self.pruned_heads
        )
        # Prune linear layers
        self.q_lin = prune_linear_layer(self.q_lin, index)
        self.k_lin = prune_linear_layer(self.k_lin, index)
        self.v_lin = prune_linear_layer(self.v_lin, index)
        self.out_lin = prune_linear_layer(self.out_lin, index, dim=1)
        # Update hyper params
        self.n_heads = self.n_heads - len(heads)
        self.dim = self.attention_head_size * self.n_heads
        self.pruned_heads = self.pruned_heads.union(heads)

    def forward(
        self,
        query: mindspore.Tensor,
        key: mindspore.Tensor,
        value: mindspore.Tensor,
        mask: mindspore.Tensor,
        head_mask: Optional[mindspore.Tensor] = None,
        output_attentions: bool = False,
    ) -> Tuple[mindspore.Tensor, ...]:
        """
        Parameters:
            query: mindspore.Tensor(bs, seq_length, dim)
            key: mindspore.Tensor(bs, seq_length, dim)
            value: mindspore.Tensor(bs, seq_length, dim)
            mask: mindspore.Tensor(bs, seq_length)

        Returns:
            weights: mindspore.Tensor(bs, n_heads, seq_length, seq_length)
            Attention weights context: mindspore.Tensor(bs, seq_length, dim) Contextualized layer.
                Optional: only if `output_attentions=True`
        """
        bs = query.shape[0]
        k_length = key.shape[1]
        # assert dim == self.dim, f'Dimensions do not match: {dim} input vs {self.dim} configured'
        # assert key.shape == value.shape

        dim_per_head = self.dim // self.n_heads

        mask_reshp = (bs, 1, 1, k_length)

        def shape(x: mindspore.Tensor) -> mindspore.Tensor:
            """separate heads"""
            return x.view(bs, -1, self.n_heads, dim_per_head).swapaxes(1, 2)

        def unshape(x: mindspore.Tensor) -> mindspore.Tensor:
            """group heads"""
            return x.swapaxes(1, 2).view(bs, -1, self.n_heads * dim_per_head)

        q = shape(self.q_lin(query))  # (bs, n_heads, q_length, dim_per_head)
        k = shape(self.k_lin(key))  # (bs, n_heads, k_length, dim_per_head)
        v = shape(self.v_lin(value))  # (bs, n_heads, k_length, dim_per_head)

        q = q / math.sqrt(dim_per_head)  # (bs, n_heads, q_length, dim_per_head)
        scores = ops.matmul(q, k.swapaxes(2, 3))  # (bs, n_heads, q_length, k_length)
        mask = (mask == 0).view(mask_reshp).expand_as(scores)  # (bs, n_heads, q_length, k_length)
        scores = scores.masked_fill(
            mask, finfo(scores.dtype, 'min')
        )  # (bs, n_heads, q_length, k_length)

        weights = ops.softmax(scores, axis=-1)  # (bs, n_heads, q_length, k_length)
        weights = self.dropout(weights)  # (bs, n_heads, q_length, k_length)

        # Mask heads if we want to
        if head_mask is not None:
            weights = weights * head_mask

        context = ops.matmul(weights, v)  # (bs, n_heads, q_length, dim_per_head)
        context = unshape(context)  # (bs, q_length, dim)
        context = self.out_lin(context)  # (bs, q_length, dim)

        if output_attentions:
            return (context, weights)
        return (context,)

mindnlp.transformers.models.distilbert.modeling_distilbert.MultiHeadSelfAttention.__init__(config)

Initializes the MultiHeadSelfAttention class.

PARAMETER	DESCRIPTION
self	The instance of the MultiHeadSelfAttention class.
config	The configuration object used for initializing the MultiHeadSelfAttention. It contains parameters such as number of heads (n_heads), dimension (dim), and attention dropout (attention_dropout). TYPE: PretrainedConfig

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If the number of heads (n_heads) specified in the config does not evenly divide the dimension (dim).

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def __init__(self, config: PretrainedConfig):
    """
    Initializes the MultiHeadSelfAttention class.

    Args:
        self: The instance of the MultiHeadSelfAttention class.
        config (PretrainedConfig): The configuration object used for initializing the MultiHeadSelfAttention.
            It contains parameters such as number of heads (n_heads), dimension (dim), and
            attention dropout (attention_dropout).

    Returns:
        None.

    Raises:
        ValueError: If the number of heads (n_heads) specified in the config does not evenly divide
            the dimension (dim).

    """
    super().__init__()
    self.config = config

    self.n_heads = config.n_heads
    self.dim = config.dim
    self.dropout = nn.Dropout(p=config.attention_dropout)
    self.is_causal = False

    # Have an even number of multi heads that divide the dimensions
    if self.dim % self.n_heads != 0:
        # Raise value errors for even multi-head attention nodes
        raise ValueError(f"self.n_heads: {self.n_heads} must divide self.dim: {self.dim} evenly")

    self.q_lin = nn.Linear(config.dim, config.dim)
    self.k_lin = nn.Linear(config.dim, config.dim)
    self.v_lin = nn.Linear(config.dim, config.dim)
    self.out_lin = nn.Linear(config.dim, config.dim)

    self.pruned_heads: Set[int] = set()
    self.attention_head_size = self.dim // self.n_heads

mindnlp.transformers.models.distilbert.modeling_distilbert.MultiHeadSelfAttention.forward(query, key, value, mask, head_mask=None, output_attentions=False)

PARAMETER	DESCRIPTION
query	mindspore.Tensor(bs, seq_length, dim) TYPE: Tensor
key	mindspore.Tensor(bs, seq_length, dim) TYPE: Tensor
value	mindspore.Tensor(bs, seq_length, dim) TYPE: Tensor
mask	mindspore.Tensor(bs, seq_length) TYPE: Tensor

RETURNS	DESCRIPTION
weights	mindspore.Tensor(bs, n_heads, seq_length, seq_length) TYPE: Tensor
...	Attention weights context: mindspore.Tensor(bs, seq_length, dim) Contextualized layer. Optional: only if output_attentions=True

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def forward(
    self,
    query: mindspore.Tensor,
    key: mindspore.Tensor,
    value: mindspore.Tensor,
    mask: mindspore.Tensor,
    head_mask: Optional[mindspore.Tensor] = None,
    output_attentions: bool = False,
) -> Tuple[mindspore.Tensor, ...]:
    """
    Parameters:
        query: mindspore.Tensor(bs, seq_length, dim)
        key: mindspore.Tensor(bs, seq_length, dim)
        value: mindspore.Tensor(bs, seq_length, dim)
        mask: mindspore.Tensor(bs, seq_length)

    Returns:
        weights: mindspore.Tensor(bs, n_heads, seq_length, seq_length)
        Attention weights context: mindspore.Tensor(bs, seq_length, dim) Contextualized layer.
            Optional: only if `output_attentions=True`
    """
    bs = query.shape[0]
    k_length = key.shape[1]
    # assert dim == self.dim, f'Dimensions do not match: {dim} input vs {self.dim} configured'
    # assert key.shape == value.shape

    dim_per_head = self.dim // self.n_heads

    mask_reshp = (bs, 1, 1, k_length)

    def shape(x: mindspore.Tensor) -> mindspore.Tensor:
        """separate heads"""
        return x.view(bs, -1, self.n_heads, dim_per_head).swapaxes(1, 2)

    def unshape(x: mindspore.Tensor) -> mindspore.Tensor:
        """group heads"""
        return x.swapaxes(1, 2).view(bs, -1, self.n_heads * dim_per_head)

    q = shape(self.q_lin(query))  # (bs, n_heads, q_length, dim_per_head)
    k = shape(self.k_lin(key))  # (bs, n_heads, k_length, dim_per_head)
    v = shape(self.v_lin(value))  # (bs, n_heads, k_length, dim_per_head)

    q = q / math.sqrt(dim_per_head)  # (bs, n_heads, q_length, dim_per_head)
    scores = ops.matmul(q, k.swapaxes(2, 3))  # (bs, n_heads, q_length, k_length)
    mask = (mask == 0).view(mask_reshp).expand_as(scores)  # (bs, n_heads, q_length, k_length)
    scores = scores.masked_fill(
        mask, finfo(scores.dtype, 'min')
    )  # (bs, n_heads, q_length, k_length)

    weights = ops.softmax(scores, axis=-1)  # (bs, n_heads, q_length, k_length)
    weights = self.dropout(weights)  # (bs, n_heads, q_length, k_length)

    # Mask heads if we want to
    if head_mask is not None:
        weights = weights * head_mask

    context = ops.matmul(weights, v)  # (bs, n_heads, q_length, dim_per_head)
    context = unshape(context)  # (bs, q_length, dim)
    context = self.out_lin(context)  # (bs, q_length, dim)

    if output_attentions:
        return (context, weights)
    return (context,)

mindnlp.transformers.models.distilbert.modeling_distilbert.MultiHeadSelfAttention.prune_heads(heads)

This method 'prune_heads' is defined within the 'MultiHeadSelfAttention' class and is used to prune specific attention heads in the multi-head self-attention mechanism.

PARAMETER	DESCRIPTION
self	The instance of the MultiHeadSelfAttention class.
heads	A list of integers representing the attention heads to be pruned. It identifies the specific attention heads to be removed from the attention mechanism. TYPE: List[int]

RETURNS	DESCRIPTION
None	This method does not return a value as it operates directly on the instance attributes.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def prune_heads(self, heads: List[int]):
    """
    This method 'prune_heads' is defined within the 'MultiHeadSelfAttention' class and is used to prune specific 
    attention heads in the multi-head self-attention mechanism.

    Args:
        self: The instance of the MultiHeadSelfAttention class.
        heads (List[int]): A list of integers representing the attention heads to be pruned. 
            It identifies the specific attention heads to be removed from the attention mechanism.

    Returns:
        None: This method does not return a value as it operates directly on the instance attributes.

    Raises:
        None.
    """
    if len(heads) == 0:
        return
    heads, index = find_pruneable_heads_and_indices(
        heads, self.n_heads, self.attention_head_size, self.pruned_heads
    )
    # Prune linear layers
    self.q_lin = prune_linear_layer(self.q_lin, index)
    self.k_lin = prune_linear_layer(self.k_lin, index)
    self.v_lin = prune_linear_layer(self.v_lin, index)
    self.out_lin = prune_linear_layer(self.out_lin, index, dim=1)
    # Update hyper params
    self.n_heads = self.n_heads - len(heads)
    self.dim = self.attention_head_size * self.n_heads
    self.pruned_heads = self.pruned_heads.union(heads)

mindnlp.transformers.models.distilbert.modeling_distilbert.Transformer

Bases: Module

This class represents a Transformer model, which is a type of neural network architecture commonly used in natural language processing tasks.

The Transformer class initializes with a configuration object that contains information about the model's architecture. It creates a list of TransformerBlock instances, one for each layer specified in the configuration. The number of layers is stored in the 'n_layers' attribute.

The 'forward' method takes as input an embedded input sequence tensor, 'x', and optionally an attention mask tensor, 'attn_mask'. It returns the hidden state tensor of the last layer, 'hidden_state', along with additional outputs depending on the specified options. If 'output_hidden_states' is True, it also returns a tuple containing the hidden states from each layer, 'all_hidden_states'. If 'output_attentions' is True, it also returns a tuple containing the attention weights from each layer, 'all_attentions'. The method also supports returning the outputs as a 'BaseModelOutput' object if the 'return_dict' flag is set to True.

Note

The 'forward' method internally iterates through each layer of the Transformer model and applies the layer module to the input tensor. It accumulates the hidden states and attention weights if the corresponding output options are enabled.

PARAMETER	DESCRIPTION
x	A tensor of shape (bs, seq_length, dim) representing the embedded input sequence.
attn_mask	An optional tensor of shape (bs, seq_length) representing the attention mask on the sequence.

RETURNS	DESCRIPTION
hidden_state	A tensor of shape (bs, seq_length, dim) representing the sequence of hidden states in the last (top) layer.
all_hidden_states	A tuple of tensors, each of shape (bs, seq_length, dim), containing the hidden states from each layer. This is only returned if 'output_hidden_states' is set to True.
all_attentions	A tuple of tensors, each of shape (bs, n_heads, seq_length, seq_length), containing the attention weights from each layer. This is only returned if 'output_attentions' is set to True.

RAISES	DESCRIPTION
ValueError	If the length of 'layer_outputs' is not as expected based on the output options.

Note

The 'forward' method builds the Transformer model by sequentially applying the layer modules to the input tensor. It uses the 'head_mask' tensor for applying head-wise masking during attention operations.

Example

>>> config = PretrainedConfig(n_layers=6)
>>> transformer = Transformer(config)
>>> input_tensor = mindspore.Tensor(bs, seq_length, dim)
>>> output = transformer.forward(input_tensor, attn_mask, head_mask, output_attentions=True, output_hidden_states=True)

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

class Transformer(nn.Module):

    """
    This class represents a Transformer model, which is a type of neural network architecture commonly used in
    natural language processing tasks.

    The Transformer class initializes with a configuration object that contains information about the model's architecture.
    It creates a list of TransformerBlock instances, one for each layer specified in the configuration.
    The number of layers is stored in the 'n_layers' attribute.

    The 'forward' method takes as input an embedded input sequence tensor, 'x', and optionally an attention mask tensor,
    'attn_mask'. It returns the hidden state tensor of the last layer, 'hidden_state', along with additional outputs
    depending on the specified options.
    If 'output_hidden_states' is True, it also returns a tuple containing the hidden states from each layer, 'all_hidden_states'.
    If 'output_attentions' is True, it also returns a tuple containing the attention weights from each layer,
    'all_attentions'.
    The method also supports returning the outputs as a 'BaseModelOutput' object if the 'return_dict' flag is set to True.

    Note:
        The 'forward' method internally iterates through each layer of the Transformer model and applies the layer
        module to the input tensor.
        It accumulates the hidden states and attention weights if the corresponding output options are enabled.

    Parameters:
        x: A tensor of shape (bs, seq_length, dim) representing the embedded input sequence.
        attn_mask: An optional tensor of shape (bs, seq_length) representing the attention mask on the sequence.

    Returns:
        hidden_state: A tensor of shape (bs, seq_length, dim) representing the sequence of hidden states in the last (top) layer.
        all_hidden_states: A tuple of tensors, each of shape (bs, seq_length, dim),
            containing the hidden states from each layer. This is only returned if 'output_hidden_states' is set to True.
        all_attentions: A tuple of tensors, each of shape (bs, n_heads, seq_length, seq_length),
            containing the attention weights from each layer. This is only returned if 'output_attentions' is set to True.

    Raises:
        ValueError: If the length of 'layer_outputs' is not as expected based on the output options.

    Note:
        The 'forward' method builds the Transformer model by sequentially applying the layer modules to the input tensor.
        It uses the 'head_mask' tensor for applying head-wise masking during attention operations.

    Example:
        ```python
        >>> config = PretrainedConfig(n_layers=6)
        >>> transformer = Transformer(config)
        >>> input_tensor = mindspore.Tensor(bs, seq_length, dim)
        >>> output = transformer.forward(input_tensor, attn_mask, head_mask, output_attentions=True, output_hidden_states=True)
        ```
    """
    def __init__(self, config: PretrainedConfig):
        """
        Initializes an instance of the Transformer class.

        Args:
            self (Transformer): The instance of the Transformer class.
            config (PretrainedConfig): A PretrainedConfig object containing configuration parameters for the Transformer.
                The 'n_layers' attribute of the PretrainedConfig object specifies the number of layers in the Transformer.
                This parameter is required for setting up the Transformer instance.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.n_layers = config.n_layers
        self.layer = nn.ModuleList([TransformerBlock(config) for _ in range(config.n_layers)])
        self.gradient_checkpointing = False

    def forward(
        self,
        x: mindspore.Tensor,
        attn_mask: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        output_attentions: bool = False,
        output_hidden_states: bool = False,
        return_dict: Optional[bool] = None,
    ) -> Union[BaseModelOutput, Tuple[mindspore.Tensor, ...]]:  # docstyle-ignore
        """
        Parameters:
            x: mindspore.Tensor(bs, seq_length, dim) Input sequence embedded.
            attn_mask: mindspore.Tensor(bs, seq_length) Attention mask on the sequence.

        Returns:
            hidden_state: mindspore.Tensor(bs, seq_length, dim) Sequence of hidden states in the last (top)
            layer all_hidden_states: Tuple[mindspore.Tensor(bs, seq_length, dim)]
                Tuple of length n_layers with the hidden states from each layer.
                Optional: only if output_hidden_states=True
            all_attentions: Tuple[mindspore.Tensor(bs, n_heads, seq_length, seq_length)]
                Tuple of length n_layers with the attention weights from each layer
                Optional: only if output_attentions=True
        """
        all_hidden_states = () if output_hidden_states else None
        all_attentions = () if output_attentions else None

        hidden_state = x
        for i, layer_module in enumerate(self.layer):
            if output_hidden_states:
                all_hidden_states = all_hidden_states + (hidden_state,)

            layer_outputs = layer_module(
                hidden_state,
                attn_mask,
                head_mask[i],
                output_attentions,
            )

            hidden_state = layer_outputs[-1]

            if output_attentions:
                if len(layer_outputs) != 2:
                    raise ValueError(f"The length of the layer_outputs should be 2, but it is {len(layer_outputs)}")

                attentions = layer_outputs[0]
                all_attentions = all_attentions + (attentions,)
            else:
                if len(layer_outputs) != 1:
                    raise ValueError(f"The length of the layer_outputs should be 1, but it is {len(layer_outputs)}")

        # Add last layer
        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_state,)

        if not return_dict:
            return tuple(v for v in [hidden_state, all_hidden_states, all_attentions] if v is not None)
        return BaseModelOutput(
            last_hidden_state=hidden_state, hidden_states=all_hidden_states, attentions=all_attentions
        )

mindnlp.transformers.models.distilbert.modeling_distilbert.Transformer.__init__(config)

Initializes an instance of the Transformer class.

PARAMETER	DESCRIPTION
self	The instance of the Transformer class. TYPE: Transformer
config	A PretrainedConfig object containing configuration parameters for the Transformer. The 'n_layers' attribute of the PretrainedConfig object specifies the number of layers in the Transformer. This parameter is required for setting up the Transformer instance. TYPE: PretrainedConfig

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def __init__(self, config: PretrainedConfig):
    """
    Initializes an instance of the Transformer class.

    Args:
        self (Transformer): The instance of the Transformer class.
        config (PretrainedConfig): A PretrainedConfig object containing configuration parameters for the Transformer.
            The 'n_layers' attribute of the PretrainedConfig object specifies the number of layers in the Transformer.
            This parameter is required for setting up the Transformer instance.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.n_layers = config.n_layers
    self.layer = nn.ModuleList([TransformerBlock(config) for _ in range(config.n_layers)])
    self.gradient_checkpointing = False

mindnlp.transformers.models.distilbert.modeling_distilbert.Transformer.forward(x, attn_mask=None, head_mask=None, output_attentions=False, output_hidden_states=False, return_dict=None)

PARAMETER	DESCRIPTION
x	mindspore.Tensor(bs, seq_length, dim) Input sequence embedded. TYPE: Tensor
attn_mask	mindspore.Tensor(bs, seq_length) Attention mask on the sequence. TYPE: Optional[Tensor] DEFAULT: None

RETURNS	DESCRIPTION
hidden_state	mindspore.Tensor(bs, seq_length, dim) Sequence of hidden states in the last (top) TYPE: Union[BaseModelOutput, Tuple[Tensor, ...]]
Union[BaseModelOutput, Tuple[Tensor, ...]]	layer all_hidden_states: Tuple[mindspore.Tensor(bs, seq_length, dim)] Tuple of length n_layers with the hidden states from each layer. Optional: only if output_hidden_states=True
all_attentions	Tuple[mindspore.Tensor(bs, n_heads, seq_length, seq_length)] Tuple of length n_layers with the attention weights from each layer Optional: only if output_attentions=True TYPE: Union[BaseModelOutput, Tuple[Tensor, ...]]

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def forward(
    self,
    x: mindspore.Tensor,
    attn_mask: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    output_attentions: bool = False,
    output_hidden_states: bool = False,
    return_dict: Optional[bool] = None,
) -> Union[BaseModelOutput, Tuple[mindspore.Tensor, ...]]:  # docstyle-ignore
    """
    Parameters:
        x: mindspore.Tensor(bs, seq_length, dim) Input sequence embedded.
        attn_mask: mindspore.Tensor(bs, seq_length) Attention mask on the sequence.

    Returns:
        hidden_state: mindspore.Tensor(bs, seq_length, dim) Sequence of hidden states in the last (top)
        layer all_hidden_states: Tuple[mindspore.Tensor(bs, seq_length, dim)]
            Tuple of length n_layers with the hidden states from each layer.
            Optional: only if output_hidden_states=True
        all_attentions: Tuple[mindspore.Tensor(bs, n_heads, seq_length, seq_length)]
            Tuple of length n_layers with the attention weights from each layer
            Optional: only if output_attentions=True
    """
    all_hidden_states = () if output_hidden_states else None
    all_attentions = () if output_attentions else None

    hidden_state = x
    for i, layer_module in enumerate(self.layer):
        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_state,)

        layer_outputs = layer_module(
            hidden_state,
            attn_mask,
            head_mask[i],
            output_attentions,
        )

        hidden_state = layer_outputs[-1]

        if output_attentions:
            if len(layer_outputs) != 2:
                raise ValueError(f"The length of the layer_outputs should be 2, but it is {len(layer_outputs)}")

            attentions = layer_outputs[0]
            all_attentions = all_attentions + (attentions,)
        else:
            if len(layer_outputs) != 1:
                raise ValueError(f"The length of the layer_outputs should be 1, but it is {len(layer_outputs)}")

    # Add last layer
    if output_hidden_states:
        all_hidden_states = all_hidden_states + (hidden_state,)

    if not return_dict:
        return tuple(v for v in [hidden_state, all_hidden_states, all_attentions] if v is not None)
    return BaseModelOutput(
        last_hidden_state=hidden_state, hidden_states=all_hidden_states, attentions=all_attentions
    )

mindnlp.transformers.models.distilbert.modeling_distilbert.TransformerBlock

Bases: Module

The TransformerBlock class represents a single block of the Transformer architecture. It is used to process the input data and produce contextualized outputs.

This class inherits from the nn.Module class.

METHOD

DESCRIPTION

forward

Constructs the TransformerBlock using the given input tensors and parameters. Parameters: x: mindspore.Tensor(bs, seq_length, dim). The input tensor representing the data to be processed. attn_mask: Optional[mindspore.Tensor]. An optional attention mask tensor of shape (bs, seq_length) to mask out specific positions in the input tensor. head_mask: Optional[mindspore.Tensor]. An optional head mask tensor to mask out specific heads of the attention mechanism. output_attentions: bool. A flag indicating whether to return the attention weights in the output. Returns: Tuple[mindspore.Tensor, ...]. A tuple containing the following elements: sa_weights: mindspore.Tensor(bs, n_heads, seq_length, seq_length). The attention weights. ffn_output: mindspore.Tensor(bs, seq_length, dim). The output of the transformer block contextualization. Raises: TypeError: If sa_output is not a tuple when output_attentions is True.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

class TransformerBlock(nn.Module):

    """
    The TransformerBlock class represents a single block of the Transformer architecture.
    It is used to process the input data and produce contextualized outputs.

    This class inherits from the nn.Module class.

    Methods:
        forward:
            Constructs the TransformerBlock using the given input tensors and parameters.

            Parameters:

            - x: mindspore.Tensor(bs, seq_length, dim). The input tensor representing the data to be processed.
            - attn_mask: Optional[mindspore.Tensor]. An optional attention mask tensor of shape (bs, seq_length) to mask out specific positions in the input tensor.
            - head_mask: Optional[mindspore.Tensor]. An optional head mask tensor to mask out specific heads of the attention mechanism.
            - output_attentions: bool. A flag indicating whether to return the attention weights in the output.

            Returns: Tuple[mindspore.Tensor, ...]. A tuple containing the following elements:

            - sa_weights: mindspore.Tensor(bs, n_heads, seq_length, seq_length). The attention weights.
            - ffn_output: mindspore.Tensor(bs, seq_length, dim). The output of the transformer block contextualization.

            Raises:

            - TypeError: If sa_output is not a tuple when output_attentions is True.
    """
    def __init__(self, config: PretrainedConfig):
        """
        Initialize a TransformerBlock instance with the provided configuration.

        Args:
            self (TransformerBlock): The instance of the TransformerBlock class.
            config (PretrainedConfig): The configuration object containing parameters for the TransformerBlock.
                It must be an instance of PretrainedConfig class.
                The 'dim' parameter specifies the dimensionality of the input data.
                The 'n_heads' parameter specifies the number of attention heads to use.
                'n_heads' must be a factor of 'dim' for proper division.

        Returns:
            None.

        Raises:
            ValueError: If the division of 'dim' by 'n_heads' results in a non-zero remainder,
                indicating that 'n_heads' does not evenly divide 'dim'.
        """
        super().__init__()

        # Have an even number of Configure multi-heads
        if config.dim % config.n_heads != 0:
            raise ValueError(f"config.n_heads {config.n_heads} must divide config.dim {config.dim} evenly")

        self.attention = DISTILBERT_ATTENTION_CLASSES["eager"](config)
        self.sa_layer_norm = nn.LayerNorm(normalized_shape=config.dim, eps=1e-12)

        self.ffn = FFN(config)
        self.output_layer_norm = nn.LayerNorm(normalized_shape=config.dim, eps=1e-12)

    def forward(
        self,
        x: mindspore.Tensor,
        attn_mask: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        output_attentions: bool = False,
    ) -> Tuple[mindspore.Tensor, ...]:
        """
        Parameters:
            x: mindspore.Tensor(bs, seq_length, dim)
            attn_mask: mindspore.Tensor(bs, seq_length)

        Returns:
            sa_weights (mindspore.Tensor(bs, n_heads, seq_length, seq_length)): The attention weights
            ffn_output (mindspore.Tensor(bs, seq_length, dim)): The output of the transformer block contextualization.
        """
        # Self-Attention
        sa_output = self.attention(
            query=x,
            key=x,
            value=x,
            mask=attn_mask,
            head_mask=head_mask,
            output_attentions=output_attentions,
        )
        if output_attentions:
            sa_output, sa_weights = sa_output  # (bs, seq_length, dim), (bs, n_heads, seq_length, seq_length)
        else:  # To handle these `output_attentions` or `output_hidden_states` cases returning tuples
            if not isinstance(sa_output, tuple):
                raise TypeError(f"sa_output must be a tuple but it is {type(sa_output)} type")

            sa_output = sa_output[0]
        sa_output = self.sa_layer_norm(sa_output + x)  # (bs, seq_length, dim)

        # Feed Forward Network
        ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
        ffn_output: mindspore.Tensor = self.output_layer_norm(ffn_output + sa_output)  # (bs, seq_length, dim)

        output = (ffn_output,)
        if output_attentions:
            output = (sa_weights,) + output
        return output

mindnlp.transformers.models.distilbert.modeling_distilbert.TransformerBlock.__init__(config)

Initialize a TransformerBlock instance with the provided configuration.

PARAMETER	DESCRIPTION
self	The instance of the TransformerBlock class. TYPE: TransformerBlock
config	The configuration object containing parameters for the TransformerBlock. It must be an instance of PretrainedConfig class. The 'dim' parameter specifies the dimensionality of the input data. The 'n_heads' parameter specifies the number of attention heads to use. 'n_heads' must be a factor of 'dim' for proper division. TYPE: PretrainedConfig

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If the division of 'dim' by 'n_heads' results in a non-zero remainder, indicating that 'n_heads' does not evenly divide 'dim'.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def __init__(self, config: PretrainedConfig):
    """
    Initialize a TransformerBlock instance with the provided configuration.

    Args:
        self (TransformerBlock): The instance of the TransformerBlock class.
        config (PretrainedConfig): The configuration object containing parameters for the TransformerBlock.
            It must be an instance of PretrainedConfig class.
            The 'dim' parameter specifies the dimensionality of the input data.
            The 'n_heads' parameter specifies the number of attention heads to use.
            'n_heads' must be a factor of 'dim' for proper division.

    Returns:
        None.

    Raises:
        ValueError: If the division of 'dim' by 'n_heads' results in a non-zero remainder,
            indicating that 'n_heads' does not evenly divide 'dim'.
    """
    super().__init__()

    # Have an even number of Configure multi-heads
    if config.dim % config.n_heads != 0:
        raise ValueError(f"config.n_heads {config.n_heads} must divide config.dim {config.dim} evenly")

    self.attention = DISTILBERT_ATTENTION_CLASSES["eager"](config)
    self.sa_layer_norm = nn.LayerNorm(normalized_shape=config.dim, eps=1e-12)

    self.ffn = FFN(config)
    self.output_layer_norm = nn.LayerNorm(normalized_shape=config.dim, eps=1e-12)

mindnlp.transformers.models.distilbert.modeling_distilbert.TransformerBlock.forward(x, attn_mask=None, head_mask=None, output_attentions=False)

PARAMETER	DESCRIPTION
x	mindspore.Tensor(bs, seq_length, dim) TYPE: Tensor
attn_mask	mindspore.Tensor(bs, seq_length) TYPE: Optional[Tensor] DEFAULT: None

RETURNS	DESCRIPTION
sa_weights	The attention weights TYPE: Tensor(bs, n_heads, seq_length, seq_length)
ffn_output	The output of the transformer block contextualization. TYPE: Tensor(bs, seq_length, dim)

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def forward(
    self,
    x: mindspore.Tensor,
    attn_mask: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    output_attentions: bool = False,
) -> Tuple[mindspore.Tensor, ...]:
    """
    Parameters:
        x: mindspore.Tensor(bs, seq_length, dim)
        attn_mask: mindspore.Tensor(bs, seq_length)

    Returns:
        sa_weights (mindspore.Tensor(bs, n_heads, seq_length, seq_length)): The attention weights
        ffn_output (mindspore.Tensor(bs, seq_length, dim)): The output of the transformer block contextualization.
    """
    # Self-Attention
    sa_output = self.attention(
        query=x,
        key=x,
        value=x,
        mask=attn_mask,
        head_mask=head_mask,
        output_attentions=output_attentions,
    )
    if output_attentions:
        sa_output, sa_weights = sa_output  # (bs, seq_length, dim), (bs, n_heads, seq_length, seq_length)
    else:  # To handle these `output_attentions` or `output_hidden_states` cases returning tuples
        if not isinstance(sa_output, tuple):
            raise TypeError(f"sa_output must be a tuple but it is {type(sa_output)} type")

        sa_output = sa_output[0]
    sa_output = self.sa_layer_norm(sa_output + x)  # (bs, seq_length, dim)

    # Feed Forward Network
    ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
    ffn_output: mindspore.Tensor = self.output_layer_norm(ffn_output + sa_output)  # (bs, seq_length, dim)

    output = (ffn_output,)
    if output_attentions:
        output = (sa_weights,) + output
    return output

mindnlp.transformers.models.distilbert.modeling_distilbert.create_sinusoidal_embeddings(n_pos, dim, out)

Create sinusoidal embeddings for given positions and dimensions.

PARAMETER	DESCRIPTION
n_pos	The number of positions for which to create embeddings. TYPE: int
dim	The dimension of the embeddings. TYPE: int
out	The output tensor to store the sinusoidal embeddings. TYPE: Tensor

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/distilbert/modeling_distilbert.py

def create_sinusoidal_embeddings(n_pos: int, dim: int, out: mindspore.Tensor):
    """
    Create sinusoidal embeddings for given positions and dimensions.

    Args:
        n_pos (int): The number of positions for which to create embeddings.
        dim (int): The dimension of the embeddings.
        out (mindspore.Tensor): The output tensor to store the sinusoidal embeddings.

    Returns:
        None.

    Raises:
        None
    """
    position_enc = np.array([[pos / np.power(10000, 2 * (j // 2) / dim) for j in range(dim)] for pos in range(n_pos)])
    out.requires_grad = False
    out[:, 0::2] = mindspore.Tensor(np.sin(position_enc[:, 0::2]))
    out[:, 1::2] = mindspore.Tensor(np.cos(position_enc[:, 1::2]))

mindnlp.transformers.models.distilbert.tokenization_distilbert

Tokenization classes for DistilBERT.

mindnlp.transformers.models.distilbert.tokenization_distilbert.BasicTokenizer

Constructs a BasicTokenizer that will run basic tokenization (punctuation splitting, lower casing, etc.).

PARAMETER	DESCRIPTION
do_lower_case	Whether or not to lowercase the input when tokenizing. TYPE: `bool`, *optional*, defaults to `True` DEFAULT: True
never_split	Collection of tokens which will never be split during tokenization. Only has an effect when do_basic_tokenize=True TYPE: `Iterable`, *optional* DEFAULT: None
tokenize_chinese_chars	Whether or not to tokenize Chinese characters. This should likely be deactivated for Japanese (see this issue). TYPE: `bool`, *optional*, defaults to `True` DEFAULT: True
strip_accents	Whether or not to strip all accents. If this option is not specified, then it will be determined by the value for lowercase (as in the original BERT). TYPE: `bool`, *optional* DEFAULT: None
do_split_on_punc	In some instances we want to skip the basic punctuation splitting so that later tokenization can capture the full context of the words, such as contractions. TYPE: `bool`, *optional*, defaults to `True` DEFAULT: True

Source code in mindnlp/transformers/models/distilbert/tokenization_distilbert.py

class BasicTokenizer:
    """
    Constructs a BasicTokenizer that will run basic tokenization (punctuation splitting, lower casing, etc.).

    Args:
        do_lower_case (`bool`, *optional*, defaults to `True`):
            Whether or not to lowercase the input when tokenizing.
        never_split (`Iterable`, *optional*):
            Collection of tokens which will never be split during tokenization. Only has an effect when
            `do_basic_tokenize=True`
        tokenize_chinese_chars (`bool`, *optional*, defaults to `True`):
            Whether or not to tokenize Chinese characters.

            This should likely be deactivated for Japanese (see this
            [issue](https://github.com/huggingface/transformers/issues/328)).
        strip_accents (`bool`, *optional*):
            Whether or not to strip all accents. If this option is not specified, then it will be determined by the
            value for `lowercase` (as in the original BERT).
        do_split_on_punc (`bool`, *optional*, defaults to `True`):
            In some instances we want to skip the basic punctuation splitting so that later tokenization can capture
            the full context of the words, such as contractions.
    """
    def __init__(
        self,
        do_lower_case=True,
        never_split=None,
        tokenize_chinese_chars=True,
        strip_accents=None,
        do_split_on_punc=True,
    ):
        """
        Initializes an instance of the BasicTokenizer class.

        Args:
            self: The instance of the class.
            do_lower_case (bool, optional): Specifies whether the tokenizer should convert text to lowercase.
                Defaults to True.
            never_split (list, optional): A list of tokens that should never be split. Defaults to None.
            tokenize_chinese_chars (bool, optional): Specifies whether the tokenizer should tokenize Chinese characters.
                Defaults to True.
            strip_accents (None or str, optional): Specifies whether accents should be stripped from tokens.
                Defaults to None.
            do_split_on_punc (bool, optional): Specifies whether the tokenizer should split tokens on punctuation marks.
                Defaults to True.

        Returns:
            None.

        Raises:
            None.
        """
        if never_split is None:
            never_split = []
        self.do_lower_case = do_lower_case
        self.never_split = set(never_split)
        self.tokenize_chinese_chars = tokenize_chinese_chars
        self.strip_accents = strip_accents
        self.do_split_on_punc = do_split_on_punc

    def tokenize(self, text, never_split=None):
        """
        Basic Tokenization of a piece of text. For sub-word tokenization, see WordPieceTokenizer.

        Args:
            never_split (`List[str]`, *optional*):
                Kept for backward compatibility purposes. Now implemented directly at the base class level (see
                [`PreTrainedTokenizer.tokenize`]) List of token not to split.
        """
        # union() returns a new set by concatenating the two sets.
        never_split = self.never_split.union(set(never_split)) if never_split else self.never_split
        text = self._clean_text(text)

        # This was added on November 1st, 2018 for the multilingual and Chinese
        # models. This is also applied to the English models now, but it doesn't
        # matter since the English models were not trained on any Chinese data
        # and generally don't have any Chinese data in them (there are Chinese
        # characters in the vocabulary because Wikipedia does have some Chinese
        # words in the English Wikipedia.).
        if self.tokenize_chinese_chars:
            text = self._tokenize_chinese_chars(text)
        # prevents treating the same character with different unicode codepoints as different characters
        unicode_normalized_text = unicodedata.normalize("NFC", text)
        orig_tokens = whitespace_tokenize(unicode_normalized_text)
        split_tokens = []
        for token in orig_tokens:
            if token not in never_split:
                if self.do_lower_case:
                    token = token.lower()
                    if self.strip_accents is not False:
                        token = self._run_strip_accents(token)
                elif self.strip_accents:
                    token = self._run_strip_accents(token)
            split_tokens.extend(self._run_split_on_punc(token, never_split))

        output_tokens = whitespace_tokenize(" ".join(split_tokens))
        return output_tokens

    def _run_strip_accents(self, text):
        """Strips accents from a piece of text."""
        text = unicodedata.normalize("NFD", text)
        output = []
        for char in text:
            cat = unicodedata.category(char)
            if cat == "Mn":
                continue
            output.append(char)
        return "".join(output)

    def _run_split_on_punc(self, text, never_split=None):
        """Splits punctuation on a piece of text."""
        if not self.do_split_on_punc or (never_split is not None and text in never_split):
            return [text]
        chars = list(text)
        i = 0
        start_new_word = True
        output = []
        while i < len(chars):
            char = chars[i]
            if _is_punctuation(char):
                output.append([char])
                start_new_word = True
            else:
                if start_new_word:
                    output.append([])
                start_new_word = False
                output[-1].append(char)
            i += 1

        return ["".join(x) for x in output]

    def _tokenize_chinese_chars(self, text):
        """Adds whitespace around any CJK character."""
        output = []
        for char in text:
            cp = ord(char)
            if self._is_chinese_char(cp):
                output.append(" ")
                output.append(char)
                output.append(" ")
            else:
                output.append(char)
        return "".join(output)

    def _is_chinese_char(self, cp):
        """Checks whether CP is the codepoint of a CJK character."""
        # This defines a "chinese character" as anything in the CJK Unicode block:
        #   https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block)
        #
        # Note that the CJK Unicode block is NOT all Japanese and Korean characters,
        # despite its name. The modern Korean Hangul alphabet is a different block,
        # as is Japanese Hiragana and Katakana. Those alphabets are used to write
        # space-separated words, so they are not treated specially and handled
        # like the all of the other languages.
        if (
            (cp >= 0x4E00 and cp <= 0x9FFF)
            or (cp >= 0x3400 and cp <= 0x4DBF)  #
            or (cp >= 0x20000 and cp <= 0x2A6DF)  #
            or (cp >= 0x2A700 and cp <= 0x2B73F)  #
            or (cp >= 0x2B740 and cp <= 0x2B81F)  #
            or (cp >= 0x2B820 and cp <= 0x2CEAF)  #
            or (cp >= 0xF900 and cp <= 0xFAFF)
            or (cp >= 0x2F800 and cp <= 0x2FA1F)  #
        ):  #
            return True

        return False

    def _clean_text(self, text):
        """Performs invalid character removal and whitespace cleanup on text."""
        output = []
        for char in text:
            cp = ord(char)
            if cp == 0 or cp == 0xFFFD or _is_control(char):
                continue
            if _is_whitespace(char):
                output.append(" ")
            else:
                output.append(char)
        return "".join(output)

mindnlp.transformers.models.distilbert.tokenization_distilbert.BasicTokenizer.__init__(do_lower_case=True, never_split=None, tokenize_chinese_chars=True, strip_accents=None, do_split_on_punc=True)

Initializes an instance of the BasicTokenizer class.

PARAMETER	DESCRIPTION
self	The instance of the class.
do_lower_case	Specifies whether the tokenizer should convert text to lowercase. Defaults to True. TYPE: bool DEFAULT: True
never_split	A list of tokens that should never be split. Defaults to None. TYPE: list DEFAULT: None
tokenize_chinese_chars	Specifies whether the tokenizer should tokenize Chinese characters. Defaults to True. TYPE: bool DEFAULT: True
strip_accents	Specifies whether accents should be stripped from tokens. Defaults to None. TYPE: None or str DEFAULT: None
do_split_on_punc	Specifies whether the tokenizer should split tokens on punctuation marks. Defaults to True. TYPE: bool DEFAULT: True

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/distilbert/tokenization_distilbert.py

def __init__(
    self,
    do_lower_case=True,
    never_split=None,
    tokenize_chinese_chars=True,
    strip_accents=None,
    do_split_on_punc=True,
):
    """
    Initializes an instance of the BasicTokenizer class.

    Args:
        self: The instance of the class.
        do_lower_case (bool, optional): Specifies whether the tokenizer should convert text to lowercase.
            Defaults to True.
        never_split (list, optional): A list of tokens that should never be split. Defaults to None.
        tokenize_chinese_chars (bool, optional): Specifies whether the tokenizer should tokenize Chinese characters.
            Defaults to True.
        strip_accents (None or str, optional): Specifies whether accents should be stripped from tokens.
            Defaults to None.
        do_split_on_punc (bool, optional): Specifies whether the tokenizer should split tokens on punctuation marks.
            Defaults to True.

    Returns:
        None.

    Raises:
        None.
    """
    if never_split is None:
        never_split = []
    self.do_lower_case = do_lower_case
    self.never_split = set(never_split)
    self.tokenize_chinese_chars = tokenize_chinese_chars
    self.strip_accents = strip_accents
    self.do_split_on_punc = do_split_on_punc

mindnlp.transformers.models.distilbert.tokenization_distilbert.BasicTokenizer.tokenize(text, never_split=None)

Basic Tokenization of a piece of text. For sub-word tokenization, see WordPieceTokenizer.

PARAMETER	DESCRIPTION
never_split	Kept for backward compatibility purposes. Now implemented directly at the base class level (see [PreTrainedTokenizer.tokenize]) List of token not to split. TYPE: `List[str]`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/distilbert/tokenization_distilbert.py

def tokenize(self, text, never_split=None):
    """
    Basic Tokenization of a piece of text. For sub-word tokenization, see WordPieceTokenizer.

    Args:
        never_split (`List[str]`, *optional*):
            Kept for backward compatibility purposes. Now implemented directly at the base class level (see
            [`PreTrainedTokenizer.tokenize`]) List of token not to split.
    """
    # union() returns a new set by concatenating the two sets.
    never_split = self.never_split.union(set(never_split)) if never_split else self.never_split
    text = self._clean_text(text)

    # This was added on November 1st, 2018 for the multilingual and Chinese
    # models. This is also applied to the English models now, but it doesn't
    # matter since the English models were not trained on any Chinese data
    # and generally don't have any Chinese data in them (there are Chinese
    # characters in the vocabulary because Wikipedia does have some Chinese
    # words in the English Wikipedia.).
    if self.tokenize_chinese_chars:
        text = self._tokenize_chinese_chars(text)
    # prevents treating the same character with different unicode codepoints as different characters
    unicode_normalized_text = unicodedata.normalize("NFC", text)
    orig_tokens = whitespace_tokenize(unicode_normalized_text)
    split_tokens = []
    for token in orig_tokens:
        if token not in never_split:
            if self.do_lower_case:
                token = token.lower()
                if self.strip_accents is not False:
                    token = self._run_strip_accents(token)
            elif self.strip_accents:
                token = self._run_strip_accents(token)
        split_tokens.extend(self._run_split_on_punc(token, never_split))

    output_tokens = whitespace_tokenize(" ".join(split_tokens))
    return output_tokens

mindnlp.transformers.models.distilbert.tokenization_distilbert.DistilBertTokenizer

Bases: PreTrainedTokenizer

Construct a DistilBERT tokenizer. Based on WordPiece.

This tokenizer inherits from [PreTrainedTokenizer] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

PARAMETER	DESCRIPTION
vocab_file	File containing the vocabulary. TYPE: `str`
do_lower_case	Whether or not to lowercase the input when tokenizing. TYPE: `bool`, *optional*, defaults to `True` DEFAULT: True
do_basic_tokenize	Whether or not to do basic tokenization before WordPiece. TYPE: `bool`, *optional*, defaults to `True` DEFAULT: True
never_split	Collection of tokens which will never be split during tokenization. Only has an effect when do_basic_tokenize=True TYPE: `Iterable`, *optional* DEFAULT: None
unk_token	The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. TYPE: `str`, *optional*, defaults to `"[UNK]"` DEFAULT: '[UNK]'
sep_token	The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. TYPE: `str`, *optional*, defaults to `"[SEP]"` DEFAULT: '[SEP]'
pad_token	The token used for padding, for example when batching sequences of different lengths. TYPE: `str`, *optional*, defaults to `"[PAD]"` DEFAULT: '[PAD]'
cls_token	The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens. TYPE: `str`, *optional*, defaults to `"[CLS]"` DEFAULT: '[CLS]'
mask_token	The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict. TYPE: `str`, *optional*, defaults to `"[MASK]"` DEFAULT: '[MASK]'
tokenize_chinese_chars	Whether or not to tokenize Chinese characters. This should likely be deactivated for Japanese (see this issue). TYPE: `bool`, *optional*, defaults to `True` DEFAULT: True
strip_accents	Whether or not to strip all accents. If this option is not specified, then it will be determined by the value for lowercase (as in the original BERT). TYPE: `bool`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/distilbert/tokenization_distilbert.py

class DistilBertTokenizer(PreTrainedTokenizer):
    r"""
    Construct a DistilBERT tokenizer. Based on WordPiece.

    This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
    this superclass for more information regarding those methods.

    Args:
        vocab_file (`str`):
            File containing the vocabulary.
        do_lower_case (`bool`, *optional*, defaults to `True`):
            Whether or not to lowercase the input when tokenizing.
        do_basic_tokenize (`bool`, *optional*, defaults to `True`):
            Whether or not to do basic tokenization before WordPiece.
        never_split (`Iterable`, *optional*):
            Collection of tokens which will never be split during tokenization. Only has an effect when
            `do_basic_tokenize=True`
        unk_token (`str`, *optional*, defaults to `"[UNK]"`):
            The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
            token instead.
        sep_token (`str`, *optional*, defaults to `"[SEP]"`):
            The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
            sequence classification or for a text and a question for question answering. It is also used as the last
            token of a sequence built with special tokens.
        pad_token (`str`, *optional*, defaults to `"[PAD]"`):
            The token used for padding, for example when batching sequences of different lengths.
        cls_token (`str`, *optional*, defaults to `"[CLS]"`):
            The classifier token which is used when doing sequence classification (classification of the whole sequence
            instead of per-token classification). It is the first token of the sequence when built with special tokens.
        mask_token (`str`, *optional*, defaults to `"[MASK]"`):
            The token used for masking values. This is the token used when training this model with masked language
            modeling. This is the token which the model will try to predict.
        tokenize_chinese_chars (`bool`, *optional*, defaults to `True`):
            Whether or not to tokenize Chinese characters.

            This should likely be deactivated for Japanese (see this
            [issue](https://github.com/huggingface/transformers/issues/328)).
        strip_accents (`bool`, *optional*):
            Whether or not to strip all accents. If this option is not specified, then it will be determined by the
            value for `lowercase` (as in the original BERT).
    """
    vocab_files_names = VOCAB_FILES_NAMES
    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
    pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION
    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
    model_input_names = ["input_ids", "attention_mask"]

    def __init__(
        self,
        vocab_file,
        do_lower_case=True,
        do_basic_tokenize=True,
        never_split=None,
        unk_token="[UNK]",
        sep_token="[SEP]",
        pad_token="[PAD]",
        cls_token="[CLS]",
        mask_token="[MASK]",
        tokenize_chinese_chars=True,
        strip_accents=None,
        **kwargs,
    ):
        """
        Args:
            self: The instance of the class.
            vocab_file (str): The path to the vocabulary file. If the file does not exist, a ValueError is raised.
            do_lower_case (bool, optional): Flag to indicate whether the tokens should be lower-cased. Defaults to True.
            do_basic_tokenize (bool, optional): Flag to indicate whether basic tokenization should be performed. Defaults to True.
            never_split (list, optional): List of tokens that should never be split. Defaults to None.
            unk_token (str, optional): The token to be used for unknown words. Defaults to '[UNK]'.
            sep_token (str, optional): The token to be used for separation. Defaults to '[SEP]'.
            pad_token (str, optional): The token to be used for padding. Defaults to '[PAD]'.
            cls_token (str, optional): The token to be used for classification. Defaults to '[CLS]'.
            mask_token (str, optional): The token to be used for masking. Defaults to '[MASK]'.
            tokenize_chinese_chars (bool, optional): Flag to indicate whether to tokenize Chinese characters. Defaults to True.
            strip_accents (str, optional): Flag to indicate whether to strip accents. Defaults to None.

        Returns:
            None.

        Raises:
            ValueError: If the vocabulary file specified by 'vocab_file' does not exist.
        """
        if not os.path.isfile(vocab_file):
            raise ValueError(
                f"Can't find a vocabulary file at path '{vocab_file}'. To load the vocabulary from a Google pretrained"
                " model use `tokenizer = DistilBertTokenizer.from_pretrained(PRETRAINED_MODEL_NAME)`"
            )
        self.vocab = load_vocab(vocab_file)
        self.ids_to_tokens = collections.OrderedDict([(ids, tok) for tok, ids in self.vocab.items()])
        self.do_basic_tokenize = do_basic_tokenize
        if do_basic_tokenize:
            self.basic_tokenizer = BasicTokenizer(
                do_lower_case=do_lower_case,
                never_split=never_split,
                tokenize_chinese_chars=tokenize_chinese_chars,
                strip_accents=strip_accents,
            )
        self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab, unk_token=str(unk_token))

        super().__init__(
            do_lower_case=do_lower_case,
            do_basic_tokenize=do_basic_tokenize,
            never_split=never_split,
            unk_token=unk_token,
            sep_token=sep_token,
            pad_token=pad_token,
            cls_token=cls_token,
            mask_token=mask_token,
            tokenize_chinese_chars=tokenize_chinese_chars,
            strip_accents=strip_accents,
            **kwargs,
        )

    @property
    # Copied from transformers.models.bert.tokenization_bert.BertTokenizer.do_lower_case
    def do_lower_case(self):
        """ 
        Method to get the flag indicating if the tokenizer should convert all text to lower case.

        Args:
            self (DistilBertTokenizer): The instance of the DistilBertTokenizer class.

        Returns:
            None.

        Raises:
            None.
        """
        return self.basic_tokenizer.do_lower_case

    @property
    # Copied from transformers.models.bert.tokenization_bert.BertTokenizer.vocab_size
    def vocab_size(self):
        """
        Get the vocabulary size of the DistilBertTokenizer.

        Args:
            self (DistilBertTokenizer): An instance of the DistilBertTokenizer class.

        Returns:
            int: The number of unique tokens in the vocabulary of the tokenizer.

        Raises:
            None.

        """
        return len(self.vocab)

    # Copied from transformers.models.bert.tokenization_bert.BertTokenizer.get_vocab
    def get_vocab(self):
        """
        Returns the vocabulary of the DistilBertTokenizer instance.

        Args:
            self (DistilBertTokenizer): The current instance of the DistilBertTokenizer class.

        Returns:
            dict: A dictionary containing the vocabulary of the tokenizer, including any added tokens.

        Raises:
            None.

        Note:
            The vocabulary is a dictionary that maps tokens to their corresponding IDs. The method combines
            the original vocabulary of the tokenizer with any additional tokens that were added using the
            `add_tokens` method. The resulting dictionary is returned as the output of this method.

        Example:
            ```python
            >>> tokenizer = DistilBertTokenizer()
            >>> vocab = tokenizer.get_vocab()
            >>> print(vocab)
            {'<pad>': 0, '<s>': 1, '</s>': 2, '<unk>': 3, '<mask>': 4, '<cls>': 5, '<sep>': 6, '<eod>': 7, '<eop>': 8}
            ```
        """
        return dict(self.vocab, **self.added_tokens_encoder)

    # Copied from transformers.models.bert.tokenization_bert.BertTokenizer._tokenize
    def _tokenize(self, text, split_special_tokens=False):
        """
        Tokenizes a given text into a list of tokens using the DistilBertTokenizer.

        Args:
            self (DistilBertTokenizer): An instance of the DistilBertTokenizer class.
            text (str): The input text to be tokenized.
            split_special_tokens (bool): Flag indicating whether to split special tokens or not. Default is False.

        Returns:
            list: A list of tokens generated from the input text.

        Raises:
            None.

        Note:
            This method tokenizes the input text by either using the basic tokenizer followed by the wordpiece tokenizer,
            or directly using the wordpiece tokenizer, depending on the value of `do_basic_tokenize` attribute of the tokenizer.
            If `do_basic_tokenize` is True, special tokens are never split unless `split_special_tokens` is True.
            The resulting tokens are returned as a list.

        Example:
            ```python
            >>> tokenizer = DistilBertTokenizer()
            >>> tokens = tokenizer._tokenize("Hello world!")
            >>> print(tokens)
            >>> # Output: ['hello', 'world', '!']
            ```
        """
        split_tokens = []
        if self.do_basic_tokenize:
            for token in self.basic_tokenizer.tokenize(
                text, never_split=self.all_special_tokens if not split_special_tokens else None
            ):
                # If the token is part of the never_split set
                if token in self.basic_tokenizer.never_split:
                    split_tokens.append(token)
                else:
                    split_tokens += self.wordpiece_tokenizer.tokenize(token)
        else:
            split_tokens = self.wordpiece_tokenizer.tokenize(text)
        return split_tokens

    # Copied from transformers.models.bert.tokenization_bert.BertTokenizer._convert_token_to_id
    def _convert_token_to_id(self, token):
        """Converts a token (str) in an id using the vocab."""
        return self.vocab.get(token, self.vocab.get(self.unk_token))

    # Copied from transformers.models.bert.tokenization_bert.BertTokenizer._convert_id_to_token
    def _convert_id_to_token(self, index):
        """Converts an index (integer) in a token (str) using the vocab."""
        return self.ids_to_tokens.get(index, self.unk_token)

    # Copied from transformers.models.bert.tokenization_bert.BertTokenizer.convert_tokens_to_string
    def convert_tokens_to_string(self, tokens):
        """Converts a sequence of tokens (string) in a single string."""
        out_string = " ".join(tokens).replace(" ##", "").strip()
        return out_string

    # Copied from transformers.models.bert.tokenization_bert.BertTokenizer.build_inputs_with_special_tokens
    def build_inputs_with_special_tokens(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
    ) -> List[int]:
        """
        Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
        adding special tokens. A BERT sequence has the following format:

        - single sequence: `[CLS] X [SEP]`
        - pair of sequences: `[CLS] A [SEP] B [SEP]`

        Args:
            token_ids_0 (`List[int]`):
                List of IDs to which the special tokens will be added.
            token_ids_1 (`List[int]`, *optional*):
                Optional second list of IDs for sequence pairs.

        Returns:
            `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
        """
        if token_ids_1 is None:
            return [self.cls_token_id] + token_ids_0 + [self.sep_token_id]
        cls = [self.cls_token_id]
        sep = [self.sep_token_id]
        return cls + token_ids_0 + sep + token_ids_1 + sep

    # Copied from transformers.models.bert.tokenization_bert.BertTokenizer.get_special_tokens_mask
    def get_special_tokens_mask(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False
    ) -> List[int]:
        """
        Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
        special tokens using the tokenizer `prepare_for_model` method.

        Args:
            token_ids_0 (`List[int]`):
                List of IDs.
            token_ids_1 (`List[int]`, *optional*):
                Optional second list of IDs for sequence pairs.
            already_has_special_tokens (`bool`, *optional*, defaults to `False`):
                Whether or not the token list is already formatted with special tokens for the model.

        Returns:
            `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
        """
        if already_has_special_tokens:
            return super().get_special_tokens_mask(
                token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True
            )

        if token_ids_1 is not None:
            return [1] + ([0] * len(token_ids_0)) + [1] + ([0] * len(token_ids_1)) + [1]
        return [1] + ([0] * len(token_ids_0)) + [1]

    # Copied from transformers.models.bert.tokenization_bert.BertTokenizer.create_token_type_ids_from_sequences
    def create_token_type_ids_from_sequences(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
    ) -> List[int]:
        """
        Create a mask from the two sequences passed to be used in a sequence-pair classification task. A BERT sequence
        pair mask has the following format:

        ```
        0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
        | first sequence    | second sequence |
        ```

        If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s).

        Args:
            token_ids_0 (`List[int]`):
                List of IDs.
            token_ids_1 (`List[int]`, *optional*):
                Optional second list of IDs for sequence pairs.

        Returns:
            `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s).
        """
        sep = [self.sep_token_id]
        cls = [self.cls_token_id]
        if token_ids_1 is None:
            return len(cls + token_ids_0 + sep) * [0]
        return len(cls + token_ids_0 + sep) * [0] + len(token_ids_1 + sep) * [1]

    # Copied from transformers.models.bert.tokenization_bert.BertTokenizer.save_vocabulary
    def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
        """
        Saves the vocabulary of the DistilBertTokenizer to a file.

        Args:
            self (DistilBertTokenizer): An instance of the DistilBertTokenizer class.
            save_directory (str): The directory where the vocabulary file will be saved.
            filename_prefix (Optional[str], optional): An optional prefix to prepend to the filename. Defaults to None.

        Returns:
            Tuple[str]: A tuple containing the path of the saved vocabulary file.

        Raises:
            None.

        Note:
            The vocabulary file will be saved in the specified directory with the filename forwarded using the provided
            filename_prefix (if any) and the default vocabulary file name ('vocab.txt'). If the save_directory is not a valid
            directory, the vocabulary file will be saved with the provided save_directory as the filename.

            The vocabulary file will contain each token in the vocabulary, separated by a newline character. The tokens will be
            sorted based on their token indices. If the token indices are not consecutive, a warning message will be logged.
            This can indicate a corrupted vocabulary.

        Example:
            ```python
            >>> tokenizer = DistilBertTokenizer()
            >>> tokenizer.save_vocabulary('/path/to/save', 'my_model')
            ```
        """
        index = 0
        if os.path.isdir(save_directory):
            vocab_file = os.path.join(
                save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
            )
        else:
            vocab_file = (filename_prefix + "-" if filename_prefix else "") + save_directory
        with open(vocab_file, "w", encoding="utf-8") as writer:
            for token, token_index in sorted(self.vocab.items(), key=lambda kv: kv[1]):
                if index != token_index:
                    logger.warning(
                        f"Saving vocabulary to {vocab_file}: vocabulary indices are not consecutive."
                        " Please check that the vocabulary is not corrupted!"
                    )
                    index = token_index
                writer.write(token + "\n")
                index += 1
        return (vocab_file,)

mindnlp.transformers.models.distilbert.tokenization_distilbert.DistilBertTokenizer.do_lower_case property

Method to get the flag indicating if the tokenizer should convert all text to lower case.

PARAMETER	DESCRIPTION
self	The instance of the DistilBertTokenizer class. TYPE: DistilBertTokenizer

RETURNS	DESCRIPTION
	None.

mindnlp.transformers.models.distilbert.tokenization_distilbert.DistilBertTokenizer.vocab_size property

Get the vocabulary size of the DistilBertTokenizer.

PARAMETER	DESCRIPTION
self	An instance of the DistilBertTokenizer class. TYPE: DistilBertTokenizer

RETURNS	DESCRIPTION
int	The number of unique tokens in the vocabulary of the tokenizer.

mindnlp.transformers.models.distilbert.tokenization_distilbert.DistilBertTokenizer.__init__(vocab_file, do_lower_case=True, do_basic_tokenize=True, never_split=None, unk_token='[UNK]', sep_token='[SEP]', pad_token='[PAD]', cls_token='[CLS]', mask_token='[MASK]', tokenize_chinese_chars=True, strip_accents=None, **kwargs)

PARAMETER	DESCRIPTION
self	The instance of the class.
vocab_file	The path to the vocabulary file. If the file does not exist, a ValueError is raised. TYPE: str
do_lower_case	Flag to indicate whether the tokens should be lower-cased. Defaults to True. TYPE: bool DEFAULT: True
do_basic_tokenize	Flag to indicate whether basic tokenization should be performed. Defaults to True. TYPE: bool DEFAULT: True
never_split	List of tokens that should never be split. Defaults to None. TYPE: list DEFAULT: None
unk_token	The token to be used for unknown words. Defaults to '[UNK]'. TYPE: str DEFAULT: '[UNK]'
sep_token	The token to be used for separation. Defaults to '[SEP]'. TYPE: str DEFAULT: '[SEP]'
pad_token	The token to be used for padding. Defaults to '[PAD]'. TYPE: str DEFAULT: '[PAD]'
cls_token	The token to be used for classification. Defaults to '[CLS]'. TYPE: str DEFAULT: '[CLS]'
mask_token	The token to be used for masking. Defaults to '[MASK]'. TYPE: str DEFAULT: '[MASK]'
tokenize_chinese_chars	Flag to indicate whether to tokenize Chinese characters. Defaults to True. TYPE: bool DEFAULT: True
strip_accents	Flag to indicate whether to strip accents. Defaults to None. TYPE: str DEFAULT: None

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If the vocabulary file specified by 'vocab_file' does not exist.

Source code in mindnlp/transformers/models/distilbert/tokenization_distilbert.py

def __init__(
    self,
    vocab_file,
    do_lower_case=True,
    do_basic_tokenize=True,
    never_split=None,
    unk_token="[UNK]",
    sep_token="[SEP]",
    pad_token="[PAD]",
    cls_token="[CLS]",
    mask_token="[MASK]",
    tokenize_chinese_chars=True,
    strip_accents=None,
    **kwargs,
):
    """
    Args:
        self: The instance of the class.
        vocab_file (str): The path to the vocabulary file. If the file does not exist, a ValueError is raised.
        do_lower_case (bool, optional): Flag to indicate whether the tokens should be lower-cased. Defaults to True.
        do_basic_tokenize (bool, optional): Flag to indicate whether basic tokenization should be performed. Defaults to True.
        never_split (list, optional): List of tokens that should never be split. Defaults to None.
        unk_token (str, optional): The token to be used for unknown words. Defaults to '[UNK]'.
        sep_token (str, optional): The token to be used for separation. Defaults to '[SEP]'.
        pad_token (str, optional): The token to be used for padding. Defaults to '[PAD]'.
        cls_token (str, optional): The token to be used for classification. Defaults to '[CLS]'.
        mask_token (str, optional): The token to be used for masking. Defaults to '[MASK]'.
        tokenize_chinese_chars (bool, optional): Flag to indicate whether to tokenize Chinese characters. Defaults to True.
        strip_accents (str, optional): Flag to indicate whether to strip accents. Defaults to None.

    Returns:
        None.

    Raises:
        ValueError: If the vocabulary file specified by 'vocab_file' does not exist.
    """
    if not os.path.isfile(vocab_file):
        raise ValueError(
            f"Can't find a vocabulary file at path '{vocab_file}'. To load the vocabulary from a Google pretrained"
            " model use `tokenizer = DistilBertTokenizer.from_pretrained(PRETRAINED_MODEL_NAME)`"
        )
    self.vocab = load_vocab(vocab_file)
    self.ids_to_tokens = collections.OrderedDict([(ids, tok) for tok, ids in self.vocab.items()])
    self.do_basic_tokenize = do_basic_tokenize
    if do_basic_tokenize:
        self.basic_tokenizer = BasicTokenizer(
            do_lower_case=do_lower_case,
            never_split=never_split,
            tokenize_chinese_chars=tokenize_chinese_chars,
            strip_accents=strip_accents,
        )
    self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab, unk_token=str(unk_token))

    super().__init__(
        do_lower_case=do_lower_case,
        do_basic_tokenize=do_basic_tokenize,
        never_split=never_split,
        unk_token=unk_token,
        sep_token=sep_token,
        pad_token=pad_token,
        cls_token=cls_token,
        mask_token=mask_token,
        tokenize_chinese_chars=tokenize_chinese_chars,
        strip_accents=strip_accents,
        **kwargs,
    )

mindnlp.transformers.models.distilbert.tokenization_distilbert.DistilBertTokenizer.build_inputs_with_special_tokens(token_ids_0, token_ids_1=None)

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BERT sequence has the following format:

• single sequence: [CLS] X [SEP]
• pair of sequences: [CLS] A [SEP] B [SEP]

PARAMETER	DESCRIPTION
token_ids_0	List of IDs to which the special tokens will be added. TYPE: `List[int]`
token_ids_1	Optional second list of IDs for sequence pairs. TYPE: `List[int]`, *optional* DEFAULT: None

RETURNS	DESCRIPTION
List[int]	List[int]: List of input IDs with the appropriate special tokens.

Source code in mindnlp/transformers/models/distilbert/tokenization_distilbert.py

def build_inputs_with_special_tokens(
    self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
) -> List[int]:
    """
    Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
    adding special tokens. A BERT sequence has the following format:

    - single sequence: `[CLS] X [SEP]`
    - pair of sequences: `[CLS] A [SEP] B [SEP]`

    Args:
        token_ids_0 (`List[int]`):
            List of IDs to which the special tokens will be added.
        token_ids_1 (`List[int]`, *optional*):
            Optional second list of IDs for sequence pairs.

    Returns:
        `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
    """
    if token_ids_1 is None:
        return [self.cls_token_id] + token_ids_0 + [self.sep_token_id]
    cls = [self.cls_token_id]
    sep = [self.sep_token_id]
    return cls + token_ids_0 + sep + token_ids_1 + sep

mindnlp.transformers.models.distilbert.tokenization_distilbert.DistilBertTokenizer.convert_tokens_to_string(tokens)

Converts a sequence of tokens (string) in a single string.

Source code in mindnlp/transformers/models/distilbert/tokenization_distilbert.py

def convert_tokens_to_string(self, tokens):
    """Converts a sequence of tokens (string) in a single string."""
    out_string = " ".join(tokens).replace(" ##", "").strip()
    return out_string

mindnlp.transformers.models.distilbert.tokenization_distilbert.DistilBertTokenizer.create_token_type_ids_from_sequences(token_ids_0, token_ids_1=None)

Create a mask from the two sequences passed to be used in a sequence-pair classification task. A BERT sequence pair mask has the following format:

0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
| first sequence    | second sequence |

If token_ids_1 is None, this method only returns the first portion of the mask (0s).

PARAMETER	DESCRIPTION
token_ids_0	List of IDs. TYPE: `List[int]`
token_ids_1	Optional second list of IDs for sequence pairs. TYPE: `List[int]`, *optional* DEFAULT: None

RETURNS	DESCRIPTION
List[int]	List[int]: List of token type IDs according to the given sequence(s).

Source code in mindnlp/transformers/models/distilbert/tokenization_distilbert.py

def create_token_type_ids_from_sequences(
    self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
) -> List[int]:
    """
    Create a mask from the two sequences passed to be used in a sequence-pair classification task. A BERT sequence
    pair mask has the following format:

    ```
    0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
    | first sequence    | second sequence |
    ```

    If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s).

    Args:
        token_ids_0 (`List[int]`):
            List of IDs.
        token_ids_1 (`List[int]`, *optional*):
            Optional second list of IDs for sequence pairs.

    Returns:
        `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s).
    """
    sep = [self.sep_token_id]
    cls = [self.cls_token_id]
    if token_ids_1 is None:
        return len(cls + token_ids_0 + sep) * [0]
    return len(cls + token_ids_0 + sep) * [0] + len(token_ids_1 + sep) * [1]

mindnlp.transformers.models.distilbert.tokenization_distilbert.DistilBertTokenizer.get_special_tokens_mask(token_i