Skip to content

ernie

mindnlp.transformers.models.ernie.modeling_ernie

MindSpore ERNIE model.

mindnlp.transformers.models.ernie.modeling_ernie.ErnieAttention

Bases: Module

This class represents the ErnieAttention module, which is a part of the ERNIE (Enhanced Representation through kNowledge Integration) model. The ErnieAttention module is used for self-attention mechanism and output processing. It includes methods for head pruning and attention forwardion. This class inherits from nn.Module and is designed to be used within the ERNIE model architecture for natural language processing tasks.

Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
class ErnieAttention(nn.Module):

    '''
    This class represents the ErnieAttention module, which is a part of the ERNIE
    (Enhanced Representation through kNowledge Integration) model.
    The ErnieAttention module is used for self-attention mechanism and output processing.
    It includes methods for head pruning and attention forwardion.
    This class inherits from nn.Module and is designed to be used within the ERNIE model architecture for
    natural language processing tasks.
    '''
    def __init__(self, config, position_embedding_type=None):
        """
        Initializes an instance of the ErnieAttention class.

        Args:
            self (object): The instance of the class.
            config (object): The configuration object containing the model's settings and hyperparameters.
            position_embedding_type (str, optional): The type of position embedding to be used. Defaults to None.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.self = ErnieSelfAttention(config, position_embedding_type=position_embedding_type)
        self.output = ErnieSelfOutput(config)
        self.pruned_heads = set()

    def prune_heads(self, heads):
        """
        This method 'prune_heads' is defined within the class 'ErnieAttention' and is responsible for
        pruning the attention heads based on the provided 'heads' parameter.

        Args:
            self (ErnieAttention): The instance of the ErnieAttention class.
                This parameter represents the instance of the ErnieAttention class which contains the attention heads to be pruned.

            heads (list): A list of integers representing the indices of attention heads to be pruned.
                This parameter specifies the indices of the attention heads that need to be pruned from the model.

        Returns:
            None: This method does not return any value.
                It operates by modifying the attributes of the ErnieAttention instance in-place.

        Raises:
            None.
        """
        if len(heads) == 0:
            return
        heads, index = find_pruneable_heads_and_indices(
            heads, self.self.num_attention_heads, self.self.attention_head_size, self.pruned_heads
        )

        # Prune linear layers
        self.self.query = prune_linear_layer(self.self.query, index)
        self.self.key = prune_linear_layer(self.self.key, index)
        self.self.value = prune_linear_layer(self.self.value, index)
        self.output.dense = prune_linear_layer(self.output.dense, index, dim=1)

        # Update hyper params and store pruned heads
        self.self.num_attention_heads = self.self.num_attention_heads - len(heads)
        self.self.all_head_size = self.self.attention_head_size * self.self.num_attention_heads
        self.pruned_heads = self.pruned_heads.union(heads)

    def forward(
        self,
        hidden_states: mindspore.Tensor,
        attention_mask: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        encoder_hidden_states: Optional[mindspore.Tensor] = None,
        encoder_attention_mask: Optional[mindspore.Tensor] = None,
        past_key_value: Optional[Tuple[Tuple[mindspore.Tensor]]] = None,
        output_attentions: Optional[bool] = False,
    ) -> Tuple[mindspore.Tensor]:
        """
        This method forwards the attention mechanism for the Ernie model.

        Args:
            self (ErnieAttention): The instance of the ErnieAttention class.
            hidden_states (mindspore.Tensor): The input hidden states for the attention mechanism.
            attention_mask (Optional[mindspore.Tensor]): An optional mask tensor for the attention scores.
                Defaults to None.
            head_mask (Optional[mindspore.Tensor]): An optional mask tensor for controlling the attention heads.
                Defaults to None.
            encoder_hidden_states (Optional[mindspore.Tensor]):
                An optional tensor containing the hidden states of the encoder. Defaults to None.
            encoder_attention_mask (Optional[mindspore.Tensor]):
                An optional mask tensor for the encoder attention scores. Defaults to None.
            past_key_value (Optional[Tuple[Tuple[mindspore.Tensor]]]):
                An optional tuple containing the past key and value tensors. Defaults to None.
            output_attentions (Optional[bool]): A flag indicating whether to output attentions. Defaults to False.

        Returns:
            Tuple[mindspore.Tensor]:
                A tuple containing the attention output tensor and any additional outputs from the attention mechanism.

        Raises:
            None
        """
        self_outputs = self.self(
            hidden_states,
            attention_mask,
            head_mask,
            encoder_hidden_states,
            encoder_attention_mask,
            past_key_value,
            output_attentions,
        )
        attention_output = self.output(self_outputs[0], hidden_states)
        outputs = (attention_output,) + self_outputs[1:]  # add attentions if we output them
        return outputs

mindnlp.transformers.models.ernie.modeling_ernie.ErnieAttention.__init__(config, position_embedding_type=None)

Initializes an instance of the ErnieAttention class.

PARAMETER DESCRIPTION
self

The instance of the class.

TYPE: object

config

The configuration object containing the model's settings and hyperparameters.

TYPE: object

position_embedding_type

The type of position embedding to be used. Defaults to None.

TYPE: str DEFAULT: None

RETURNS DESCRIPTION

None.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
def __init__(self, config, position_embedding_type=None):
    """
    Initializes an instance of the ErnieAttention class.

    Args:
        self (object): The instance of the class.
        config (object): The configuration object containing the model's settings and hyperparameters.
        position_embedding_type (str, optional): The type of position embedding to be used. Defaults to None.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.self = ErnieSelfAttention(config, position_embedding_type=position_embedding_type)
    self.output = ErnieSelfOutput(config)
    self.pruned_heads = set()

mindnlp.transformers.models.ernie.modeling_ernie.ErnieAttention.forward(hidden_states, attention_mask=None, head_mask=None, encoder_hidden_states=None, encoder_attention_mask=None, past_key_value=None, output_attentions=False)

This method forwards the attention mechanism for the Ernie model.

PARAMETER DESCRIPTION
self

The instance of the ErnieAttention class.

TYPE: ErnieAttention

hidden_states

The input hidden states for the attention mechanism.

TYPE: Tensor

attention_mask

An optional mask tensor for the attention scores. Defaults to None.

TYPE: Optional[Tensor] DEFAULT: None

head_mask

An optional mask tensor for controlling the attention heads. Defaults to None.

TYPE: Optional[Tensor] DEFAULT: None

encoder_hidden_states

An optional tensor containing the hidden states of the encoder. Defaults to None.

TYPE: Optional[Tensor] DEFAULT: None

encoder_attention_mask

An optional mask tensor for the encoder attention scores. Defaults to None.

TYPE: Optional[Tensor] DEFAULT: None

past_key_value

An optional tuple containing the past key and value tensors. Defaults to None.

TYPE: Optional[Tuple[Tuple[Tensor]]] DEFAULT: None

output_attentions

A flag indicating whether to output attentions. Defaults to False.

TYPE: Optional[bool] DEFAULT: False

RETURNS DESCRIPTION
Tuple[Tensor]

Tuple[mindspore.Tensor]: A tuple containing the attention output tensor and any additional outputs from the attention mechanism.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
def forward(
    self,
    hidden_states: mindspore.Tensor,
    attention_mask: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    encoder_hidden_states: Optional[mindspore.Tensor] = None,
    encoder_attention_mask: Optional[mindspore.Tensor] = None,
    past_key_value: Optional[Tuple[Tuple[mindspore.Tensor]]] = None,
    output_attentions: Optional[bool] = False,
) -> Tuple[mindspore.Tensor]:
    """
    This method forwards the attention mechanism for the Ernie model.

    Args:
        self (ErnieAttention): The instance of the ErnieAttention class.
        hidden_states (mindspore.Tensor): The input hidden states for the attention mechanism.
        attention_mask (Optional[mindspore.Tensor]): An optional mask tensor for the attention scores.
            Defaults to None.
        head_mask (Optional[mindspore.Tensor]): An optional mask tensor for controlling the attention heads.
            Defaults to None.
        encoder_hidden_states (Optional[mindspore.Tensor]):
            An optional tensor containing the hidden states of the encoder. Defaults to None.
        encoder_attention_mask (Optional[mindspore.Tensor]):
            An optional mask tensor for the encoder attention scores. Defaults to None.
        past_key_value (Optional[Tuple[Tuple[mindspore.Tensor]]]):
            An optional tuple containing the past key and value tensors. Defaults to None.
        output_attentions (Optional[bool]): A flag indicating whether to output attentions. Defaults to False.

    Returns:
        Tuple[mindspore.Tensor]:
            A tuple containing the attention output tensor and any additional outputs from the attention mechanism.

    Raises:
        None
    """
    self_outputs = self.self(
        hidden_states,
        attention_mask,
        head_mask,
        encoder_hidden_states,
        encoder_attention_mask,
        past_key_value,
        output_attentions,
    )
    attention_output = self.output(self_outputs[0], hidden_states)
    outputs = (attention_output,) + self_outputs[1:]  # add attentions if we output them
    return outputs

mindnlp.transformers.models.ernie.modeling_ernie.ErnieAttention.prune_heads(heads)

This method 'prune_heads' is defined within the class 'ErnieAttention' and is responsible for pruning the attention heads based on the provided 'heads' parameter.

PARAMETER DESCRIPTION
self

The instance of the ErnieAttention class. This parameter represents the instance of the ErnieAttention class which contains the attention heads to be pruned.

TYPE: ErnieAttention

heads

A list of integers representing the indices of attention heads to be pruned. This parameter specifies the indices of the attention heads that need to be pruned from the model.

TYPE: list

RETURNS DESCRIPTION
None

This method does not return any value. It operates by modifying the attributes of the ErnieAttention instance in-place.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
def prune_heads(self, heads):
    """
    This method 'prune_heads' is defined within the class 'ErnieAttention' and is responsible for
    pruning the attention heads based on the provided 'heads' parameter.

    Args:
        self (ErnieAttention): The instance of the ErnieAttention class.
            This parameter represents the instance of the ErnieAttention class which contains the attention heads to be pruned.

        heads (list): A list of integers representing the indices of attention heads to be pruned.
            This parameter specifies the indices of the attention heads that need to be pruned from the model.

    Returns:
        None: This method does not return any value.
            It operates by modifying the attributes of the ErnieAttention instance in-place.

    Raises:
        None.
    """
    if len(heads) == 0:
        return
    heads, index = find_pruneable_heads_and_indices(
        heads, self.self.num_attention_heads, self.self.attention_head_size, self.pruned_heads
    )

    # Prune linear layers
    self.self.query = prune_linear_layer(self.self.query, index)
    self.self.key = prune_linear_layer(self.self.key, index)
    self.self.value = prune_linear_layer(self.self.value, index)
    self.output.dense = prune_linear_layer(self.output.dense, index, dim=1)

    # Update hyper params and store pruned heads
    self.self.num_attention_heads = self.self.num_attention_heads - len(heads)
    self.self.all_head_size = self.self.attention_head_size * self.self.num_attention_heads
    self.pruned_heads = self.pruned_heads.union(heads)

mindnlp.transformers.models.ernie.modeling_ernie.ErnieEmbeddings

Bases: Module

Construct the embeddings from word, position and token_type embeddings.

Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
class ErnieEmbeddings(nn.Module):
    """Construct the embeddings from word, position and token_type embeddings."""
    def __init__(self, config):
        """
        Initializes an instance of the ErnieEmbeddings class.

        Args:
            self: The instance of the ErnieEmbeddings class.
            config: An object containing configuration parameters for the ErnieEmbeddings class.
                The config object should have the following attributes:

                - vocab_size (int): The size of the vocabulary.
                - hidden_size (int): The size of the hidden layers.
                - pad_token_id (int): The ID of the padding token.
                - max_position_embeddings (int): The maximum number of position embeddings.
                - type_vocab_size (int): The size of the token type vocabulary.
                - use_task_id (bool): Whether to use task IDs.
                - task_type_vocab_size (int): The size of the task type vocabulary.
                - layer_norm_eps (float): The epsilon value for layer normalization.
                - hidden_dropout_prob (float): The dropout probability for hidden layers.
                - position_embedding_type (str): The type of position embedding to use. Default is 'absolute'.

        Returns:
            None

        Raises:
            None
        """
        super().__init__()
        self.word_embeddings = nn.Embedding(config.vocab_size, config.hidden_size, padding_idx=config.pad_token_id)
        self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.hidden_size)
        self.token_type_embeddings = nn.Embedding(config.type_vocab_size, config.hidden_size)
        self.use_task_id = config.use_task_id
        if config.use_task_id:
            self.task_type_embeddings = nn.Embedding(config.task_type_vocab_size, config.hidden_size)

        # self.LayerNorm is not snake-cased to stick with TensorFlow model variable name and be able to load
        # any TensorFlow checkpoint file
        self.LayerNorm = nn.LayerNorm([config.hidden_size], eps=config.layer_norm_eps)
        self.dropout = nn.Dropout(p=config.hidden_dropout_prob)
        # position_ids (1, len position emb) is contiguous in memory and exported when serialized
        self.position_embedding_type = getattr(config, "position_embedding_type", "absolute")
        self.position_ids = ops.arange(config.max_position_embeddings).broadcast_to((1, -1))
        self.token_type_ids = ops.zeros(self.position_ids.shape, dtype=mindspore.int64)

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        token_type_ids: Optional[mindspore.Tensor] = None,
        task_type_ids: Optional[mindspore.Tensor] = None,
        position_ids: Optional[mindspore.Tensor] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        past_key_values_length: int = 0,
    ) -> mindspore.Tensor:
        """
        Constructs the embeddings for the ERNIE model.

        Args:
            self (ErnieEmbeddings): The instance of the ErnieEmbeddings class.
            input_ids (Optional[mindspore.Tensor]): The input tensor of shape [batch_size, sequence_length].
            token_type_ids (Optional[mindspore.Tensor]): The token type tensor of shape [batch_size, sequence_length].
            task_type_ids (Optional[mindspore.Tensor]): The task type tensor of shape [batch_size, sequence_length].
            position_ids (Optional[mindspore.Tensor]): The position ids tensor of shape [batch_size, sequence_length].
            inputs_embeds (Optional[mindspore.Tensor]): The input embeddings tensor of shape [batch_size, sequence_length, embedding_size].
            past_key_values_length (int): The length of past key values.

        Returns:
            mindspore.Tensor: The embeddings tensor of shape [batch_size, sequence_length, embedding_size].

        Raises:
            None
        """
        if input_ids is not None:
            input_shape = input_ids.shape
        else:
            input_shape = inputs_embeds.shape[:-1]

        seq_length = input_shape[1]

        if position_ids is None:
            position_ids = self.position_ids[:, past_key_values_length : seq_length + past_key_values_length]

        # Setting the token_type_ids to the registered buffer in forwardor where it is all zeros, which usually occurs
        # when its auto-generated, registered buffer helps users when tracing the model without passing token_type_ids, solves
        # issue #5664
        if token_type_ids is None:
            if hasattr(self, "token_type_ids"):
                buffered_token_type_ids = self.token_type_ids[:, :seq_length]
                buffered_token_type_ids_expanded = buffered_token_type_ids.broadcast_to((input_shape[0], seq_length))
                token_type_ids = buffered_token_type_ids_expanded
            else:
                token_type_ids = ops.zeros(input_shape, dtype=mindspore.int64)

        if inputs_embeds is None:
            inputs_embeds = self.word_embeddings(input_ids)
        token_type_embeddings = self.token_type_embeddings(token_type_ids)

        embeddings = inputs_embeds + token_type_embeddings
        if self.position_embedding_type == "absolute":
            position_embeddings = self.position_embeddings(position_ids)
            embeddings += position_embeddings

        # add `task_type_id` for ERNIE model
        if self.use_task_id:
            if task_type_ids is None:
                task_type_ids = ops.zeros(input_shape, dtype=mindspore.int64)
            task_type_embeddings = self.task_type_embeddings(task_type_ids)
            embeddings += task_type_embeddings

        embeddings = self.LayerNorm(embeddings)
        embeddings = self.dropout(embeddings)
        return embeddings

mindnlp.transformers.models.ernie.modeling_ernie.ErnieEmbeddings.__init__(config)

Initializes an instance of the ErnieEmbeddings class.

PARAMETER DESCRIPTION
self

The instance of the ErnieEmbeddings class.
config

An object containing configuration parameters for the ErnieEmbeddings class. The config object should have the following attributes:

  • vocab_size (int): The size of the vocabulary.
  • hidden_size (int): The size of the hidden layers.
  • pad_token_id (int): The ID of the padding token.
  • max_position_embeddings (int): The maximum number of position embeddings.
  • type_vocab_size (int): The size of the token type vocabulary.
  • use_task_id (bool): Whether to use task IDs.
  • task_type_vocab_size (int): The size of the task type vocabulary.
  • layer_norm_eps (float): The epsilon value for layer normalization.
  • hidden_dropout_prob (float): The dropout probability for hidden layers.
  • position_embedding_type (str): The type of position embedding to use. Default is 'absolute'.
RETURNS DESCRIPTION

None
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
def __init__(self, config):
    """
    Initializes an instance of the ErnieEmbeddings class.

    Args:
        self: The instance of the ErnieEmbeddings class.
        config: An object containing configuration parameters for the ErnieEmbeddings class.
            The config object should have the following attributes:

            - vocab_size (int): The size of the vocabulary.
            - hidden_size (int): The size of the hidden layers.
            - pad_token_id (int): The ID of the padding token.
            - max_position_embeddings (int): The maximum number of position embeddings.
            - type_vocab_size (int): The size of the token type vocabulary.
            - use_task_id (bool): Whether to use task IDs.
            - task_type_vocab_size (int): The size of the task type vocabulary.
            - layer_norm_eps (float): The epsilon value for layer normalization.
            - hidden_dropout_prob (float): The dropout probability for hidden layers.
            - position_embedding_type (str): The type of position embedding to use. Default is 'absolute'.

    Returns:
        None

    Raises:
        None
    """
    super().__init__()
    self.word_embeddings = nn.Embedding(config.vocab_size, config.hidden_size, padding_idx=config.pad_token_id)
    self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.hidden_size)
    self.token_type_embeddings = nn.Embedding(config.type_vocab_size, config.hidden_size)
    self.use_task_id = config.use_task_id
    if config.use_task_id:
        self.task_type_embeddings = nn.Embedding(config.task_type_vocab_size, config.hidden_size)

    # self.LayerNorm is not snake-cased to stick with TensorFlow model variable name and be able to load
    # any TensorFlow checkpoint file
    self.LayerNorm = nn.LayerNorm([config.hidden_size], eps=config.layer_norm_eps)
    self.dropout = nn.Dropout(p=config.hidden_dropout_prob)
    # position_ids (1, len position emb) is contiguous in memory and exported when serialized
    self.position_embedding_type = getattr(config, "position_embedding_type", "absolute")
    self.position_ids = ops.arange(config.max_position_embeddings).broadcast_to((1, -1))
    self.token_type_ids = ops.zeros(self.position_ids.shape, dtype=mindspore.int64)

mindnlp.transformers.models.ernie.modeling_ernie.ErnieEmbeddings.forward(input_ids=None, token_type_ids=None, task_type_ids=None, position_ids=None, inputs_embeds=None, past_key_values_length=0)

Constructs the embeddings for the ERNIE model.

PARAMETER DESCRIPTION
self

The instance of the ErnieEmbeddings class.

TYPE: ErnieEmbeddings

input_ids

The input tensor of shape [batch_size, sequence_length].

TYPE: Optional[Tensor] DEFAULT: None

token_type_ids

The token type tensor of shape [batch_size, sequence_length].

TYPE: Optional[Tensor] DEFAULT: None

task_type_ids

The task type tensor of shape [batch_size, sequence_length].

TYPE: Optional[Tensor] DEFAULT: None

position_ids

The position ids tensor of shape [batch_size, sequence_length].

TYPE: Optional[Tensor] DEFAULT: None

inputs_embeds

The input embeddings tensor of shape [batch_size, sequence_length, embedding_size].

TYPE: Optional[Tensor] DEFAULT: None

past_key_values_length

The length of past key values.

TYPE: int DEFAULT: 0

RETURNS DESCRIPTION
Tensor

mindspore.Tensor: The embeddings tensor of shape [batch_size, sequence_length, embedding_size].
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    token_type_ids: Optional[mindspore.Tensor] = None,
    task_type_ids: Optional[mindspore.Tensor] = None,
    position_ids: Optional[mindspore.Tensor] = None,
    inputs_embeds: Optional[mindspore.Tensor] = None,
    past_key_values_length: int = 0,
) -> mindspore.Tensor:
    """
    Constructs the embeddings for the ERNIE model.

    Args:
        self (ErnieEmbeddings): The instance of the ErnieEmbeddings class.
        input_ids (Optional[mindspore.Tensor]): The input tensor of shape [batch_size, sequence_length].
        token_type_ids (Optional[mindspore.Tensor]): The token type tensor of shape [batch_size, sequence_length].
        task_type_ids (Optional[mindspore.Tensor]): The task type tensor of shape [batch_size, sequence_length].
        position_ids (Optional[mindspore.Tensor]): The position ids tensor of shape [batch_size, sequence_length].
        inputs_embeds (Optional[mindspore.Tensor]): The input embeddings tensor of shape [batch_size, sequence_length, embedding_size].
        past_key_values_length (int): The length of past key values.

    Returns:
        mindspore.Tensor: The embeddings tensor of shape [batch_size, sequence_length, embedding_size].

    Raises:
        None
    """
    if input_ids is not None:
        input_shape = input_ids.shape
    else:
        input_shape = inputs_embeds.shape[:-1]

    seq_length = input_shape[1]

    if position_ids is None:
        position_ids = self.position_ids[:, past_key_values_length : seq_length + past_key_values_length]

    # Setting the token_type_ids to the registered buffer in forwardor where it is all zeros, which usually occurs
    # when its auto-generated, registered buffer helps users when tracing the model without passing token_type_ids, solves
    # issue #5664
    if token_type_ids is None:
        if hasattr(self, "token_type_ids"):
            buffered_token_type_ids = self.token_type_ids[:, :seq_length]
            buffered_token_type_ids_expanded = buffered_token_type_ids.broadcast_to((input_shape[0], seq_length))
            token_type_ids = buffered_token_type_ids_expanded
        else:
            token_type_ids = ops.zeros(input_shape, dtype=mindspore.int64)

    if inputs_embeds is None:
        inputs_embeds = self.word_embeddings(input_ids)
    token_type_embeddings = self.token_type_embeddings(token_type_ids)

    embeddings = inputs_embeds + token_type_embeddings
    if self.position_embedding_type == "absolute":
        position_embeddings = self.position_embeddings(position_ids)
        embeddings += position_embeddings

    # add `task_type_id` for ERNIE model
    if self.use_task_id:
        if task_type_ids is None:
            task_type_ids = ops.zeros(input_shape, dtype=mindspore.int64)
        task_type_embeddings = self.task_type_embeddings(task_type_ids)
        embeddings += task_type_embeddings

    embeddings = self.LayerNorm(embeddings)
    embeddings = self.dropout(embeddings)
    return embeddings

mindnlp.transformers.models.ernie.modeling_ernie.ErnieEncoder

Bases: Module

The ErnieEncoder class represents a multi-layer Ernie (Enhanced Representation through kNowledge Integration) encoder module for processing sequential inputs. It inherits from the nn.Module class.

ATTRIBUTE DESCRIPTION
config

The configuration settings for the ErnieEncoder.
layer

A list of ErnieLayer instances representing the individual layers of the encoder.
gradient_checkpointing

A boolean indicating whether gradient checkpointing is enabled.
METHOD DESCRIPTION
__init__

Initializes the ErnieEncoder with the provided configuration.
forward

Constructs the ErnieEncoder module with the given inputs and returns the output either as a tuple of tensors or as a BaseModelOutputWithPastAndCrossAttentions object.
Notes
  • The forward method supports various optional input parameters and returns different types of outputs based on the provided arguments.
  • The class supports gradient checkpointing when enabled during training.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
 882
 883
 884
 885
 886
 887
 888
 889
 890
 891
 892
 893
 894
 895
 896
 897
 898
 899
 900
 901
 902
 903
 904
 905
 906
 907
 908
 909
 910
 911
 912
 913
 914
 915
 916
 917
 918
 919
 920
 921
 922
 923
 924
 925
 926
 927
 928
 929
 930
 931
 932
 933
 934
 935
 936
 937
 938
 939
 940
 941
 942
 943
 944
 945
 946
 947
 948
 949
 950
 951
 952
 953
 954
 955
 956
 957
 958
 959
 960
 961
 962
 963
 964
 965
 966
 967
 968
 969
 970
 971
 972
 973
 974
 975
 976
 977
 978
 979
 980
 981
 982
 983
 984
 985
 986
 987
 988
 989
 990
 991
 992
 993
 994
 995
 996
 997
 998
 999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
class ErnieEncoder(nn.Module):

    """
    The ErnieEncoder class represents a multi-layer Ernie (Enhanced Representation through kNowledge Integration)
    encoder module for processing sequential inputs. It inherits from the nn.Module class.

    Attributes:
        config: The configuration settings for the ErnieEncoder.
        layer: A list of ErnieLayer instances representing the individual layers of the encoder.
        gradient_checkpointing: A boolean indicating whether gradient checkpointing is enabled.

    Methods:
        __init__: Initializes the ErnieEncoder with the provided configuration.
        forward:
            Constructs the ErnieEncoder module with the given inputs and returns the output either as a tuple of tensors
            or as a BaseModelOutputWithPastAndCrossAttentions object.

    Notes:
        - The forward method supports various optional input parameters and returns different types of outputs based
        on the provided arguments.
        - The class supports gradient checkpointing when enabled during training.
    """
    def __init__(self, config):
        """
        Initialize the ErnieEncoder class.

        Args:
            self (ErnieEncoder): The instance of the ErnieEncoder class.
            config (dict): A dictionary containing configuration parameters for the ErnieEncoder.
                It should include the following keys:

                - num_hidden_layers (int): The number of hidden layers in the ErnieEncoder.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.config = config
        self.layer = nn.ModuleList([ErnieLayer(config) for _ in range(config.num_hidden_layers)])
        self.gradient_checkpointing = False

    def forward(
        self,
        hidden_states: mindspore.Tensor,
        attention_mask: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        encoder_hidden_states: Optional[mindspore.Tensor] = None,
        encoder_attention_mask: Optional[mindspore.Tensor] = None,
        past_key_values: Optional[Tuple[Tuple[mindspore.Tensor]]] = None,
        use_cache: Optional[bool] = None,
        output_attentions: Optional[bool] = False,
        output_hidden_states: Optional[bool] = False,
        return_dict: Optional[bool] = True,
    ) -> Union[Tuple[mindspore.Tensor], BaseModelOutputWithPastAndCrossAttentions]:
        '''
        Constructs the ErnieEncoder.

        Args:
            self (ErnieEncoder): The instance of the ErnieEncoder class.
            hidden_states (mindspore.Tensor): The input hidden states of the encoder.
            attention_mask (Optional[mindspore.Tensor]): The attention mask tensor. Defaults to None.
            head_mask (Optional[mindspore.Tensor]): The head mask tensor. Defaults to None.
            encoder_hidden_states (Optional[mindspore.Tensor]): The hidden states of the encoder. Defaults to None.
            encoder_attention_mask (Optional[mindspore.Tensor]): The attention mask tensor for the encoder.
                Defaults to None.
            past_key_values (Optional[Tuple[Tuple[mindspore.Tensor]]]): The past key values. Defaults to None.
            use_cache (Optional[bool]): Whether to use cache. Defaults to None.
            output_attentions (Optional[bool]): Whether to output attentions. Defaults to False.
            output_hidden_states (Optional[bool]): Whether to output hidden states. Defaults to False.
            return_dict (Optional[bool]): Whether to return a dictionary. Defaults to True.

        Returns:
            Union[Tuple[mindspore.Tensor], BaseModelOutputWithPastAndCrossAttentions]: The output of the ErnieEncoder.

        Raises:
            None.
        '''
        all_hidden_states = () if output_hidden_states else None
        all_self_attentions = () if output_attentions else None
        all_cross_attentions = () if output_attentions and self.config.add_cross_attention else None

        if self.gradient_checkpointing and self.training:
            if use_cache:
                logger.warning_once(
                    "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
                )
                use_cache = False

        next_decoder_cache = () if use_cache else None
        for i, layer_module in enumerate(self.layer):
            if output_hidden_states:
                all_hidden_states = all_hidden_states + (hidden_states,)

            layer_head_mask = head_mask[i] if head_mask is not None else None
            past_key_value = past_key_values[i] if past_key_values is not None else None

            if self.gradient_checkpointing and self.training:
                layer_outputs = self._gradient_checkpointing_func(
                    layer_module.__call__,
                    hidden_states,
                    attention_mask,
                    layer_head_mask,
                    encoder_hidden_states,
                    encoder_attention_mask,
                    past_key_value,
                    output_attentions,
                )
            else:
                layer_outputs = layer_module(
                    hidden_states,
                    attention_mask,
                    layer_head_mask,
                    encoder_hidden_states,
                    encoder_attention_mask,
                    past_key_value,
                    output_attentions,
                )

            hidden_states = layer_outputs[0]
            if use_cache:
                next_decoder_cache += (layer_outputs[-1],)
            if output_attentions:
                all_self_attentions = all_self_attentions + (layer_outputs[1],)
                if self.config.add_cross_attention:
                    all_cross_attentions = all_cross_attentions + (layer_outputs[2],)

        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        if not return_dict:
            return tuple(
                v
                for v in [
                    hidden_states,
                    next_decoder_cache,
                    all_hidden_states,
                    all_self_attentions,
                    all_cross_attentions,
                ]
                if v is not None
            )
        return BaseModelOutputWithPastAndCrossAttentions(
            last_hidden_state=hidden_states,
            past_key_values=next_decoder_cache,
            hidden_states=all_hidden_states,
            attentions=all_self_attentions,
            cross_attentions=all_cross_attentions,
        )

mindnlp.transformers.models.ernie.modeling_ernie.ErnieEncoder.__init__(config)

Initialize the ErnieEncoder class.

PARAMETER DESCRIPTION
self

The instance of the ErnieEncoder class.

TYPE: ErnieEncoder

config

A dictionary containing configuration parameters for the ErnieEncoder. It should include the following keys:

  • num_hidden_layers (int): The number of hidden layers in the ErnieEncoder.

TYPE: dict

RETURNS DESCRIPTION

None.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
def __init__(self, config):
    """
    Initialize the ErnieEncoder class.

    Args:
        self (ErnieEncoder): The instance of the ErnieEncoder class.
        config (dict): A dictionary containing configuration parameters for the ErnieEncoder.
            It should include the following keys:

            - num_hidden_layers (int): The number of hidden layers in the ErnieEncoder.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.config = config
    self.layer = nn.ModuleList([ErnieLayer(config) for _ in range(config.num_hidden_layers)])
    self.gradient_checkpointing = False

mindnlp.transformers.models.ernie.modeling_ernie.ErnieEncoder.forward(hidden_states, attention_mask=None, head_mask=None, encoder_hidden_states=None, encoder_attention_mask=None, past_key_values=None, use_cache=None, output_attentions=False, output_hidden_states=False, return_dict=True)

Constructs the ErnieEncoder.

PARAMETER DESCRIPTION
self

The instance of the ErnieEncoder class.

TYPE: ErnieEncoder

hidden_states

The input hidden states of the encoder.

TYPE: Tensor

attention_mask

The attention mask tensor. Defaults to None.

TYPE: Optional[Tensor] DEFAULT: None

head_mask

The head mask tensor. Defaults to None.

TYPE: Optional[Tensor] DEFAULT: None

encoder_hidden_states

The hidden states of the encoder. Defaults to None.

TYPE: Optional[Tensor] DEFAULT: None

encoder_attention_mask

The attention mask tensor for the encoder. Defaults to None.

TYPE: Optional[Tensor] DEFAULT: None

past_key_values

The past key values. Defaults to None.

TYPE: Optional[Tuple[Tuple[Tensor]]] DEFAULT: None

use_cache

Whether to use cache. Defaults to None.

TYPE: Optional[bool] DEFAULT: None

output_attentions

Whether to output attentions. Defaults to False.

TYPE: Optional[bool] DEFAULT: False

output_hidden_states

Whether to output hidden states. Defaults to False.

TYPE: Optional[bool] DEFAULT: False

return_dict

Whether to return a dictionary. Defaults to True.

TYPE: Optional[bool] DEFAULT: True

RETURNS DESCRIPTION
Union[Tuple[Tensor], BaseModelOutputWithPastAndCrossAttentions]

Union[Tuple[mindspore.Tensor], BaseModelOutputWithPastAndCrossAttentions]: The output of the ErnieEncoder.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
 926
 927
 928
 929
 930
 931
 932
 933
 934
 935
 936
 937
 938
 939
 940
 941
 942
 943
 944
 945
 946
 947
 948
 949
 950
 951
 952
 953
 954
 955
 956
 957
 958
 959
 960
 961
 962
 963
 964
 965
 966
 967
 968
 969
 970
 971
 972
 973
 974
 975
 976
 977
 978
 979
 980
 981
 982
 983
 984
 985
 986
 987
 988
 989
 990
 991
 992
 993
 994
 995
 996
 997
 998
 999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
def forward(
    self,
    hidden_states: mindspore.Tensor,
    attention_mask: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    encoder_hidden_states: Optional[mindspore.Tensor] = None,
    encoder_attention_mask: Optional[mindspore.Tensor] = None,
    past_key_values: Optional[Tuple[Tuple[mindspore.Tensor]]] = None,
    use_cache: Optional[bool] = None,
    output_attentions: Optional[bool] = False,
    output_hidden_states: Optional[bool] = False,
    return_dict: Optional[bool] = True,
) -> Union[Tuple[mindspore.Tensor], BaseModelOutputWithPastAndCrossAttentions]:
    '''
    Constructs the ErnieEncoder.

    Args:
        self (ErnieEncoder): The instance of the ErnieEncoder class.
        hidden_states (mindspore.Tensor): The input hidden states of the encoder.
        attention_mask (Optional[mindspore.Tensor]): The attention mask tensor. Defaults to None.
        head_mask (Optional[mindspore.Tensor]): The head mask tensor. Defaults to None.
        encoder_hidden_states (Optional[mindspore.Tensor]): The hidden states of the encoder. Defaults to None.
        encoder_attention_mask (Optional[mindspore.Tensor]): The attention mask tensor for the encoder.
            Defaults to None.
        past_key_values (Optional[Tuple[Tuple[mindspore.Tensor]]]): The past key values. Defaults to None.
        use_cache (Optional[bool]): Whether to use cache. Defaults to None.
        output_attentions (Optional[bool]): Whether to output attentions. Defaults to False.
        output_hidden_states (Optional[bool]): Whether to output hidden states. Defaults to False.
        return_dict (Optional[bool]): Whether to return a dictionary. Defaults to True.

    Returns:
        Union[Tuple[mindspore.Tensor], BaseModelOutputWithPastAndCrossAttentions]: The output of the ErnieEncoder.

    Raises:
        None.
    '''
    all_hidden_states = () if output_hidden_states else None
    all_self_attentions = () if output_attentions else None
    all_cross_attentions = () if output_attentions and self.config.add_cross_attention else None

    if self.gradient_checkpointing and self.training:
        if use_cache:
            logger.warning_once(
                "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
            )
            use_cache = False

    next_decoder_cache = () if use_cache else None
    for i, layer_module in enumerate(self.layer):
        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        layer_head_mask = head_mask[i] if head_mask is not None else None
        past_key_value = past_key_values[i] if past_key_values is not None else None

        if self.gradient_checkpointing and self.training:
            layer_outputs = self._gradient_checkpointing_func(
                layer_module.__call__,
                hidden_states,
                attention_mask,
                layer_head_mask,
                encoder_hidden_states,
                encoder_attention_mask,
                past_key_value,
                output_attentions,
            )
        else:
            layer_outputs = layer_module(
                hidden_states,
                attention_mask,
                layer_head_mask,
                encoder_hidden_states,
                encoder_attention_mask,
                past_key_value,
                output_attentions,
            )

        hidden_states = layer_outputs[0]
        if use_cache:
            next_decoder_cache += (layer_outputs[-1],)
        if output_attentions:
            all_self_attentions = all_self_attentions + (layer_outputs[1],)
            if self.config.add_cross_attention:
                all_cross_attentions = all_cross_attentions + (layer_outputs[2],)

    if output_hidden_states:
        all_hidden_states = all_hidden_states + (hidden_states,)

    if not return_dict:
        return tuple(
            v
            for v in [
                hidden_states,
                next_decoder_cache,
                all_hidden_states,
                all_self_attentions,
                all_cross_attentions,
            ]
            if v is not None
        )
    return BaseModelOutputWithPastAndCrossAttentions(
        last_hidden_state=hidden_states,
        past_key_values=next_decoder_cache,
        hidden_states=all_hidden_states,
        attentions=all_self_attentions,
        cross_attentions=all_cross_attentions,
    )

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForCausalLM

Bases: ErniePreTrainedModel

This class represents a causal language modeling model based on the ERNIE (Enhanced Representation through kNowledge Integration) architecture. It is designed for generating text predictions based on input sequences, with a focus on predicting the next word in a sequence. The model includes functionality for forwarding the model, setting and getting output embeddings, preparing inputs for text generation, and reordering cache during generation.

The class includes methods for initializing the model, forwarding the model for inference or training, setting and getting output embeddings, preparing inputs for text generation, and reordering cache during generation.

The 'forward' method forwards the model for inference or training, taking various input tensors such as input ids, attention masks, token type ids, and more. It returns the model outputs including the language modeling loss and predictions.

The 'prepare_inputs_for_generation' method prepares input tensors for text generation, including handling past key values and attention masks. It returns a dictionary containing the input ids, attention mask, past key values, and use_cache flag.

The '_reorder_cache' method reorders the past key values during generation based on the beam index used for parallel decoding.

For more detailed information on each method's parameters and return values, refer to the method docstrings within the class code.

Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
1863
1864
1865
1866
1867
1868
1869
1870
1871
1872
1873
1874
1875
1876
1877
1878
1879
1880
1881
1882
1883
1884
1885
1886
1887
1888
1889
1890
1891
1892
1893
1894
1895
1896
1897
1898
1899
1900
1901
1902
1903
1904
1905
1906
1907
1908
1909
1910
1911
1912
1913
1914
1915
1916
1917
1918
1919
1920
1921
1922
1923
1924
1925
1926
1927
1928
1929
1930
1931
1932
1933
1934
1935
1936
1937
1938
1939
1940
1941
1942
1943
1944
1945
1946
1947
1948
1949
1950
1951
1952
1953
1954
1955
1956
1957
1958
1959
1960
1961
1962
1963
1964
1965
1966
1967
1968
1969
1970
1971
1972
1973
1974
1975
1976
1977
1978
1979
1980
1981
1982
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
2027
2028
2029
2030
2031
2032
2033
2034
2035
2036
2037
2038
2039
2040
2041
2042
2043
2044
2045
2046
2047
2048
2049
2050
2051
2052
2053
2054
2055
2056
2057
2058
2059
2060
2061
2062
2063
2064
2065
2066
2067
2068
2069
2070
2071
2072
2073
2074
2075
2076
2077
2078
2079
2080
2081
2082
2083
2084
2085
2086
2087
2088
2089
2090
2091
2092
2093
2094
2095
2096
2097
2098
2099
2100
2101
2102
2103
2104
2105
2106
2107
2108
2109
2110
2111
2112
2113
2114
2115
2116
2117
2118
2119
2120
2121
2122
class ErnieForCausalLM(ErniePreTrainedModel):

    """
    This class represents a causal language modeling model based on the ERNIE
    (Enhanced Representation through kNowledge Integration) architecture.
    It is designed for generating text predictions based on input sequences, with a focus on predicting the next word
    in a sequence.
    The model includes functionality for forwarding the model, setting and getting output embeddings, preparing inputs
    for text generation, and reordering cache during generation.

    The class includes methods for initializing the model, forwarding the model for inference or training, setting
    and getting output embeddings, preparing inputs for text generation, and reordering cache during generation.

    The 'forward' method forwards the model for inference or training, taking various input tensors such as
    input ids, attention masks, token type ids, and more. It returns the model outputs including the language modeling
    loss and predictions.

    The 'prepare_inputs_for_generation' method prepares input tensors for text generation, including handling past key
    values and attention masks. It returns a dictionary containing the input ids, attention  mask, past key values,
    and use_cache flag.

    The '_reorder_cache' method reorders the past key values during generation based on the beam index used for parallel
    decoding.

    For more detailed information on each method's parameters and return values, refer to the method docstrings within
    the class code.
    """
    _tied_weights_keys = ["cls.predictions.decoder.bias", "cls.predictions.decoder.weight"]

    # Copied from transformers.models.bert.modeling_bert.BertLMHeadModel.__init__ with BertLMHeadModel->ErnieForCausalLM,Bert->Ernie,bert->ernie
    def __init__(self, config):
        """
        Initializes an instance of the `ErnieForCausalLM` class.

        Args:
            self: The instance of the class.
            config (object):
                The configuration object containing various settings for the model.

                - Type: object
                - Purpose: Specifies the configuration settings for the model.
                - Restrictions: None

        Returns:
            None

        Raises:
            None
        """
        super().__init__(config)

        if not config.is_decoder:
            logger.warning("If you want to use `ErnieForCausalLM` as a standalone, add `is_decoder=True.`")

        self.ernie = ErnieModel(config, add_pooling_layer=False)
        self.cls = ErnieOnlyMLMHead(config)

        # Initialize weights and apply final processing
        self.post_init()

    # Copied from transformers.models.bert.modeling_bert.BertLMHeadModel.get_output_embeddings
    def get_output_embeddings(self):
        """
        Retrieve the output embeddings from the ErnieForCausalLM model.

        Args:
            self (ErnieForCausalLM): The instance of the ErnieForCausalLM class.

        Returns:
            decoder: This method returns the output embeddings from the model's predictions decoder layer.

        Raises:
            None.
        """
        return self.cls.predictions.decoder

    # Copied from transformers.models.bert.modeling_bert.BertLMHeadModel.set_output_embeddings
    def set_output_embeddings(self, new_embeddings):
        """
        Sets the output embeddings for the ErnieForCausalLM model.

        Args:
            self (ErnieForCausalLM): The instance of the ErnieForCausalLM class.
            new_embeddings: The new embeddings to be set as output embeddings.
                It should be of the same shape as the existing embeddings.

        Returns:
            None.

        Raises:
            None.

        Note:
            This method updates the output embeddings of the ErnieForCausalLM model to the provided new_embeddings.
            The new_embeddings should be of the same shape as the existing embeddings.

        Example:
            ```python
            >>> model = ErnieForCausalLM()
            >>> new_embeddings = torch.Tensor([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]])
            >>> model.set_output_embeddings(new_embeddings)
            ```
        """
        self.cls.predictions.decoder = new_embeddings

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        token_type_ids: Optional[mindspore.Tensor] = None,
        task_type_ids: Optional[mindspore.Tensor] = None,
        position_ids: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        encoder_hidden_states: Optional[mindspore.Tensor] = None,
        encoder_attention_mask: Optional[mindspore.Tensor] = None,
        labels: Optional[mindspore.Tensor] = None,
        past_key_values: Optional[List[mindspore.Tensor]] = None,
        use_cache: Optional[bool] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple[mindspore.Tensor], CausalLMOutputWithCrossAttentions]:
        r"""
        Args:
            encoder_hidden_states  (`mindspore.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*):
                Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if
                the model is configured as a decoder.
            encoder_attention_mask (`mindspore.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
                Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in
                the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`:

                - 1 for tokens that are **not masked**,
                - 0 for tokens that are **masked**.
            labels (`mindspore.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
                Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in
                `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are
                ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]`
            past_key_values (`tuple(tuple(mindspore.Tensor))` of length `config.n_layers` with each tuple having 4
                tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`):
                Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding.
                If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that
                don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all
                `decoder_input_ids` of shape `(batch_size, sequence_length)`.
            use_cache (`bool`, *optional*):
                If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see
                `past_key_values`).
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
        if labels is not None:
            use_cache = False

        outputs = self.ernie(
            input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            task_type_ids=task_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            encoder_hidden_states=encoder_hidden_states,
            encoder_attention_mask=encoder_attention_mask,
            past_key_values=past_key_values,
            use_cache=use_cache,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        sequence_output = outputs[0]
        prediction_scores = self.cls(sequence_output)

        lm_loss = None
        if labels is not None:
            # we are doing next-token prediction; shift prediction scores and input ids by one
            shifted_prediction_scores = prediction_scores[:, :-1, :]
            labels = labels[:, 1:]
            lm_loss = ops.cross_entropy(shifted_prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))

        if not return_dict:
            output = (prediction_scores,) + outputs[2:]
            return ((lm_loss,) + output) if lm_loss is not None else output

        return CausalLMOutputWithCrossAttentions(
            loss=lm_loss,
            logits=prediction_scores,
            past_key_values=outputs.past_key_values,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
            cross_attentions=outputs.cross_attentions,
        )

    # Copied from transformers.models.bert.modeling_bert.BertLMHeadModel.prepare_inputs_for_generation
    def prepare_inputs_for_generation(
        self, input_ids, past_key_values=None, attention_mask=None, use_cache=True, **model_kwargs
    ):
        """
        Prepare inputs for generation.

        Args:
            self: The instance of the class.
            input_ids (torch.Tensor): The input tensor containing the input ids.
            past_key_values (tuple, optional): The tuple containing past key values. Defaults to None.
            attention_mask (torch.Tensor, optional): The attention mask tensor. Defaults to None.
            use_cache (bool, optional): Flag indicating whether to use cache. Defaults to True.

        Returns:
            dict: A dictionary containing the prepared input_ids, attention_mask, past_key_values, and use_cache.

        Raises:
            ValueError: If input_ids shape is incompatible with past_key_values.
        """
        input_shape = input_ids.shape
        # if model is used as a decoder in encoder-decoder model, the decoder attention mask is created on the fly
        if attention_mask is None:
            attention_mask = input_ids.new_ones(input_shape)

        # cut decoder_input_ids if past_key_values is used
        if past_key_values is not None:
            past_length = past_key_values[0][0].shape[2]

            # Some generation methods already pass only the last input ID
            if input_ids.shape[1] > past_length:
                remove_prefix_length = past_length
            else:
                # Default to old behavior: keep only final ID
                remove_prefix_length = input_ids.shape[1] - 1

            input_ids = input_ids[:, remove_prefix_length:]

        return {
            "input_ids": input_ids,
            "attention_mask": attention_mask,
            "past_key_values": past_key_values,
            "use_cache": use_cache,
        }

    # Copied from transformers.models.bert.modeling_bert.BertLMHeadModel._reorder_cache
    def _reorder_cache(self, past_key_values, beam_idx):
        """
        This method '_reorder_cache' reorders the past states based on the provided beam indices.

        Args:
            self (ErnieForCausalLM): The instance of the ErnieForCausalLM class.
            past_key_values (tuple): A tuple containing past states for each layer.
            beam_idx (Tensor): A tensor representing the beam indices used for reordering.

        Returns:
            None: This method does not return any value but updates the 'reordered_past' variable within the method.

        Raises:
            IndexError: If the provided beam indices are out of bounds.
            TypeError: If the input types are not as expected.
        """
        reordered_past = ()
        for layer_past in past_key_values:
            reordered_past += (
                tuple(past_state.index_select(0, beam_idx) for past_state in layer_past),
            )
        return reordered_past

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForCausalLM.__init__(config)

Initializes an instance of the ErnieForCausalLM class.

PARAMETER DESCRIPTION
self

The instance of the class.
config

The configuration object containing various settings for the model.

  • Type: object
  • Purpose: Specifies the configuration settings for the model.
  • Restrictions: None

TYPE: object

RETURNS DESCRIPTION

None
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
1893
1894
1895
1896
1897
1898
1899
1900
1901
1902
1903
1904
1905
1906
1907
1908
1909
1910
1911
1912
1913
1914
1915
1916
1917
1918
1919
1920
1921
def __init__(self, config):
    """
    Initializes an instance of the `ErnieForCausalLM` class.

    Args:
        self: The instance of the class.
        config (object):
            The configuration object containing various settings for the model.

            - Type: object
            - Purpose: Specifies the configuration settings for the model.
            - Restrictions: None

    Returns:
        None

    Raises:
        None
    """
    super().__init__(config)

    if not config.is_decoder:
        logger.warning("If you want to use `ErnieForCausalLM` as a standalone, add `is_decoder=True.`")

    self.ernie = ErnieModel(config, add_pooling_layer=False)
    self.cls = ErnieOnlyMLMHead(config)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForCausalLM.forward(input_ids=None, attention_mask=None, token_type_ids=None, task_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None, encoder_hidden_states=None, encoder_attention_mask=None, labels=None, past_key_values=None, use_cache=None, output_attentions=None, output_hidden_states=None, return_dict=None)

PARAMETER DESCRIPTION
encoder_hidden_states

Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder.

TYPE: (`mindspore.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional* DEFAULT: None

encoder_attention_mask

Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in [0, 1]:

  • 1 for tokens that are not masked,
  • 0 for tokens that are masked.

TYPE: `mindspore.Tensor` of shape `(batch_size, sequence_length)`, *optional* DEFAULT: None

labels

Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in [-100, 0, ..., config.vocab_size] (see input_ids docstring) Tokens with indices set to -100 are ignored (masked), the loss is only computed for the tokens with labels n [0, ..., config.vocab_size]

TYPE: `mindspore.Tensor` of shape `(batch_size, sequence_length)`, *optional* DEFAULT: None

use_cache

If set to True, past_key_values key value states are returned and can be used to speed up decoding (see past_key_values).

TYPE: `bool`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
1968
1969
1970
1971
1972
1973
1974
1975
1976
1977
1978
1979
1980
1981
1982
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
2027
2028
2029
2030
2031
2032
2033
2034
2035
2036
2037
2038
2039
2040
2041
2042
2043
2044
2045
2046
2047
2048
2049
2050
2051
2052
2053
def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    token_type_ids: Optional[mindspore.Tensor] = None,
    task_type_ids: Optional[mindspore.Tensor] = None,
    position_ids: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    inputs_embeds: Optional[mindspore.Tensor] = None,
    encoder_hidden_states: Optional[mindspore.Tensor] = None,
    encoder_attention_mask: Optional[mindspore.Tensor] = None,
    labels: Optional[mindspore.Tensor] = None,
    past_key_values: Optional[List[mindspore.Tensor]] = None,
    use_cache: Optional[bool] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple[mindspore.Tensor], CausalLMOutputWithCrossAttentions]:
    r"""
    Args:
        encoder_hidden_states  (`mindspore.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*):
            Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if
            the model is configured as a decoder.
        encoder_attention_mask (`mindspore.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
            Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in
            the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`:

            - 1 for tokens that are **not masked**,
            - 0 for tokens that are **masked**.
        labels (`mindspore.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
            Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in
            `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are
            ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]`
        past_key_values (`tuple(tuple(mindspore.Tensor))` of length `config.n_layers` with each tuple having 4
            tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`):
            Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding.
            If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that
            don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all
            `decoder_input_ids` of shape `(batch_size, sequence_length)`.
        use_cache (`bool`, *optional*):
            If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see
            `past_key_values`).
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    if labels is not None:
        use_cache = False

    outputs = self.ernie(
        input_ids,
        attention_mask=attention_mask,
        token_type_ids=token_type_ids,
        task_type_ids=task_type_ids,
        position_ids=position_ids,
        head_mask=head_mask,
        inputs_embeds=inputs_embeds,
        encoder_hidden_states=encoder_hidden_states,
        encoder_attention_mask=encoder_attention_mask,
        past_key_values=past_key_values,
        use_cache=use_cache,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    sequence_output = outputs[0]
    prediction_scores = self.cls(sequence_output)

    lm_loss = None
    if labels is not None:
        # we are doing next-token prediction; shift prediction scores and input ids by one
        shifted_prediction_scores = prediction_scores[:, :-1, :]
        labels = labels[:, 1:]
        lm_loss = ops.cross_entropy(shifted_prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))

    if not return_dict:
        output = (prediction_scores,) + outputs[2:]
        return ((lm_loss,) + output) if lm_loss is not None else output

    return CausalLMOutputWithCrossAttentions(
        loss=lm_loss,
        logits=prediction_scores,
        past_key_values=outputs.past_key_values,
        hidden_states=outputs.hidden_states,
        attentions=outputs.attentions,
        cross_attentions=outputs.cross_attentions,
    )

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForCausalLM.get_output_embeddings()

Retrieve the output embeddings from the ErnieForCausalLM model.

PARAMETER DESCRIPTION
self

The instance of the ErnieForCausalLM class.

TYPE: ErnieForCausalLM

RETURNS DESCRIPTION
decoder

This method returns the output embeddings from the model's predictions decoder layer.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
1924
1925
1926
1927
1928
1929
1930
1931
1932
1933
1934
1935
1936
1937
def get_output_embeddings(self):
    """
    Retrieve the output embeddings from the ErnieForCausalLM model.

    Args:
        self (ErnieForCausalLM): The instance of the ErnieForCausalLM class.

    Returns:
        decoder: This method returns the output embeddings from the model's predictions decoder layer.

    Raises:
        None.
    """
    return self.cls.predictions.decoder

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForCausalLM.prepare_inputs_for_generation(input_ids, past_key_values=None, attention_mask=None, use_cache=True, **model_kwargs)

Prepare inputs for generation.

PARAMETER DESCRIPTION
self

The instance of the class.
input_ids

The input tensor containing the input ids.

TYPE: Tensor

past_key_values

The tuple containing past key values. Defaults to None.

TYPE: tuple DEFAULT: None

attention_mask

The attention mask tensor. Defaults to None.

TYPE: Tensor DEFAULT: None

use_cache

Flag indicating whether to use cache. Defaults to True.

TYPE: bool DEFAULT: True

RETURNS DESCRIPTION
dict

A dictionary containing the prepared input_ids, attention_mask, past_key_values, and use_cache.
RAISES DESCRIPTION
ValueError

If input_ids shape is incompatible with past_key_values.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
2056
2057
2058
2059
2060
2061
2062
2063
2064
2065
2066
2067
2068
2069
2070
2071
2072
2073
2074
2075
2076
2077
2078
2079
2080
2081
2082
2083
2084
2085
2086
2087
2088
2089
2090
2091
2092
2093
2094
2095
2096
2097
2098
def prepare_inputs_for_generation(
    self, input_ids, past_key_values=None, attention_mask=None, use_cache=True, **model_kwargs
):
    """
    Prepare inputs for generation.

    Args:
        self: The instance of the class.
        input_ids (torch.Tensor): The input tensor containing the input ids.
        past_key_values (tuple, optional): The tuple containing past key values. Defaults to None.
        attention_mask (torch.Tensor, optional): The attention mask tensor. Defaults to None.
        use_cache (bool, optional): Flag indicating whether to use cache. Defaults to True.

    Returns:
        dict: A dictionary containing the prepared input_ids, attention_mask, past_key_values, and use_cache.

    Raises:
        ValueError: If input_ids shape is incompatible with past_key_values.
    """
    input_shape = input_ids.shape
    # if model is used as a decoder in encoder-decoder model, the decoder attention mask is created on the fly
    if attention_mask is None:
        attention_mask = input_ids.new_ones(input_shape)

    # cut decoder_input_ids if past_key_values is used
    if past_key_values is not None:
        past_length = past_key_values[0][0].shape[2]

        # Some generation methods already pass only the last input ID
        if input_ids.shape[1] > past_length:
            remove_prefix_length = past_length
        else:
            # Default to old behavior: keep only final ID
            remove_prefix_length = input_ids.shape[1] - 1

        input_ids = input_ids[:, remove_prefix_length:]

    return {
        "input_ids": input_ids,
        "attention_mask": attention_mask,
        "past_key_values": past_key_values,
        "use_cache": use_cache,
    }

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForCausalLM.set_output_embeddings(new_embeddings)

Sets the output embeddings for the ErnieForCausalLM model.

PARAMETER DESCRIPTION
self

The instance of the ErnieForCausalLM class.

TYPE: ErnieForCausalLM

new_embeddings

The new embeddings to be set as output embeddings. It should be of the same shape as the existing embeddings.
RETURNS DESCRIPTION

None.
Note

This method updates the output embeddings of the ErnieForCausalLM model to the provided new_embeddings. The new_embeddings should be of the same shape as the existing embeddings.

Example 
>>> model = ErnieForCausalLM()
>>> new_embeddings = torch.Tensor([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]])
>>> model.set_output_embeddings(new_embeddings)
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
1940
1941
1942
1943
1944
1945
1946
1947
1948
1949
1950
1951
1952
1953
1954
1955
1956
1957
1958
1959
1960
1961
1962
1963
1964
1965
1966
def set_output_embeddings(self, new_embeddings):
    """
    Sets the output embeddings for the ErnieForCausalLM model.

    Args:
        self (ErnieForCausalLM): The instance of the ErnieForCausalLM class.
        new_embeddings: The new embeddings to be set as output embeddings.
            It should be of the same shape as the existing embeddings.

    Returns:
        None.

    Raises:
        None.

    Note:
        This method updates the output embeddings of the ErnieForCausalLM model to the provided new_embeddings.
        The new_embeddings should be of the same shape as the existing embeddings.

    Example:
        ```python
        >>> model = ErnieForCausalLM()
        >>> new_embeddings = torch.Tensor([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]])
        >>> model.set_output_embeddings(new_embeddings)
        ```
    """
    self.cls.predictions.decoder = new_embeddings

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForMaskedLM

Bases: ErniePreTrainedModel

This class represents a model for Masked Language Modeling using the ERNIE (Enhanced Representation through kNowledge Integration) architecture. It is designed for generating predictions for masked tokens within a sequence of text.

The class inherits from ErniePreTrainedModel and implements methods for initializing the model, getting and setting output embeddings, forwarding the model for training or inference, and preparing inputs for text generation.

METHOD DESCRIPTION
__init__

Initializes the ErnieForMaskedLM model with the given configuration.
get_output_embeddings

Retrieves the output embeddings from the model.
set_output_embeddings

Sets new output embeddings for the model.
forward

Constructs the model for training or inference, computing the masked language modeling loss and prediction scores.
prepare_inputs_for_generation

Prepares inputs for text generation, including handling padding and dummy tokens.
Note

This class assumes the existence of the ErnieModel and ErnieOnlyMLMHead classes for the ERNIE architecture.

Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
2125
2126
2127
2128
2129
2130
2131
2132
2133
2134
2135
2136
2137
2138
2139
2140
2141
2142
2143
2144
2145
2146
2147
2148
2149
2150
2151
2152
2153
2154
2155
2156
2157
2158
2159
2160
2161
2162
2163
2164
2165
2166
2167
2168
2169
2170
2171
2172
2173
2174
2175
2176
2177
2178
2179
2180
2181
2182
2183
2184
2185
2186
2187
2188
2189
2190
2191
2192
2193
2194
2195
2196
2197
2198
2199
2200
2201
2202
2203
2204
2205
2206
2207
2208
2209
2210
2211
2212
2213
2214
2215
2216
2217
2218
2219
2220
2221
2222
2223
2224
2225
2226
2227
2228
2229
2230
2231
2232
2233
2234
2235
2236
2237
2238
2239
2240
2241
2242
2243
2244
2245
2246
2247
2248
2249
2250
2251
2252
2253
2254
2255
2256
2257
2258
2259
2260
2261
2262
2263
2264
2265
2266
2267
2268
2269
2270
2271
2272
2273
2274
2275
2276
2277
2278
2279
2280
2281
2282
2283
2284
2285
2286
2287
2288
2289
2290
2291
2292
2293
2294
2295
2296
2297
2298
2299
2300
2301
2302
2303
2304
2305
2306
2307
2308
2309
2310
2311
2312
2313
2314
2315
2316
2317
2318
class ErnieForMaskedLM(ErniePreTrainedModel):

    """
    This class represents a model for Masked Language Modeling using the ERNIE
    (Enhanced Representation through kNowledge Integration) architecture.
    It is designed for generating predictions for masked tokens within a sequence of text.

    The class inherits from ErniePreTrainedModel and implements methods for initializing the model, getting and setting
    output embeddings, forwarding the model for training or inference, and preparing inputs for text generation.

    Methods:
        __init__: Initializes the ErnieForMaskedLM model with the given configuration.
        get_output_embeddings: Retrieves the output embeddings from the model.
        set_output_embeddings: Sets new output embeddings for the model.
        forward: Constructs the model for training or inference, computing the masked language modeling loss
            and prediction scores.
        prepare_inputs_for_generation: Prepares inputs for text generation, including handling padding and dummy tokens.

    Note:
        This class assumes the existence of the ErnieModel and ErnieOnlyMLMHead classes for the ERNIE architecture.
    """
    _tied_weights_keys = ["cls.predictions.decoder.bias", "cls.predictions.decoder.weight"]

    # Copied from transformers.models.bert.modeling_bert.BertForMaskedLM.__init__ with Bert->Ernie,bert->ernie
    def __init__(self, config):
        """
        Initializes an instance of the 'ErnieForMaskedLM' class.

        Args:
            self: The current object instance.
            config: An instance of the 'Config' class containing the configuration settings for the model.

        Returns:
            None

        Raises:
            None

        Description:
            This method initializes the 'ErnieForMaskedLM' class by setting the configuration and initializing the
            'ErnieModel' and 'ErnieOnlyMLMHead' objects.

            The 'config' parameter is an instance of the 'Config' class, which contains various configuration settings for the model.
            This method also logs a warning if the 'is_decoder' flag in the 'config' parameter is set to True,
            indicating that the model is being used as a decoder.

            The 'ErnieModel' object is initialized with the given 'config' and the 'add_pooling_layer' flag set to False.

            The 'ErnieOnlyMLMHead' object is also initialized with the given 'config'.

            Finally, the 'post_init' method is called to perform any additional initialization steps.
        """
        super().__init__(config)

        if config.is_decoder:
            logger.warning(
                "If you want to use `ErnieForMaskedLM` make sure `config.is_decoder=False` for "
                "bi-directional self-attention."
            )

        self.ernie = ErnieModel(config, add_pooling_layer=False)
        self.cls = ErnieOnlyMLMHead(config)

        # Initialize weights and apply final processing
        self.post_init()

    # Copied from transformers.models.bert.modeling_bert.BertForMaskedLM.get_output_embeddings
    def get_output_embeddings(self):
        """
        Retrieve the output embeddings from the ErnieForMaskedLM model.

        Args:
            self (ErnieForMaskedLM): An instance of the ErnieForMaskedLM class.
                Represents the model object that contains the output embeddings.

        Returns:
            None: This method returns the output embeddings stored in the 'decoder' of the 'predictions' object
                within the ErnieForMaskedLM model.

        Raises:
            None.
        """
        return self.cls.predictions.decoder

    # Copied from transformers.models.bert.modeling_bert.BertForMaskedLM.set_output_embeddings
    def set_output_embeddings(self, new_embeddings):
        """
        Sets the output embeddings for the ErnieForMaskedLM model.

        Args:
            self (ErnieForMaskedLM): The instance of the ErnieForMaskedLM class.
            new_embeddings (object): The new embeddings to be set for the model's output.
                It can be any object that is compatible with the existing model's output embeddings.
                The new embeddings will replace the current embeddings.

        Returns:
            None.

        Raises:
            None.
        """
        self.cls.predictions.decoder = new_embeddings

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        token_type_ids: Optional[mindspore.Tensor] = None,
        task_type_ids: Optional[mindspore.Tensor] = None,
        position_ids: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        encoder_hidden_states: Optional[mindspore.Tensor] = None,
        encoder_attention_mask: Optional[mindspore.Tensor] = None,
        labels: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple[mindspore.Tensor], MaskedLMOutput]:
        r"""
        Args:
            labels (`mindspore.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
                Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ...,
                config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the
                loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        outputs = self.ernie(
            input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            task_type_ids=task_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            encoder_hidden_states=encoder_hidden_states,
            encoder_attention_mask=encoder_attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        sequence_output = outputs[0]
        prediction_scores = self.cls(sequence_output)

        masked_lm_loss = None
        if labels is not None:
            masked_lm_loss = ops.cross_entropy(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))

        if not return_dict:
            output = (prediction_scores,) + outputs[2:]
            return ((masked_lm_loss,) + output) if masked_lm_loss is not None else output

        return MaskedLMOutput(
            loss=masked_lm_loss,
            logits=prediction_scores,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )

    # Copied from transformers.models.bert.modeling_bert.BertForMaskedLM.prepare_inputs_for_generation
    def prepare_inputs_for_generation(self, input_ids, attention_mask=None, **model_kwargs):
        """
        Prepare inputs for generation.

        This method prepares input data for generation in the ErnieForMaskedLM model.

        Args:
            self: The instance of the ErnieForMaskedLM class.
            input_ids (Tensor): The input token IDs. Shape (batch_size, sequence_length).
            attention_mask (Tensor, optional): The attention mask tensor. Shape (batch_size, sequence_length).
            **model_kwargs: Additional model-specific keyword arguments.

        Returns:
            dict: A dictionary containing the prepared input_ids and attention_mask.

        Raises:
            ValueError: If the PAD token is not defined for generation.
        """
        input_shape = input_ids.shape
        effective_batch_size = input_shape[0]

        #  add a dummy token
        if self.config.pad_token_id is None:
            raise ValueError("The PAD token should be defined for generation")

        attention_mask = ops.cat([attention_mask, attention_mask.new_zeros((attention_mask.shape[0], 1))], axis=-1)
        dummy_token = ops.full(
            (effective_batch_size, 1), self.config.pad_token_id, dtype=mindspore.int64
        )
        input_ids = ops.cat([input_ids, dummy_token], axis=1)

        return {"input_ids": input_ids, "attention_mask": attention_mask}

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForMaskedLM.__init__(config)

Initializes an instance of the 'ErnieForMaskedLM' class.

PARAMETER DESCRIPTION
self

The current object instance.
config

An instance of the 'Config' class containing the configuration settings for the model.
RETURNS DESCRIPTION

None
Description

This method initializes the 'ErnieForMaskedLM' class by setting the configuration and initializing the 'ErnieModel' and 'ErnieOnlyMLMHead' objects.

The 'config' parameter is an instance of the 'Config' class, which contains various configuration settings for the model. This method also logs a warning if the 'is_decoder' flag in the 'config' parameter is set to True, indicating that the model is being used as a decoder.

The 'ErnieModel' object is initialized with the given 'config' and the 'add_pooling_layer' flag set to False.

The 'ErnieOnlyMLMHead' object is also initialized with the given 'config'.

Finally, the 'post_init' method is called to perform any additional initialization steps.

Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
2149
2150
2151
2152
2153
2154
2155
2156
2157
2158
2159
2160
2161
2162
2163
2164
2165
2166
2167
2168
2169
2170
2171
2172
2173
2174
2175
2176
2177
2178
2179
2180
2181
2182
2183
2184
2185
2186
2187
2188
2189
def __init__(self, config):
    """
    Initializes an instance of the 'ErnieForMaskedLM' class.

    Args:
        self: The current object instance.
        config: An instance of the 'Config' class containing the configuration settings for the model.

    Returns:
        None

    Raises:
        None

    Description:
        This method initializes the 'ErnieForMaskedLM' class by setting the configuration and initializing the
        'ErnieModel' and 'ErnieOnlyMLMHead' objects.

        The 'config' parameter is an instance of the 'Config' class, which contains various configuration settings for the model.
        This method also logs a warning if the 'is_decoder' flag in the 'config' parameter is set to True,
        indicating that the model is being used as a decoder.

        The 'ErnieModel' object is initialized with the given 'config' and the 'add_pooling_layer' flag set to False.

        The 'ErnieOnlyMLMHead' object is also initialized with the given 'config'.

        Finally, the 'post_init' method is called to perform any additional initialization steps.
    """
    super().__init__(config)

    if config.is_decoder:
        logger.warning(
            "If you want to use `ErnieForMaskedLM` make sure `config.is_decoder=False` for "
            "bi-directional self-attention."
        )

    self.ernie = ErnieModel(config, add_pooling_layer=False)
    self.cls = ErnieOnlyMLMHead(config)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForMaskedLM.forward(input_ids=None, attention_mask=None, token_type_ids=None, task_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None, encoder_hidden_states=None, encoder_attention_mask=None, labels=None, output_attentions=None, output_hidden_states=None, return_dict=None)

PARAMETER DESCRIPTION
labels

Labels for computing the masked language modeling loss. Indices should be in [-100, 0, ..., config.vocab_size] (see input_ids docstring) Tokens with indices set to -100 are ignored (masked), the loss is only computed for the tokens with labels in [0, ..., config.vocab_size]

TYPE: `mindspore.Tensor` of shape `(batch_size, sequence_length)`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
2228
2229
2230
2231
2232
2233
2234
2235
2236
2237
2238
2239
2240
2241
2242
2243
2244
2245
2246
2247
2248
2249
2250
2251
2252
2253
2254
2255
2256
2257
2258
2259
2260
2261
2262
2263
2264
2265
2266
2267
2268
2269
2270
2271
2272
2273
2274
2275
2276
2277
2278
2279
2280
2281
2282
2283
2284
def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    token_type_ids: Optional[mindspore.Tensor] = None,
    task_type_ids: Optional[mindspore.Tensor] = None,
    position_ids: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    inputs_embeds: Optional[mindspore.Tensor] = None,
    encoder_hidden_states: Optional[mindspore.Tensor] = None,
    encoder_attention_mask: Optional[mindspore.Tensor] = None,
    labels: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple[mindspore.Tensor], MaskedLMOutput]:
    r"""
    Args:
        labels (`mindspore.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
            Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ...,
            config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the
            loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    outputs = self.ernie(
        input_ids,
        attention_mask=attention_mask,
        token_type_ids=token_type_ids,
        task_type_ids=task_type_ids,
        position_ids=position_ids,
        head_mask=head_mask,
        inputs_embeds=inputs_embeds,
        encoder_hidden_states=encoder_hidden_states,
        encoder_attention_mask=encoder_attention_mask,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    sequence_output = outputs[0]
    prediction_scores = self.cls(sequence_output)

    masked_lm_loss = None
    if labels is not None:
        masked_lm_loss = ops.cross_entropy(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))

    if not return_dict:
        output = (prediction_scores,) + outputs[2:]
        return ((masked_lm_loss,) + output) if masked_lm_loss is not None else output

    return MaskedLMOutput(
        loss=masked_lm_loss,
        logits=prediction_scores,
        hidden_states=outputs.hidden_states,
        attentions=outputs.attentions,
    )

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForMaskedLM.get_output_embeddings()

Retrieve the output embeddings from the ErnieForMaskedLM model.

PARAMETER DESCRIPTION
self

An instance of the ErnieForMaskedLM class. Represents the model object that contains the output embeddings.

TYPE: ErnieForMaskedLM

RETURNS DESCRIPTION
None

This method returns the output embeddings stored in the 'decoder' of the 'predictions' object within the ErnieForMaskedLM model.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
2192
2193
2194
2195
2196
2197
2198
2199
2200
2201
2202
2203
2204
2205
2206
2207
def get_output_embeddings(self):
    """
    Retrieve the output embeddings from the ErnieForMaskedLM model.

    Args:
        self (ErnieForMaskedLM): An instance of the ErnieForMaskedLM class.
            Represents the model object that contains the output embeddings.

    Returns:
        None: This method returns the output embeddings stored in the 'decoder' of the 'predictions' object
            within the ErnieForMaskedLM model.

    Raises:
        None.
    """
    return self.cls.predictions.decoder

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForMaskedLM.prepare_inputs_for_generation(input_ids, attention_mask=None, **model_kwargs)

Prepare inputs for generation.

This method prepares input data for generation in the ErnieForMaskedLM model.

PARAMETER DESCRIPTION
self

The instance of the ErnieForMaskedLM class.
input_ids

The input token IDs. Shape (batch_size, sequence_length).

TYPE: Tensor

attention_mask

The attention mask tensor. Shape (batch_size, sequence_length).

TYPE: Tensor DEFAULT: None

**model_kwargs

Additional model-specific keyword arguments.

DEFAULT: {}

RETURNS DESCRIPTION
dict

A dictionary containing the prepared input_ids and attention_mask.
RAISES DESCRIPTION
ValueError

If the PAD token is not defined for generation.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
2287
2288
2289
2290
2291
2292
2293
2294
2295
2296
2297
2298
2299
2300
2301
2302
2303
2304
2305
2306
2307
2308
2309
2310
2311
2312
2313
2314
2315
2316
2317
2318
def prepare_inputs_for_generation(self, input_ids, attention_mask=None, **model_kwargs):
    """
    Prepare inputs for generation.

    This method prepares input data for generation in the ErnieForMaskedLM model.

    Args:
        self: The instance of the ErnieForMaskedLM class.
        input_ids (Tensor): The input token IDs. Shape (batch_size, sequence_length).
        attention_mask (Tensor, optional): The attention mask tensor. Shape (batch_size, sequence_length).
        **model_kwargs: Additional model-specific keyword arguments.

    Returns:
        dict: A dictionary containing the prepared input_ids and attention_mask.

    Raises:
        ValueError: If the PAD token is not defined for generation.
    """
    input_shape = input_ids.shape
    effective_batch_size = input_shape[0]

    #  add a dummy token
    if self.config.pad_token_id is None:
        raise ValueError("The PAD token should be defined for generation")

    attention_mask = ops.cat([attention_mask, attention_mask.new_zeros((attention_mask.shape[0], 1))], axis=-1)
    dummy_token = ops.full(
        (effective_batch_size, 1), self.config.pad_token_id, dtype=mindspore.int64
    )
    input_ids = ops.cat([input_ids, dummy_token], axis=1)

    return {"input_ids": input_ids, "attention_mask": attention_mask}

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForMaskedLM.set_output_embeddings(new_embeddings)

Sets the output embeddings for the ErnieForMaskedLM model.

PARAMETER DESCRIPTION
self

The instance of the ErnieForMaskedLM class.

TYPE: ErnieForMaskedLM

new_embeddings

The new embeddings to be set for the model's output. It can be any object that is compatible with the existing model's output embeddings. The new embeddings will replace the current embeddings.

TYPE: object

RETURNS DESCRIPTION

None.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
2210
2211
2212
2213
2214
2215
2216
2217
2218
2219
2220
2221
2222
2223
2224
2225
2226
def set_output_embeddings(self, new_embeddings):
    """
    Sets the output embeddings for the ErnieForMaskedLM model.

    Args:
        self (ErnieForMaskedLM): The instance of the ErnieForMaskedLM class.
        new_embeddings (object): The new embeddings to be set for the model's output.
            It can be any object that is compatible with the existing model's output embeddings.
            The new embeddings will replace the current embeddings.

    Returns:
        None.

    Raises:
        None.
    """
    self.cls.predictions.decoder = new_embeddings

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForMultipleChoice

Bases: ErniePreTrainedModel

This class represents an Ernie model for multiple choice tasks. It inherits from the ErniePreTrainedModel class.

The ErnieForMultipleChoice class initializes an Ernie model with the given configuration. It forwards the model by passing input tensors through the Ernie model layers and applies dropout and classification layers to generate the logits for multiple choice classification.

Example 
>>> model = ErnieForMultipleChoice(config)
>>> outputs = model.forward(input_ids, attention_mask, token_type_ids, task_type_ids, position_ids, head_mask, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict)
METHOD DESCRIPTION
__init__

Initializes the ErnieForMultipleChoice class with the given configuration.
forward

Constructs the Ernie model for multiple choice tasks and returns the model outputs.
RETURNS DESCRIPTION

Union[Tuple[mindspore.Tensor], MultipleChoiceModelOutput]: The model outputs, which can include the loss, logits, hidden states, and attentions.
Note

The labels argument should be provided for computing the multiple choice classification loss. Indices in labels should be in the range [0, num_choices-1], where num_choices is the size of the second dimension of the input tensors (input_ids).

Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
2606
2607
2608
2609
2610
2611
2612
2613
2614
2615
2616
2617
2618
2619
2620
2621
2622
2623
2624
2625
2626
2627
2628
2629
2630
2631
2632
2633
2634
2635
2636
2637
2638
2639
2640
2641
2642
2643
2644
2645
2646
2647
2648
2649
2650
2651
2652
2653
2654
2655
2656
2657
2658
2659
2660
2661
2662
2663
2664
2665
2666
2667
2668
2669
2670
2671
2672
2673
2674
2675
2676
2677
2678
2679
2680
2681
2682
2683
2684
2685
2686
2687
2688
2689
2690
2691
2692
2693
2694
2695
2696
2697
2698
2699
2700
2701
2702
2703
2704
2705
2706
2707
2708
2709
2710
2711
2712
2713
2714
2715
2716
2717
2718
2719
2720
2721
2722
2723
2724
2725
2726
2727
2728
2729
2730
2731
2732
2733
2734
class ErnieForMultipleChoice(ErniePreTrainedModel):

    """
    This class represents an Ernie model for multiple choice tasks. It inherits from the ErniePreTrainedModel class.

    The ErnieForMultipleChoice class initializes an Ernie model with the given configuration.
    It forwards the model by passing input tensors through the Ernie model layers and applies dropout and
    classification layers to generate the logits for multiple choice classification.

    Example:
        ```python
        >>> model = ErnieForMultipleChoice(config)
        >>> outputs = model.forward(input_ids, attention_mask, token_type_ids, task_type_ids, position_ids, head_mask, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict)
        ```
    Args:
        config (ErnieConfig): The configuration for the Ernie model.

    Methods:
        __init__:
            Initializes the ErnieForMultipleChoice class with the given configuration.

        forward:
            Constructs the Ernie model for multiple choice tasks and returns the model outputs.

    Returns:
        Union[Tuple[mindspore.Tensor], MultipleChoiceModelOutput]:
            The model outputs, which can include the loss, logits, hidden states, and attentions.

    Note:
        The labels argument should be provided for computing the multiple choice classification loss.
        Indices in labels should be in the range [0, num_choices-1], where num_choices is the size of the second
        dimension of the input tensors (input_ids).
    """
    # Copied from transformers.models.bert.modeling_bert.BertForMultipleChoice.__init__ with Bert->Ernie,bert->ernie
    def __init__(self, config):
        """
        Initializes an instance of the ErnieForMultipleChoice class.

        Args:
            self (ErnieForMultipleChoice): The instance of the ErnieForMultipleChoice class.
            config: The configuration object containing various hyperparameters and settings for the model initialization.

        Returns:
            None.

        Raises:
            TypeError: If the input parameters are not of the expected types.
            ValueError: If the configuration object is missing required attributes.
            RuntimeError: If there are issues during model initialization or post-initialization steps.
        """
        super().__init__(config)

        self.ernie = ErnieModel(config)
        classifier_dropout = (
            config.classifier_dropout if config.classifier_dropout is not None else config.hidden_dropout_prob
        )
        self.dropout = nn.Dropout(p=classifier_dropout)
        self.classifier = nn.Linear(config.hidden_size, 1)

        # Initialize weights and apply final processing
        self.post_init()

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        token_type_ids: Optional[mindspore.Tensor] = None,
        task_type_ids: Optional[mindspore.Tensor] = None,
        position_ids: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        labels: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple[mindspore.Tensor], MultipleChoiceModelOutput]:
        r"""
        Args:
            labels (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
                Labels for computing the multiple choice classification loss. Indices should be in `[0, ...,
                num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See
                `input_ids` above)
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
        num_choices = input_ids.shape[1] if input_ids is not None else inputs_embeds.shape[1]

        input_ids = input_ids.view(-1, input_ids.shape[-1]) if input_ids is not None else None
        attention_mask = attention_mask.view(-1, attention_mask.shape[-1]) if attention_mask is not None else None
        token_type_ids = token_type_ids.view(-1, token_type_ids.shape[-1]) if token_type_ids is not None else None
        position_ids = position_ids.view(-1, position_ids.shape[-1]) if position_ids is not None else None
        inputs_embeds = (
            inputs_embeds.view(-1, inputs_embeds.shape[-2], inputs_embeds.shape[-1])
            if inputs_embeds is not None
            else None
        )

        outputs = self.ernie(
            input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            task_type_ids=task_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        pooled_output = outputs[1]

        pooled_output = self.dropout(pooled_output)
        logits = self.classifier(pooled_output)
        reshaped_logits = logits.view(-1, num_choices)

        loss = None
        if labels is not None:
            loss = ops.cross_entropy(reshaped_logits, labels)

        if not return_dict:
            output = (reshaped_logits,) + outputs[2:]
            return ((loss,) + output) if loss is not None else output

        return MultipleChoiceModelOutput(
            loss=loss,
            logits=reshaped_logits,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForMultipleChoice.__init__(config)

Initializes an instance of the ErnieForMultipleChoice class.

PARAMETER DESCRIPTION
self

The instance of the ErnieForMultipleChoice class.

TYPE: ErnieForMultipleChoice

config

The configuration object containing various hyperparameters and settings for the model initialization.
RETURNS DESCRIPTION

None.
RAISES DESCRIPTION
TypeError

If the input parameters are not of the expected types.
ValueError

If the configuration object is missing required attributes.
RuntimeError

If there are issues during model initialization or post-initialization steps.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
2640
2641
2642
2643
2644
2645
2646
2647
2648
2649
2650
2651
2652
2653
2654
2655
2656
2657
2658
2659
2660
2661
2662
2663
2664
2665
2666
def __init__(self, config):
    """
    Initializes an instance of the ErnieForMultipleChoice class.

    Args:
        self (ErnieForMultipleChoice): The instance of the ErnieForMultipleChoice class.
        config: The configuration object containing various hyperparameters and settings for the model initialization.

    Returns:
        None.

    Raises:
        TypeError: If the input parameters are not of the expected types.
        ValueError: If the configuration object is missing required attributes.
        RuntimeError: If there are issues during model initialization or post-initialization steps.
    """
    super().__init__(config)

    self.ernie = ErnieModel(config)
    classifier_dropout = (
        config.classifier_dropout if config.classifier_dropout is not None else config.hidden_dropout_prob
    )
    self.dropout = nn.Dropout(p=classifier_dropout)
    self.classifier = nn.Linear(config.hidden_size, 1)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForMultipleChoice.forward(input_ids=None, attention_mask=None, token_type_ids=None, task_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None, labels=None, output_attentions=None, output_hidden_states=None, return_dict=None)

PARAMETER DESCRIPTION
labels

Labels for computing the multiple choice classification loss. Indices should be in [0, ..., num_choices-1] where num_choices is the size of the second dimension of the input tensors. (See input_ids above)

TYPE: `mindspore.Tensor` of shape `(batch_size,)`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
2668
2669
2670
2671
2672
2673
2674
2675
2676
2677
2678
2679
2680
2681
2682
2683
2684
2685
2686
2687
2688
2689
2690
2691
2692
2693
2694
2695
2696
2697
2698
2699
2700
2701
2702
2703
2704
2705
2706
2707
2708
2709
2710
2711
2712
2713
2714
2715
2716
2717
2718
2719
2720
2721
2722
2723
2724
2725
2726
2727
2728
2729
2730
2731
2732
2733
2734
def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    token_type_ids: Optional[mindspore.Tensor] = None,
    task_type_ids: Optional[mindspore.Tensor] = None,
    position_ids: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    inputs_embeds: Optional[mindspore.Tensor] = None,
    labels: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple[mindspore.Tensor], MultipleChoiceModelOutput]:
    r"""
    Args:
        labels (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
            Labels for computing the multiple choice classification loss. Indices should be in `[0, ...,
            num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See
            `input_ids` above)
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    num_choices = input_ids.shape[1] if input_ids is not None else inputs_embeds.shape[1]

    input_ids = input_ids.view(-1, input_ids.shape[-1]) if input_ids is not None else None
    attention_mask = attention_mask.view(-1, attention_mask.shape[-1]) if attention_mask is not None else None
    token_type_ids = token_type_ids.view(-1, token_type_ids.shape[-1]) if token_type_ids is not None else None
    position_ids = position_ids.view(-1, position_ids.shape[-1]) if position_ids is not None else None
    inputs_embeds = (
        inputs_embeds.view(-1, inputs_embeds.shape[-2], inputs_embeds.shape[-1])
        if inputs_embeds is not None
        else None
    )

    outputs = self.ernie(
        input_ids,
        attention_mask=attention_mask,
        token_type_ids=token_type_ids,
        task_type_ids=task_type_ids,
        position_ids=position_ids,
        head_mask=head_mask,
        inputs_embeds=inputs_embeds,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    pooled_output = outputs[1]

    pooled_output = self.dropout(pooled_output)
    logits = self.classifier(pooled_output)
    reshaped_logits = logits.view(-1, num_choices)

    loss = None
    if labels is not None:
        loss = ops.cross_entropy(reshaped_logits, labels)

    if not return_dict:
        output = (reshaped_logits,) + outputs[2:]
        return ((loss,) + output) if loss is not None else output

    return MultipleChoiceModelOutput(
        loss=loss,
        logits=reshaped_logits,
        hidden_states=outputs.hidden_states,
        attentions=outputs.attentions,
    )

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForNextSentencePrediction

Bases: ErniePreTrainedModel

ErnieForNextSentencePrediction is a class that represents a model for next sentence prediction using the ERNIE (Enhanced Representation through kNowledge IntEgration) architecture. This class inherits from the ErniePreTrainedModel class.

The ERNIE model is designed for various natural language processing tasks, including next sentence prediction. It takes input sequences and predicts whether the second sequence follows the first sequence in a given pair.

The class's code initializes an instance of the ErnieForNextSentencePrediction class with the provided configuration. It creates an ERNIE model and a next sentence prediction head. The post_init() method is called to perform additional setup after the initialization.

The forward() method forwards the model using the provided input tensors and other optional arguments. It returns the predicted next sentence relationship scores. The method also supports computing the next sequence prediction loss if labels are provided.

The labels parameter is used to compute the next sequence prediction loss. It should be a tensor of shape (batch_size,) where each value indicates the relationship between the input sequences:

  • 0 indicates sequence B is a continuation of sequence A.
  • 1 indicates sequence B is a random sequence. The method returns a tuple of the next sentence prediction loss, the next sentence relationship scores, and other optional outputs such as hidden states and attentions.
Example 
>>> from transformers import AutoTokenizer, ErnieForNextSentencePrediction
...
>>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh")
>>> model = ErnieForNextSentencePrediction.from_pretrained("nghuyong/ernie-1.0-base-zh")
...
>>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
>>> next_sentence = "The sky is blue due to the shorter wavelength of blue light."
>>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt")
...
>>> outputs = model(**encoding, labels=mindspore.Tensor([1]))
>>> logits = outputs.logits
>>> assert logits[0, 0] < logits[0, 1]  # next sentence was random
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
2321
2322
2323
2324
2325
2326
2327
2328
2329
2330
2331
2332
2333
2334
2335
2336
2337
2338
2339
2340
2341
2342
2343
2344
2345
2346
2347
2348
2349
2350
2351
2352
2353
2354
2355
2356
2357
2358
2359
2360
2361
2362
2363
2364
2365
2366
2367
2368
2369
2370
2371
2372
2373
2374
2375
2376
2377
2378
2379
2380
2381
2382
2383
2384
2385
2386
2387
2388
2389
2390
2391
2392
2393
2394
2395
2396
2397
2398
2399
2400
2401
2402
2403
2404
2405
2406
2407
2408
2409
2410
2411
2412
2413
2414
2415
2416
2417
2418
2419
2420
2421
2422
2423
2424
2425
2426
2427
2428
2429
2430
2431
2432
2433
2434
2435
2436
2437
2438
2439
2440
2441
2442
2443
2444
2445
2446
2447
2448
2449
2450
2451
2452
2453
2454
2455
2456
2457
2458
2459
2460
2461
2462
2463
2464
2465
2466
2467
2468
2469
2470
2471
class ErnieForNextSentencePrediction(ErniePreTrainedModel):

    """
    ErnieForNextSentencePrediction is a class that represents a model for next sentence prediction using the ERNIE
    (Enhanced Representation through kNowledge IntEgration) architecture.
    This class inherits from the ErniePreTrainedModel class.

    The ERNIE model is designed for various natural language processing tasks, including next sentence prediction.
    It takes input sequences and predicts whether the second sequence follows the first sequence in a given pair.

    The class's code initializes an instance of the ErnieForNextSentencePrediction class with the provided configuration.
    It creates an ERNIE model and a next sentence prediction head.
    The post_init() method is called to perform additional setup after the initialization.

    The forward() method forwards the model using the provided input tensors and other optional arguments.
    It returns the predicted next sentence relationship scores. The method also supports computing the next sequence
    prediction loss if labels are provided.

    The labels parameter is used to compute the next sequence prediction loss.
    It should be a tensor of shape (batch_size,) where each value indicates the relationship between the input sequences:

    - 0 indicates sequence B is a continuation of sequence A.
    - 1 indicates sequence B is a random sequence.
    The method returns a tuple of the next sentence prediction loss, the next sentence relationship scores,
    and other optional outputs such as hidden states and attentions.

    Example:
        ```python
        >>> from transformers import AutoTokenizer, ErnieForNextSentencePrediction
        ...
        >>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh")
        >>> model = ErnieForNextSentencePrediction.from_pretrained("nghuyong/ernie-1.0-base-zh")
        ...
        >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
        >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light."
        >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt")
        ...
        >>> outputs = model(**encoding, labels=mindspore.Tensor([1]))
        >>> logits = outputs.logits
        >>> assert logits[0, 0] < logits[0, 1]  # next sentence was random
        ```
    """
    # Copied from transformers.models.bert.modeling_bert.BertForNextSentencePrediction.__init__ with Bert->Ernie,bert->ernie
    def __init__(self, config):
        """
        Initializes an instance of ErnieForNextSentencePrediction.

        Args:
            self (ErnieForNextSentencePrediction): The instance of the ErnieForNextSentencePrediction class.
            config (dict): The configuration dictionary containing parameters for initializing the model.
                It should include necessary settings for model configuration.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(config)

        self.ernie = ErnieModel(config)
        self.cls = ErnieOnlyNSPHead(config)

        # Initialize weights and apply final processing
        self.post_init()

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        token_type_ids: Optional[mindspore.Tensor] = None,
        task_type_ids: Optional[mindspore.Tensor] = None,
        position_ids: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        labels: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
        **kwargs,
    ) -> Union[Tuple[mindspore.Tensor], NextSentencePredictorOutput]:
        r"""
        Args:
            labels (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
                Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair
                (see `input_ids` docstring). Indices should be in `[0, 1]`:

                - 0 indicates sequence B is a continuation of sequence A,
                - 1 indicates sequence B is a random sequence.

        Returns:
            Union[Tuple[mindspore.Tensor], NextSentencePredictorOutput]

        Example:
            ```python
            >>> from transformers import AutoTokenizer, ErnieForNextSentencePrediction
            ...
            ...
            >>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh")
            >>> model = ErnieForNextSentencePrediction.from_pretrained("nghuyong/ernie-1.0-base-zh")
            ...
            >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
            >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light."
            >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt")
            ...
            >>> outputs = model(**encoding, labels=mindspore.Tensor([1]))
            >>> logits = outputs.logits
            >>> assert logits[0, 0] < logits[0, 1]  # next sentence was random
            ```
        """
        if "next_sentence_label" in kwargs:
            warnings.warn(
                "The `next_sentence_label` argument is deprecated and will be removed in a future version, use"
                " `labels` instead.",
                FutureWarning,
            )
            labels = kwargs.pop("next_sentence_label")

        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        outputs = self.ernie(
            input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            task_type_ids=task_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        pooled_output = outputs[1]

        seq_relationship_scores = self.cls(pooled_output)

        next_sentence_loss = None
        if labels is not None:
            next_sentence_loss = ops.cross_entropy(seq_relationship_scores.view(-1, 2), labels.view(-1))

        if not return_dict:
            output = (seq_relationship_scores,) + outputs[2:]
            return ((next_sentence_loss,) + output) if next_sentence_loss is not None else output

        return NextSentencePredictorOutput(
            loss=next_sentence_loss,
            logits=seq_relationship_scores,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForNextSentencePrediction.__init__(config)

Initializes an instance of ErnieForNextSentencePrediction.

PARAMETER DESCRIPTION
self

The instance of the ErnieForNextSentencePrediction class.

TYPE: ErnieForNextSentencePrediction

config

The configuration dictionary containing parameters for initializing the model. It should include necessary settings for model configuration.

TYPE: dict

RETURNS DESCRIPTION

None.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
2364
2365
2366
2367
2368
2369
2370
2371
2372
2373
2374
2375
2376
2377
2378
2379
2380
2381
2382
2383
2384
2385
def __init__(self, config):
    """
    Initializes an instance of ErnieForNextSentencePrediction.

    Args:
        self (ErnieForNextSentencePrediction): The instance of the ErnieForNextSentencePrediction class.
        config (dict): The configuration dictionary containing parameters for initializing the model.
            It should include necessary settings for model configuration.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(config)

    self.ernie = ErnieModel(config)
    self.cls = ErnieOnlyNSPHead(config)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForNextSentencePrediction.forward(input_ids=None, attention_mask=None, token_type_ids=None, task_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None, labels=None, output_attentions=None, output_hidden_states=None, return_dict=None, **kwargs)

PARAMETER DESCRIPTION
labels

Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see input_ids docstring). Indices should be in [0, 1]:

  • 0 indicates sequence B is a continuation of sequence A,
  • 1 indicates sequence B is a random sequence.

TYPE: `mindspore.Tensor` of shape `(batch_size,)`, *optional* DEFAULT: None

RETURNS DESCRIPTION
Union[Tuple[Tensor], NextSentencePredictorOutput]

Union[Tuple[mindspore.Tensor], NextSentencePredictorOutput]
Example 
>>> from transformers import AutoTokenizer, ErnieForNextSentencePrediction
...
...
>>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh")
>>> model = ErnieForNextSentencePrediction.from_pretrained("nghuyong/ernie-1.0-base-zh")
...
>>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
>>> next_sentence = "The sky is blue due to the shorter wavelength of blue light."
>>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt")
...
>>> outputs = model(**encoding, labels=mindspore.Tensor([1]))
>>> logits = outputs.logits
>>> assert logits[0, 0] < logits[0, 1]  # next sentence was random
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
2387
2388
2389
2390
2391
2392
2393
2394
2395
2396
2397
2398
2399
2400
2401
2402
2403
2404
2405
2406
2407
2408
2409
2410
2411
2412
2413
2414
2415
2416
2417
2418
2419
2420
2421
2422
2423
2424
2425
2426
2427
2428
2429
2430
2431
2432
2433
2434
2435
2436
2437
2438
2439
2440
2441
2442
2443
2444
2445
2446
2447
2448
2449
2450
2451
2452
2453
2454
2455
2456
2457
2458
2459
2460
2461
2462
2463
2464
2465
2466
2467
2468
2469
2470
2471
def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    token_type_ids: Optional[mindspore.Tensor] = None,
    task_type_ids: Optional[mindspore.Tensor] = None,
    position_ids: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    inputs_embeds: Optional[mindspore.Tensor] = None,
    labels: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
    **kwargs,
) -> Union[Tuple[mindspore.Tensor], NextSentencePredictorOutput]:
    r"""
    Args:
        labels (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
            Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair
            (see `input_ids` docstring). Indices should be in `[0, 1]`:

            - 0 indicates sequence B is a continuation of sequence A,
            - 1 indicates sequence B is a random sequence.

    Returns:
        Union[Tuple[mindspore.Tensor], NextSentencePredictorOutput]

    Example:
        ```python
        >>> from transformers import AutoTokenizer, ErnieForNextSentencePrediction
        ...
        ...
        >>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh")
        >>> model = ErnieForNextSentencePrediction.from_pretrained("nghuyong/ernie-1.0-base-zh")
        ...
        >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
        >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light."
        >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt")
        ...
        >>> outputs = model(**encoding, labels=mindspore.Tensor([1]))
        >>> logits = outputs.logits
        >>> assert logits[0, 0] < logits[0, 1]  # next sentence was random
        ```
    """
    if "next_sentence_label" in kwargs:
        warnings.warn(
            "The `next_sentence_label` argument is deprecated and will be removed in a future version, use"
            " `labels` instead.",
            FutureWarning,
        )
        labels = kwargs.pop("next_sentence_label")

    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    outputs = self.ernie(
        input_ids,
        attention_mask=attention_mask,
        token_type_ids=token_type_ids,
        task_type_ids=task_type_ids,
        position_ids=position_ids,
        head_mask=head_mask,
        inputs_embeds=inputs_embeds,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    pooled_output = outputs[1]

    seq_relationship_scores = self.cls(pooled_output)

    next_sentence_loss = None
    if labels is not None:
        next_sentence_loss = ops.cross_entropy(seq_relationship_scores.view(-1, 2), labels.view(-1))

    if not return_dict:
        output = (seq_relationship_scores,) + outputs[2:]
        return ((next_sentence_loss,) + output) if next_sentence_loss is not None else output

    return NextSentencePredictorOutput(
        loss=next_sentence_loss,
        logits=seq_relationship_scores,
        hidden_states=outputs.hidden_states,
        attentions=outputs.attentions,
    )

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForPreTraining

Bases: ErniePreTrainedModel

This class represents an Ernie model for pre-training tasks. It inherits from the ErniePreTrainedModel.

The class includes methods for initializing the model, getting and setting output embeddings, and forwarding the model for pre-training tasks. The forward method takes various input tensors and optional arguments, and returns the output of the model for pre-training. It also includes detailed information about the expected input parameters, optional arguments, and return values.

The class also provides an example of how to use the model for pre-training tasks using the AutoTokenizer and example inputs. The example demonstrates how to tokenize input text, generate model outputs, and access specific logits from the model.

For more details on the usage and functionality of the ErnieForPreTraining class, refer to the provided code and docstring examples.

Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
1703
1704
1705
1706
1707
1708
1709
1710
1711
1712
1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
1725
1726
1727
1728
1729
1730
1731
1732
1733
1734
1735
1736
1737
1738
1739
1740
1741
1742
1743
1744
1745
1746
1747
1748
1749
1750
1751
1752
1753
1754
1755
1756
1757
1758
1759
1760
1761
1762
1763
1764
1765
1766
1767
1768
1769
1770
1771
1772
1773
1774
1775
1776
1777
1778
1779
1780
1781
1782
1783
1784
1785
1786
1787
1788
1789
1790
1791
1792
1793
1794
1795
1796
1797
1798
1799
1800
1801
1802
1803
1804
1805
1806
1807
1808
1809
1810
1811
1812
1813
1814
1815
1816
1817
1818
1819
1820
1821
1822
1823
1824
1825
1826
1827
1828
1829
1830
1831
1832
1833
1834
1835
1836
1837
1838
1839
1840
1841
1842
1843
1844
1845
1846
1847
1848
1849
1850
1851
1852
1853
1854
1855
1856
1857
1858
1859
1860
class ErnieForPreTraining(ErniePreTrainedModel):

    """
    This class represents an Ernie model for pre-training tasks. It inherits from the ErniePreTrainedModel.

    The class includes methods for initializing the model, getting and setting output embeddings, and forwarding the
    model for pre-training tasks. The `forward` method takes various input tensors and optional arguments, and returns
    the output of the model for pre-training. It also includes detailed information about the expected input parameters,
    optional arguments, and return values.

    The class also provides an example of how to use the model for pre-training tasks using the AutoTokenizer and
    example inputs. The example demonstrates how to tokenize input text, generate model outputs, and access specific
    logits from the model.

    For more details on the usage and functionality of the ErnieForPreTraining class, refer to the provided code and
    docstring examples.
    """
    _tied_weights_keys = ["cls.predictions.decoder.bias", "cls.predictions.decoder.weight"]

    # Copied from transformers.models.bert.modeling_bert.BertForPreTraining.__init__ with Bert->Ernie,bert->ernie
    def __init__(self, config):
        """Initializes an instance of the ErnieForPreTraining class.

        Args:
            self (ErnieForPreTraining): An instance of the ErnieForPreTraining class.
            config (object): The configuration object for the ErnieForPreTraining class.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(config)

        self.ernie = ErnieModel(config)
        self.cls = ErniePreTrainingHeads(config)

        # Initialize weights and apply final processing
        self.post_init()

    # Copied from transformers.models.bert.modeling_bert.BertForPreTraining.get_output_embeddings
    def get_output_embeddings(self):
        """
        Method to retrieve the output embeddings from the ErnieForPreTraining model.

        Args:
            self (ErnieForPreTraining): The instance of the ErnieForPreTraining class.

        Returns:
            None: This method does not return anything but directly accesses and returns the output embeddings
                from the model.

        Raises:
            None.
        """
        return self.cls.predictions.decoder

    # Copied from transformers.models.bert.modeling_bert.BertForPreTraining.set_output_embeddings
    def set_output_embeddings(self, new_embeddings):
        """
        Sets the output embeddings for the ErnieForPreTraining model.

        Args:
            self (ErnieForPreTraining): An instance of the ErnieForPreTraining class.
            new_embeddings: The new embeddings to be set for the model predictions decoder. This can be of any type.

        Returns:
            None.

        Raises:
            None.
        """
        self.cls.predictions.decoder = new_embeddings

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        token_type_ids: Optional[mindspore.Tensor] = None,
        task_type_ids: Optional[mindspore.Tensor] = None,
        position_ids: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        labels: Optional[mindspore.Tensor] = None,
        next_sentence_label: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple[mindspore.Tensor], ErnieForPreTrainingOutput]:
        r"""
        Args:
            labels (`mindspore.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
                Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ...,
                config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked),
                the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`
            next_sentence_label (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
                Labels for computing the next sequence prediction (classification) loss. Input should be a sequence
                pair (see `input_ids` docstring) Indices should be in `[0, 1]`:

                - 0 indicates sequence B is a continuation of sequence A,
                - 1 indicates sequence B is a random sequence.
            kwargs (`Dict[str, any]`, optional, defaults to *{}*):
                Used to hide legacy arguments that have been deprecated.

        Returns:
            Union[Tuple[mindspore.Tensor], ErnieForPreTrainingOutput]

        Example:
            ```python
            >>> from transformers import AutoTokenizer, ErnieForPreTraining
            ...
            ...
            >>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh")
            >>> model = ErnieForPreTraining.from_pretrained("nghuyong/ernie-1.0-base-zh")
            ...
            >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
            >>> outputs = model(**inputs)
            ...
            >>> prediction_logits = outputs.prediction_logits
            >>> seq_relationship_logits = outputs.seq_relationship_logits
            ```
            """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        outputs = self.ernie(
            input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            task_type_ids=task_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        sequence_output, pooled_output = outputs[:2]
        prediction_scores, seq_relationship_score = self.cls(sequence_output, pooled_output)

        total_loss = None
        if labels is not None and next_sentence_label is not None:
            masked_lm_loss = ops.cross_entropy(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))
            next_sentence_loss = ops.cross_entropy(seq_relationship_score.view(-1, 2), next_sentence_label.view(-1))
            total_loss = masked_lm_loss + next_sentence_loss

        if not return_dict:
            output = (prediction_scores, seq_relationship_score) + outputs[2:]
            return ((total_loss,) + output) if total_loss is not None else output

        return ErnieForPreTrainingOutput(
            loss=total_loss,
            prediction_logits=prediction_scores,
            seq_relationship_logits=seq_relationship_score,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForPreTraining.__init__(config)

Initializes an instance of the ErnieForPreTraining class.

PARAMETER DESCRIPTION
self

An instance of the ErnieForPreTraining class.

TYPE: ErnieForPreTraining

config

The configuration object for the ErnieForPreTraining class.

TYPE: object

RETURNS DESCRIPTION

None.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
1723
1724
1725
1726
1727
1728
1729
1730
1731
1732
1733
1734
1735
1736
1737
1738
1739
1740
1741
1742
def __init__(self, config):
    """Initializes an instance of the ErnieForPreTraining class.

    Args:
        self (ErnieForPreTraining): An instance of the ErnieForPreTraining class.
        config (object): The configuration object for the ErnieForPreTraining class.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(config)

    self.ernie = ErnieModel(config)
    self.cls = ErniePreTrainingHeads(config)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForPreTraining.forward(input_ids=None, attention_mask=None, token_type_ids=None, task_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None, labels=None, next_sentence_label=None, output_attentions=None, output_hidden_states=None, return_dict=None)

PARAMETER DESCRIPTION
labels

Labels for computing the masked language modeling loss. Indices should be in [-100, 0, ..., config.vocab_size] (see input_ids docstring) Tokens with indices set to -100 are ignored (masked), the loss is only computed for the tokens with labels in [0, ..., config.vocab_size]

TYPE: `mindspore.Tensor` of shape `(batch_size, sequence_length)`, *optional* DEFAULT: None

next_sentence_label

Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see input_ids docstring) Indices should be in [0, 1]:

  • 0 indicates sequence B is a continuation of sequence A,
  • 1 indicates sequence B is a random sequence.

TYPE: `mindspore.Tensor` of shape `(batch_size,)`, *optional* DEFAULT: None

kwargs

Used to hide legacy arguments that have been deprecated.

TYPE: `Dict[str, any]`, optional, defaults to *{}*

RETURNS DESCRIPTION
Union[Tuple[Tensor], ErnieForPreTrainingOutput]

Union[Tuple[mindspore.Tensor], ErnieForPreTrainingOutput]
Example 
>>> from transformers import AutoTokenizer, ErnieForPreTraining
...
...
>>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh")
>>> model = ErnieForPreTraining.from_pretrained("nghuyong/ernie-1.0-base-zh")
...
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> outputs = model(**inputs)
...
>>> prediction_logits = outputs.prediction_logits
>>> seq_relationship_logits = outputs.seq_relationship_logits
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
1778
1779
1780
1781
1782
1783
1784
1785
1786
1787
1788
1789
1790
1791
1792
1793
1794
1795
1796
1797
1798
1799
1800
1801
1802
1803
1804
1805
1806
1807
1808
1809
1810
1811
1812
1813
1814
1815
1816
1817
1818
1819
1820
1821
1822
1823
1824
1825
1826
1827
1828
1829
1830
1831
1832
1833
1834
1835
1836
1837
1838
1839
1840
1841
1842
1843
1844
1845
1846
1847
1848
1849
1850
1851
1852
1853
1854
1855
1856
1857
1858
1859
1860
def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    token_type_ids: Optional[mindspore.Tensor] = None,
    task_type_ids: Optional[mindspore.Tensor] = None,
    position_ids: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    inputs_embeds: Optional[mindspore.Tensor] = None,
    labels: Optional[mindspore.Tensor] = None,
    next_sentence_label: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple[mindspore.Tensor], ErnieForPreTrainingOutput]:
    r"""
    Args:
        labels (`mindspore.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
            Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ...,
            config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked),
            the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`
        next_sentence_label (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
            Labels for computing the next sequence prediction (classification) loss. Input should be a sequence
            pair (see `input_ids` docstring) Indices should be in `[0, 1]`:

            - 0 indicates sequence B is a continuation of sequence A,
            - 1 indicates sequence B is a random sequence.
        kwargs (`Dict[str, any]`, optional, defaults to *{}*):
            Used to hide legacy arguments that have been deprecated.

    Returns:
        Union[Tuple[mindspore.Tensor], ErnieForPreTrainingOutput]

    Example:
        ```python
        >>> from transformers import AutoTokenizer, ErnieForPreTraining
        ...
        ...
        >>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh")
        >>> model = ErnieForPreTraining.from_pretrained("nghuyong/ernie-1.0-base-zh")
        ...
        >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
        >>> outputs = model(**inputs)
        ...
        >>> prediction_logits = outputs.prediction_logits
        >>> seq_relationship_logits = outputs.seq_relationship_logits
        ```
        """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    outputs = self.ernie(
        input_ids,
        attention_mask=attention_mask,
        token_type_ids=token_type_ids,
        task_type_ids=task_type_ids,
        position_ids=position_ids,
        head_mask=head_mask,
        inputs_embeds=inputs_embeds,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    sequence_output, pooled_output = outputs[:2]
    prediction_scores, seq_relationship_score = self.cls(sequence_output, pooled_output)

    total_loss = None
    if labels is not None and next_sentence_label is not None:
        masked_lm_loss = ops.cross_entropy(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))
        next_sentence_loss = ops.cross_entropy(seq_relationship_score.view(-1, 2), next_sentence_label.view(-1))
        total_loss = masked_lm_loss + next_sentence_loss

    if not return_dict:
        output = (prediction_scores, seq_relationship_score) + outputs[2:]
        return ((total_loss,) + output) if total_loss is not None else output

    return ErnieForPreTrainingOutput(
        loss=total_loss,
        prediction_logits=prediction_scores,
        seq_relationship_logits=seq_relationship_score,
        hidden_states=outputs.hidden_states,
        attentions=outputs.attentions,
    )

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForPreTraining.get_output_embeddings()

Method to retrieve the output embeddings from the ErnieForPreTraining model.

PARAMETER DESCRIPTION
self

The instance of the ErnieForPreTraining class.

TYPE: ErnieForPreTraining

RETURNS DESCRIPTION
None

This method does not return anything but directly accesses and returns the output embeddings from the model.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
1745
1746
1747
1748
1749
1750
1751
1752
1753
1754
1755
1756
1757
1758
1759
def get_output_embeddings(self):
    """
    Method to retrieve the output embeddings from the ErnieForPreTraining model.

    Args:
        self (ErnieForPreTraining): The instance of the ErnieForPreTraining class.

    Returns:
        None: This method does not return anything but directly accesses and returns the output embeddings
            from the model.

    Raises:
        None.
    """
    return self.cls.predictions.decoder

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForPreTraining.set_output_embeddings(new_embeddings)

Sets the output embeddings for the ErnieForPreTraining model.

PARAMETER DESCRIPTION
self

An instance of the ErnieForPreTraining class.

TYPE: ErnieForPreTraining

new_embeddings

The new embeddings to be set for the model predictions decoder. This can be of any type.
RETURNS DESCRIPTION

None.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
1762
1763
1764
1765
1766
1767
1768
1769
1770
1771
1772
1773
1774
1775
1776
def set_output_embeddings(self, new_embeddings):
    """
    Sets the output embeddings for the ErnieForPreTraining model.

    Args:
        self (ErnieForPreTraining): An instance of the ErnieForPreTraining class.
        new_embeddings: The new embeddings to be set for the model predictions decoder. This can be of any type.

    Returns:
        None.

    Raises:
        None.
    """
    self.cls.predictions.decoder = new_embeddings

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForPreTrainingOutput dataclass

Bases: ModelOutput

Output type of [ErnieForPreTraining].

PARAMETER DESCRIPTION
loss

Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss.

TYPE: *optional*, returned when `labels` is provided, `mindspore.Tensor` of shape `(1,)` DEFAULT: None

prediction_logits

Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

TYPE: `mindspore.Tensor` of shape `(batch_size, sequence_length, config.vocab_size)` DEFAULT: None

seq_relationship_logits

Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax).

TYPE: `mindspore.Tensor` of shape `(batch_size, 2)` DEFAULT: None

hidden_states

Tuple of mindspore.Tensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

Hidden-states of the model at the output of each layer plus the initial embedding outputs.

TYPE: `tuple(mindspore.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True` DEFAULT: None

attentions

Tuple of mindspore.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

TYPE: `tuple(mindspore.Tensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True` DEFAULT: None

Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
@dataclass
# Copied from transformers.models.bert.modeling_bert.BertForPreTrainingOutput with Bert->Ernie
class ErnieForPreTrainingOutput(ModelOutput):
    """
    Output type of [`ErnieForPreTraining`].

    Args:
        loss (*optional*, returned when `labels` is provided, `mindspore.Tensor` of shape `(1,)`):
            Total loss as the sum of the masked language modeling loss and the next sequence prediction
            (classification) loss.
        prediction_logits (`mindspore.Tensor` of shape `(batch_size, sequence_length, config.vocab_size)`):
            Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
        seq_relationship_logits (`mindspore.Tensor` of shape `(batch_size, 2)`):
            Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation
            before SoftMax).
        hidden_states (`tuple(mindspore.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`):
            Tuple of `mindspore.Tensor` (one for the output of the embeddings + one for the output of each layer) of
            shape `(batch_size, sequence_length, hidden_size)`.

            Hidden-states of the model at the output of each layer plus the initial embedding outputs.
        attentions (`tuple(mindspore.Tensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`):
            Tuple of `mindspore.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length,
            sequence_length)`.

            Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
            heads.
    """
    loss: Optional[mindspore.Tensor] = None
    prediction_logits: mindspore.Tensor = None
    seq_relationship_logits: mindspore.Tensor = None
    hidden_states: Optional[Tuple[mindspore.Tensor]] = None
    attentions: Optional[Tuple[mindspore.Tensor]] = None

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForQuestionAnswering

Bases: ErniePreTrainedModel

ErnieForQuestionAnswering is a class that represents a model for question answering tasks using the ERNIE (Enhanced Representation through kNowledge Integration) architecture. This class inherits from ErniePreTrainedModel and provides methods for forwarding the model and performing question answering inference.

The class forwardor initializes the model with the provided configuration. The model architecture includes an ERNIE model with the option to add a pooling layer. Additionally, it includes a dense layer for question answering outputs.

The forward method takes various input tensors and performs the question answering computation. It supports optional inputs for start and end positions, attention masks, token type IDs, task type IDs, position IDs, head masks, and input embeddings. The method returns the question-answering model output, which includes the start and end logits for the predicted answer spans.

The method also allows for customizing the return of outputs by specifying the return_dict parameter. If the return_dict parameter is not provided, the method uses the default value from the model's configuration.

Overall, the ErnieForQuestionAnswering class encapsulates the functionality for performing question answering tasks using the ERNIE model and provides a high-level interface for forwarding the model and performing inference.

Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
2848
2849
2850
2851
2852
2853
2854
2855
2856
2857
2858
2859
2860
2861
2862
2863
2864
2865
2866
2867
2868
2869
2870
2871
2872
2873
2874
2875
2876
2877
2878
2879
2880
2881
2882
2883
2884
2885
2886
2887
2888
2889
2890
2891
2892
2893
2894
2895
2896
2897
2898
2899
2900
2901
2902
2903
2904
2905
2906
2907
2908
2909
2910
2911
2912
2913
2914
2915
2916
2917
2918
2919
2920
2921
2922
2923
2924
2925
2926
2927
2928
2929
2930
2931
2932
2933
2934
2935
2936
2937
2938
2939
2940
2941
2942
2943
2944
2945
2946
2947
2948
2949
2950
2951
2952
2953
2954
2955
2956
2957
2958
2959
2960
2961
2962
2963
2964
2965
2966
2967
2968
2969
2970
2971
2972
2973
2974
2975
2976
2977
2978
class ErnieForQuestionAnswering(ErniePreTrainedModel):

    """
    ErnieForQuestionAnswering is a class that represents a model for question answering tasks using the
    ERNIE (Enhanced Representation through kNowledge Integration) architecture.
    This class inherits from ErniePreTrainedModel and provides methods for forwarding the model and
    performing question answering inference.

    The class forwardor initializes the model with the provided configuration.
    The model architecture includes an ERNIE model with the option to add a pooling layer.
    Additionally, it includes a dense layer for question answering outputs.

    The forward method takes various input tensors and performs the question answering computation.
    It supports optional inputs for start and end positions, attention masks, token type IDs, task type IDs,
    position IDs, head masks, and input embeddings.
    The method returns the question-answering model output, which includes the start and end logits for the predicted
    answer spans.

    The method also allows for customizing the return of outputs by specifying the return_dict parameter.
    If the return_dict parameter is not provided, the method uses the default value from the model's configuration.

    Overall, the ErnieForQuestionAnswering class encapsulates the functionality for performing question answering tasks
    using the ERNIE model and provides a high-level interface for forwarding the model and
    performing inference.
    """
    # Copied from transformers.models.bert.modeling_bert.BertForQuestionAnswering.__init__ with Bert->Ernie,bert->ernie
    def __init__(self, config):
        """
        Initializes an instance of the ErnieForQuestionAnswering class.

        Args:
            self (ErnieForQuestionAnswering): The object instance of the ErnieForQuestionAnswering class.
            config (object): An object containing configuration settings for the Ernie model.
                This parameter is required for initializing the ErnieForQuestionAnswering instance.
                It should include the following attributes:

                - num_labels (int): The number of labels for the classification task.
                - hidden_size (int): The size of the hidden layers in the model.
                - add_pooling_layer (bool): Flag indicating whether to add a pooling layer in the Ernie model.

        Returns:
            None.

        Raises:
            TypeError: If the config parameter is not of the expected object type.
            ValueError: If the config object is missing any required attributes.
        """
        super().__init__(config)
        self.num_labels = config.num_labels

        self.ernie = ErnieModel(config, add_pooling_layer=False)
        self.qa_outputs = nn.Linear(config.hidden_size, config.num_labels)

        # Initialize weights and apply final processing
        self.post_init()

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        token_type_ids: Optional[mindspore.Tensor] = None,
        task_type_ids: Optional[mindspore.Tensor] = None,
        position_ids: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        start_positions: Optional[mindspore.Tensor] = None,
        end_positions: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple[mindspore.Tensor], QuestionAnsweringModelOutput]:
        r"""
        Args:
            start_positions (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
                Labels for position (index) of the start of the labelled span for computing the token classification loss.
                Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
                are not taken into account for computing the loss.
            end_positions (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
                Labels for position (index) of the end of the labelled span for computing the token classification loss.
                Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
                are not taken into account for computing the loss.
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        outputs = self.ernie(
            input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            task_type_ids=task_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        sequence_output = outputs[0]

        logits = self.qa_outputs(sequence_output)
        start_logits, end_logits = logits.split(1, axis=-1)
        start_logits = start_logits.squeeze(-1)
        end_logits = end_logits.squeeze(-1)

        total_loss = None
        if start_positions is not None and end_positions is not None:
            # If we are on multi-GPU, split add a dimension
            if len(start_positions.shape) > 1:
                start_positions = start_positions.squeeze(-1)
            if len(end_positions.shape) > 1:
                end_positions = end_positions.squeeze(-1)
            # sometimes the start/end positions are outside our model inputs, we ignore these terms
            ignored_index = start_logits.shape[1]
            start_positions = start_positions.clamp(0, ignored_index)
            end_positions = end_positions.clamp(0, ignored_index)

            start_loss = ops.cross_entropy(start_logits, start_positions, ignore_index=ignored_index)
            end_loss = ops.cross_entropy(end_logits, end_positions, ignore_index=ignored_index)
            total_loss = (start_loss + end_loss) / 2

        if not return_dict:
            output = (start_logits, end_logits) + outputs[2:]
            return ((total_loss,) + output) if total_loss is not None else output

        return QuestionAnsweringModelOutput(
            loss=total_loss,
            start_logits=start_logits,
            end_logits=end_logits,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForQuestionAnswering.__init__(config)

Initializes an instance of the ErnieForQuestionAnswering class.

PARAMETER DESCRIPTION
self

The object instance of the ErnieForQuestionAnswering class.

TYPE: ErnieForQuestionAnswering

config

An object containing configuration settings for the Ernie model. This parameter is required for initializing the ErnieForQuestionAnswering instance. It should include the following attributes:

  • num_labels (int): The number of labels for the classification task.
  • hidden_size (int): The size of the hidden layers in the model.
  • add_pooling_layer (bool): Flag indicating whether to add a pooling layer in the Ernie model.

TYPE: object

RETURNS DESCRIPTION

None.
RAISES DESCRIPTION
TypeError

If the config parameter is not of the expected object type.
ValueError

If the config object is missing any required attributes.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
2874
2875
2876
2877
2878
2879
2880
2881
2882
2883
2884
2885
2886
2887
2888
2889
2890
2891
2892
2893
2894
2895
2896
2897
2898
2899
2900
2901
2902
def __init__(self, config):
    """
    Initializes an instance of the ErnieForQuestionAnswering class.

    Args:
        self (ErnieForQuestionAnswering): The object instance of the ErnieForQuestionAnswering class.
        config (object): An object containing configuration settings for the Ernie model.
            This parameter is required for initializing the ErnieForQuestionAnswering instance.
            It should include the following attributes:

            - num_labels (int): The number of labels for the classification task.
            - hidden_size (int): The size of the hidden layers in the model.
            - add_pooling_layer (bool): Flag indicating whether to add a pooling layer in the Ernie model.

    Returns:
        None.

    Raises:
        TypeError: If the config parameter is not of the expected object type.
        ValueError: If the config object is missing any required attributes.
    """
    super().__init__(config)
    self.num_labels = config.num_labels

    self.ernie = ErnieModel(config, add_pooling_layer=False)
    self.qa_outputs = nn.Linear(config.hidden_size, config.num_labels)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForQuestionAnswering.forward(input_ids=None, attention_mask=None, token_type_ids=None, task_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None, start_positions=None, end_positions=None, output_attentions=None, output_hidden_states=None, return_dict=None)

PARAMETER DESCRIPTION
start_positions

Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence are not taken into account for computing the loss.

TYPE: `mindspore.Tensor` of shape `(batch_size,)`, *optional* DEFAULT: None

end_positions

Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence are not taken into account for computing the loss.

TYPE: `mindspore.Tensor` of shape `(batch_size,)`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
2904
2905
2906
2907
2908
2909
2910
2911
2912
2913
2914
2915
2916
2917
2918
2919
2920
2921
2922
2923
2924
2925
2926
2927
2928
2929
2930
2931
2932
2933
2934
2935
2936
2937
2938
2939
2940
2941
2942
2943
2944
2945
2946
2947
2948
2949
2950
2951
2952
2953
2954
2955
2956
2957
2958
2959
2960
2961
2962
2963
2964
2965
2966
2967
2968
2969
2970
2971
2972
2973
2974
2975
2976
2977
2978
def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    token_type_ids: Optional[mindspore.Tensor] = None,
    task_type_ids: Optional[mindspore.Tensor] = None,
    position_ids: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    inputs_embeds: Optional[mindspore.Tensor] = None,
    start_positions: Optional[mindspore.Tensor] = None,
    end_positions: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple[mindspore.Tensor], QuestionAnsweringModelOutput]:
    r"""
    Args:
        start_positions (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
            Labels for position (index) of the start of the labelled span for computing the token classification loss.
            Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
            are not taken into account for computing the loss.
        end_positions (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
            Labels for position (index) of the end of the labelled span for computing the token classification loss.
            Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
            are not taken into account for computing the loss.
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    outputs = self.ernie(
        input_ids,
        attention_mask=attention_mask,
        token_type_ids=token_type_ids,
        task_type_ids=task_type_ids,
        position_ids=position_ids,
        head_mask=head_mask,
        inputs_embeds=inputs_embeds,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    sequence_output = outputs[0]

    logits = self.qa_outputs(sequence_output)
    start_logits, end_logits = logits.split(1, axis=-1)
    start_logits = start_logits.squeeze(-1)
    end_logits = end_logits.squeeze(-1)

    total_loss = None
    if start_positions is not None and end_positions is not None:
        # If we are on multi-GPU, split add a dimension
        if len(start_positions.shape) > 1:
            start_positions = start_positions.squeeze(-1)
        if len(end_positions.shape) > 1:
            end_positions = end_positions.squeeze(-1)
        # sometimes the start/end positions are outside our model inputs, we ignore these terms
        ignored_index = start_logits.shape[1]
        start_positions = start_positions.clamp(0, ignored_index)
        end_positions = end_positions.clamp(0, ignored_index)

        start_loss = ops.cross_entropy(start_logits, start_positions, ignore_index=ignored_index)
        end_loss = ops.cross_entropy(end_logits, end_positions, ignore_index=ignored_index)
        total_loss = (start_loss + end_loss) / 2

    if not return_dict:
        output = (start_logits, end_logits) + outputs[2:]
        return ((total_loss,) + output) if total_loss is not None else output

    return QuestionAnsweringModelOutput(
        loss=total_loss,
        start_logits=start_logits,
        end_logits=end_logits,
        hidden_states=outputs.hidden_states,
        attentions=outputs.attentions,
    )

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForSequenceClassification

Bases: ErniePreTrainedModel

This class represents an ERNIE model for sequence classification tasks. It is a subclass of the ErniePreTrainedModel class.

The ErnieForSequenceClassification class has an initialization method and a forward method. The initialization method initializes the ERNIE model and sets up the classifier layers. The forward method performs the forward pass of the model and returns the output.

ATTRIBUTE DESCRIPTION
num_labels

The number of labels for the sequence classification task.

TYPE: int

config

The configuration object for the ERNIE model.

TYPE: ErnieConfig

ernie

The ERNIE model instance.

TYPE: ErnieModel

dropout

Dropout layer for regularization.

TYPE: Dropout

classifier

Dense layer for classification.

TYPE: Linear

METHOD DESCRIPTION
__init__

Initializes the ErnieForSequenceClassification instance.
forward

Performs the forward pass of the ERNIE model and returns the output.
Example 
>>> # Initialize the model
>>> model = ErnieForSequenceClassification(config)
...
>>> # Perform forward pass
>>> output = model.forward(input_ids, attention_mask, token_type_ids, task_type_ids, position_ids, head_mask, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict)
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
2474
2475
2476
2477
2478
2479
2480
2481
2482
2483
2484
2485
2486
2487
2488
2489
2490
2491
2492
2493
2494
2495
2496
2497
2498
2499
2500
2501
2502
2503
2504
2505
2506
2507
2508
2509
2510
2511
2512
2513
2514
2515
2516
2517
2518
2519
2520
2521
2522
2523
2524
2525
2526
2527
2528
2529
2530
2531
2532
2533
2534
2535
2536
2537
2538
2539
2540
2541
2542
2543
2544
2545
2546
2547
2548
2549
2550
2551
2552
2553
2554
2555
2556
2557
2558
2559
2560
2561
2562
2563
2564
2565
2566
2567
2568
2569
2570
2571
2572
2573
2574
2575
2576
2577
2578
2579
2580
2581
2582
2583
2584
2585
2586
2587
2588
2589
2590
2591
2592
2593
2594
2595
2596
2597
2598
2599
2600
2601
2602
2603
class ErnieForSequenceClassification(ErniePreTrainedModel):

    """
    This class represents an ERNIE model for sequence classification tasks.
    It is a subclass of the `ErniePreTrainedModel` class.

    The `ErnieForSequenceClassification` class has an initialization method and a `forward` method.
    The initialization method initializes the ERNIE model and sets up the classifier layers.
    The `forward` method performs the forward pass of the model and returns the output.

    Attributes:
        num_labels (int): The number of labels for the sequence classification task.
        config (ErnieConfig): The configuration object for the ERNIE model.
        ernie (ErnieModel): The ERNIE model instance.
        dropout (nn.Dropout): Dropout layer for regularization.
        classifier (nn.Linear): Dense layer for classification.

    Methods:
        __init__: Initializes the `ErnieForSequenceClassification` instance.
        forward: Performs the forward
            pass of the ERNIE model and returns the output.

    Example:
        ```python
        >>> # Initialize the model
        >>> model = ErnieForSequenceClassification(config)
        ...
        >>> # Perform forward pass
        >>> output = model.forward(input_ids, attention_mask, token_type_ids, task_type_ids, position_ids, head_mask, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict)
        ```
    """
    # Copied from transformers.models.bert.modeling_bert.BertForSequenceClassification.__init__ with Bert->Ernie,bert->ernie
    def __init__(self, config):
        """
        Initializes an instance of the 'ErnieForSequenceClassification' class.

        Args:
            self: The instance of the class.
            config: An instance of 'Config' class containing the configuration parameters for the model.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(config)
        self.num_labels = config.num_labels
        self.config = config

        self.ernie = ErnieModel(config)
        classifier_dropout = (
            config.classifier_dropout if config.classifier_dropout is not None else config.hidden_dropout_prob
        )
        self.dropout = nn.Dropout(p=classifier_dropout)
        self.classifier = nn.Linear(config.hidden_size, config.num_labels)

        # Initialize weights and apply final processing
        self.post_init()

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        token_type_ids: Optional[mindspore.Tensor] = None,
        task_type_ids: Optional[mindspore.Tensor] = None,
        position_ids: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        labels: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple[mindspore.Tensor], SequenceClassifierOutput]:
        r"""
        Args:
            labels (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
                Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
                config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
                `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        outputs = self.ernie(
            input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            task_type_ids=task_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        pooled_output = outputs[1]

        pooled_output = self.dropout(pooled_output)
        logits = self.classifier(pooled_output)

        loss = None
        if labels is not None:
            if self.config.problem_type is None:
                if self.num_labels == 1:
                    self.config.problem_type = "regression"
                elif self.num_labels > 1 and labels.dtype in (mindspore.int64, mindspore.int32):
                    self.config.problem_type = "single_label_classification"
                else:
                    self.config.problem_type = "multi_label_classification"

            if self.config.problem_type == "regression":
                if self.num_labels == 1:
                    loss = ops.mse_loss(logits.squeeze(), labels.squeeze())
                else:
                    loss = ops.mse_loss(logits, labels)
            elif self.config.problem_type == "single_label_classification":
                loss = ops.cross_entropy(logits.view(-1, self.num_labels), labels.view(-1))
            elif self.config.problem_type == "multi_label_classification":
                loss = ops.binary_cross_entropy_with_logits(logits, labels)
        if not return_dict:
            output = (logits,) + outputs[2:]
            return ((loss,) + output) if loss is not None else output

        return SequenceClassifierOutput(
            loss=loss,
            logits=logits,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForSequenceClassification.__init__(config)

Initializes an instance of the 'ErnieForSequenceClassification' class.

PARAMETER DESCRIPTION
self

The instance of the class.
config

An instance of 'Config' class containing the configuration parameters for the model.
RETURNS DESCRIPTION

None.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
2506
2507
2508
2509
2510
2511
2512
2513
2514
2515
2516
2517
2518
2519
2520
2521
2522
2523
2524
2525
2526
2527
2528
2529
2530
2531
2532
def __init__(self, config):
    """
    Initializes an instance of the 'ErnieForSequenceClassification' class.

    Args:
        self: The instance of the class.
        config: An instance of 'Config' class containing the configuration parameters for the model.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(config)
    self.num_labels = config.num_labels
    self.config = config

    self.ernie = ErnieModel(config)
    classifier_dropout = (
        config.classifier_dropout if config.classifier_dropout is not None else config.hidden_dropout_prob
    )
    self.dropout = nn.Dropout(p=classifier_dropout)
    self.classifier = nn.Linear(config.hidden_size, config.num_labels)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForSequenceClassification.forward(input_ids=None, attention_mask=None, token_type_ids=None, task_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None, labels=None, output_attentions=None, output_hidden_states=None, return_dict=None)

PARAMETER DESCRIPTION
labels

Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If config.num_labels > 1 a classification loss is computed (Cross-Entropy).

TYPE: `mindspore.Tensor` of shape `(batch_size,)`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
2534
2535
2536
2537
2538
2539
2540
2541
2542
2543
2544
2545
2546
2547
2548
2549
2550
2551
2552
2553
2554
2555
2556
2557
2558
2559
2560
2561
2562
2563
2564
2565
2566
2567
2568
2569
2570
2571
2572
2573
2574
2575
2576
2577
2578
2579
2580
2581
2582
2583
2584
2585
2586
2587
2588
2589
2590
2591
2592
2593
2594
2595
2596
2597
2598
2599
2600
2601
2602
2603
def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    token_type_ids: Optional[mindspore.Tensor] = None,
    task_type_ids: Optional[mindspore.Tensor] = None,
    position_ids: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    inputs_embeds: Optional[mindspore.Tensor] = None,
    labels: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple[mindspore.Tensor], SequenceClassifierOutput]:
    r"""
    Args:
        labels (`mindspore.Tensor` of shape `(batch_size,)`, *optional*):
            Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
            config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
            `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    outputs = self.ernie(
        input_ids,
        attention_mask=attention_mask,
        token_type_ids=token_type_ids,
        task_type_ids=task_type_ids,
        position_ids=position_ids,
        head_mask=head_mask,
        inputs_embeds=inputs_embeds,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    pooled_output = outputs[1]

    pooled_output = self.dropout(pooled_output)
    logits = self.classifier(pooled_output)

    loss = None
    if labels is not None:
        if self.config.problem_type is None:
            if self.num_labels == 1:
                self.config.problem_type = "regression"
            elif self.num_labels > 1 and labels.dtype in (mindspore.int64, mindspore.int32):
                self.config.problem_type = "single_label_classification"
            else:
                self.config.problem_type = "multi_label_classification"

        if self.config.problem_type == "regression":
            if self.num_labels == 1:
                loss = ops.mse_loss(logits.squeeze(), labels.squeeze())
            else:
                loss = ops.mse_loss(logits, labels)
        elif self.config.problem_type == "single_label_classification":
            loss = ops.cross_entropy(logits.view(-1, self.num_labels), labels.view(-1))
        elif self.config.problem_type == "multi_label_classification":
            loss = ops.binary_cross_entropy_with_logits(logits, labels)
    if not return_dict:
        output = (logits,) + outputs[2:]
        return ((loss,) + output) if loss is not None else output

    return SequenceClassifierOutput(
        loss=loss,
        logits=logits,
        hidden_states=outputs.hidden_states,
        attentions=outputs.attentions,
    )

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForTokenClassification

Bases: ErniePreTrainedModel

This class represents a token classification model based on the Ernie architecture. It is used for token-level classification tasks such as Named Entity Recognition (NER) and part-of-speech tagging. The model inherits from the ErniePreTrainedModel class and utilizes the ErnieModel for token embeddings and hidden representations. It includes methods for model initialization and forward propagation to compute token classification logits and loss.

The class's forwardor initializes the model with the provided configuration, sets the number of classification labels, and configures the ErnieModel with the specified parameters. Additionally, it sets up the dropout and classifier layers.

The forward method takes input tensors and optional arguments for token classification, and returns the token classification output. It also computes the token classification loss if labels are provided. The method supports various optional parameters for controlling the model's behavior during inference.

Note

The docstring is based on the provided information and does not include specific code signatures.

Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
2737
2738
2739
2740
2741
2742
2743
2744
2745
2746
2747
2748
2749
2750
2751
2752
2753
2754
2755
2756
2757
2758
2759
2760
2761
2762
2763
2764
2765
2766
2767
2768
2769
2770
2771
2772
2773
2774
2775
2776
2777
2778
2779
2780
2781
2782
2783
2784
2785
2786
2787
2788
2789
2790
2791
2792
2793
2794
2795
2796
2797
2798
2799
2800
2801
2802
2803
2804
2805
2806
2807
2808
2809
2810
2811
2812
2813
2814
2815
2816
2817
2818
2819
2820
2821
2822
2823
2824
2825
2826
2827
2828
2829
2830
2831
2832
2833
2834
2835
2836
2837
2838
2839
2840
2841
2842
2843
2844
2845
class ErnieForTokenClassification(ErniePreTrainedModel):

    """
    This class represents a token classification model based on the Ernie architecture.
    It is used for token-level classification tasks such as Named Entity Recognition (NER) and part-of-speech tagging.
    The model inherits from the ErniePreTrainedModel class and utilizes the ErnieModel for token embeddings and
    hidden representations.
    It includes methods for model initialization and forward propagation to compute token classification logits and loss.

    The class's forwardor initializes the model with the provided configuration, sets the number of classification
    labels, and configures the ErnieModel with the specified parameters.
    Additionally, it sets up the dropout and classifier layers.

    The forward method takes input tensors and optional arguments for token classification, and returns the
    token classification output. It also computes the token classification loss if labels are provided.
    The method supports various optional parameters for controlling the model's behavior during inference.

    Note:
        The docstring is based on the provided information and does not include specific code signatures.
    """
    # Copied from transformers.models.bert.modeling_bert.BertForTokenClassification.__init__ with Bert->Ernie,bert->ernie
    def __init__(self, config):
        """
        Initializes an instance of the ErnieForTokenClassification class.

        Args:
            self: The instance of the class.
            config (object): The configuration object containing the settings for the model.
                This object must have the following attributes:

                - num_labels (int): The number of labels for token classification.
                - classifier_dropout (float or None): The dropout rate for the classifier layer.
                If None, it defaults to the hidden dropout probability from the configuration.
                - hidden_dropout_prob (float): The dropout probability for the hidden layers.

        Returns:
            None.

        Raises:
            ValueError: If the config object is missing the num_labels attribute.
            TypeError: If the config object does not have the expected data types for the attributes.
            RuntimeError: If an error occurs during the initialization process.
        """
        super().__init__(config)
        self.num_labels = config.num_labels

        self.ernie = ErnieModel(config, add_pooling_layer=False)
        classifier_dropout = (
            config.classifier_dropout if config.classifier_dropout is not None else config.hidden_dropout_prob
        )
        self.dropout = nn.Dropout(p=classifier_dropout)
        self.classifier = nn.Linear(config.hidden_size, config.num_labels)

        # Initialize weights and apply final processing
        self.post_init()

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        token_type_ids: Optional[mindspore.Tensor] = None,
        task_type_ids: Optional[mindspore.Tensor] = None,
        position_ids: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        labels: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple[mindspore.Tensor], TokenClassifierOutput]:
        r"""
        Args:
            labels (`mindspore.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
                Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`.
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        outputs = self.ernie(
            input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            task_type_ids=task_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        sequence_output = outputs[0]

        sequence_output = self.dropout(sequence_output)
        logits = self.classifier(sequence_output)

        loss = None
        if labels is not None:
            loss = ops.cross_entropy(logits.view(-1, self.num_labels), labels.view(-1))

        if not return_dict:
            output = (logits,) + outputs[2:]
            return ((loss,) + output) if loss is not None else output

        return TokenClassifierOutput(
            loss=loss,
            logits=logits,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForTokenClassification.__init__(config)

Initializes an instance of the ErnieForTokenClassification class.

PARAMETER DESCRIPTION
self

The instance of the class.
config

The configuration object containing the settings for the model. This object must have the following attributes:

  • num_labels (int): The number of labels for token classification.
  • classifier_dropout (float or None): The dropout rate for the classifier layer. If None, it defaults to the hidden dropout probability from the configuration.
  • hidden_dropout_prob (float): The dropout probability for the hidden layers.

TYPE: object

RETURNS DESCRIPTION

None.
RAISES DESCRIPTION
ValueError

If the config object is missing the num_labels attribute.
TypeError

If the config object does not have the expected data types for the attributes.
RuntimeError

If an error occurs during the initialization process.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
2758
2759
2760
2761
2762
2763
2764
2765
2766
2767
2768
2769
2770
2771
2772
2773
2774
2775
2776
2777
2778
2779
2780
2781
2782
2783
2784
2785
2786
2787
2788
2789
2790
2791
def __init__(self, config):
    """
    Initializes an instance of the ErnieForTokenClassification class.

    Args:
        self: The instance of the class.
        config (object): The configuration object containing the settings for the model.
            This object must have the following attributes:

            - num_labels (int): The number of labels for token classification.
            - classifier_dropout (float or None): The dropout rate for the classifier layer.
            If None, it defaults to the hidden dropout probability from the configuration.
            - hidden_dropout_prob (float): The dropout probability for the hidden layers.

    Returns:
        None.

    Raises:
        ValueError: If the config object is missing the num_labels attribute.
        TypeError: If the config object does not have the expected data types for the attributes.
        RuntimeError: If an error occurs during the initialization process.
    """
    super().__init__(config)
    self.num_labels = config.num_labels

    self.ernie = ErnieModel(config, add_pooling_layer=False)
    classifier_dropout = (
        config.classifier_dropout if config.classifier_dropout is not None else config.hidden_dropout_prob
    )
    self.dropout = nn.Dropout(p=classifier_dropout)
    self.classifier = nn.Linear(config.hidden_size, config.num_labels)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.ernie.modeling_ernie.ErnieForTokenClassification.forward(input_ids=None, attention_mask=None, token_type_ids=None, task_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None, labels=None, output_attentions=None, output_hidden_states=None, return_dict=None)

PARAMETER DESCRIPTION
labels

Labels for computing the token classification loss. Indices should be in [0, ..., config.num_labels - 1].

TYPE: `mindspore.Tensor` of shape `(batch_size, sequence_length)`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
2793
2794
2795
2796
2797
2798
2799
2800
2801
2802
2803
2804
2805
2806
2807
2808
2809
2810
2811
2812
2813
2814
2815
2816
2817
2818
2819
2820
2821
2822
2823
2824
2825
2826
2827
2828
2829
2830
2831
2832
2833
2834
2835
2836
2837
2838
2839
2840
2841
2842
2843
2844
2845
def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    token_type_ids: Optional[mindspore.Tensor] = None,
    task_type_ids: Optional[mindspore.Tensor] = None,
    position_ids: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    inputs_embeds: Optional[mindspore.Tensor] = None,
    labels: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple[mindspore.Tensor], TokenClassifierOutput]:
    r"""
    Args:
        labels (`mindspore.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
            Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`.
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    outputs = self.ernie(
        input_ids,
        attention_mask=attention_mask,
        token_type_ids=token_type_ids,
        task_type_ids=task_type_ids,
        position_ids=position_ids,
        head_mask=head_mask,
        inputs_embeds=inputs_embeds,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    sequence_output = outputs[0]

    sequence_output = self.dropout(sequence_output)
    logits = self.classifier(sequence_output)

    loss = None
    if labels is not None:
        loss = ops.cross_entropy(logits.view(-1, self.num_labels), labels.view(-1))

    if not return_dict:
        output = (logits,) + outputs[2:]
        return ((loss,) + output) if loss is not None else output

    return TokenClassifierOutput(
        loss=loss,
        logits=logits,
        hidden_states=outputs.hidden_states,
        attentions=outputs.attentions,
    )

mindnlp.transformers.models.ernie.modeling_ernie.ErnieIntermediate

Bases: Module

Represents an intermediate layer for the ERNIE (Enhanced Representation through kNowledge Integration) model. This class provides methods to perform intermediate operations on input hidden states.

This class inherits from nn.Module and contains methods for initialization and forwarding the intermediate layer.

ATTRIBUTE DESCRIPTION
dense

A dense layer with the specified hidden size and intermediate size.

TYPE: Linear

intermediate_act_fn

The activation function applied to the intermediate hidden states.

TYPE: function

METHOD DESCRIPTION
__init__

Initializes the ERNIE intermediate layer with the provided configuration.
forward

Constructs the intermediate layer by applying dense and activation functions to the input hidden states.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
class ErnieIntermediate(nn.Module):

    '''
    Represents an intermediate layer for the ERNIE (Enhanced Representation through kNowledge Integration) model.
    This class provides methods to perform intermediate operations on input hidden states.

    This class inherits from nn.Module and contains methods for initialization and forwarding the intermediate layer.

    Attributes:
        dense (nn.Linear): A dense layer with the specified hidden size and intermediate size.
        intermediate_act_fn (function): The activation function applied to the intermediate hidden states.

    Methods:
        __init__: Initializes the ERNIE intermediate layer with the provided configuration.
        forward: Constructs the intermediate layer by applying dense and activation functions to the input hidden states.
    '''
    def __init__(self, config):
        """
        Initialize the ErnieIntermediate class with the provided configuration.

        Args:
            self (object): The instance of the ErnieIntermediate class.
            config (object):
                An object containing the configuration parameters.

                - hidden_size (int): The size of the hidden layer.
                - intermediate_size (int): The size of the intermediate layer.
                - hidden_act (str or function): The activation function for the hidden layer.
                If provided as a string, it should be a key in ACT2FN dictionary.

        Returns:
            None.

        Raises:
            ValueError: If the configuration parameters are invalid or missing.
            TypeError: If the provided hidden activation function is not a string or function.
        """
        super().__init__()
        self.dense = nn.Linear(config.hidden_size, config.intermediate_size)
        if isinstance(config.hidden_act, str):
            self.intermediate_act_fn = ACT2FN[config.hidden_act]
        else:
            self.intermediate_act_fn = config.hidden_act

    def forward(self, hidden_states: mindspore.Tensor) -> mindspore.Tensor:
        """
        Constructs the intermediate layer of the ERNIE model.

        Args:
            self (ErnieIntermediate): An instance of the ErnieIntermediate class.
            hidden_states (mindspore.Tensor): The input hidden states tensor of shape (batch_size, sequence_length, hidden_size).
                It represents the output from the previous layer of the ERNIE model.

        Returns:
            mindspore.Tensor: The tensor representing the intermediate hidden states of shape (batch_size, sequence_length, hidden_size).
                It is the result of applying the intermediate layer operations on the input hidden states.

        Raises:
            None.
        """
        hidden_states = self.dense(hidden_states)
        hidden_states = self.intermediate_act_fn(hidden_states)
        return hidden_states

mindnlp.transformers.models.ernie.modeling_ernie.ErnieIntermediate.__init__(config)

Initialize the ErnieIntermediate class with the provided configuration.

PARAMETER DESCRIPTION
self

The instance of the ErnieIntermediate class.

TYPE: object

config

An object containing the configuration parameters.

  • hidden_size (int): The size of the hidden layer.
  • intermediate_size (int): The size of the intermediate layer.
  • hidden_act (str or function): The activation function for the hidden layer. If provided as a string, it should be a key in ACT2FN dictionary.

TYPE: object

RETURNS DESCRIPTION

None.
RAISES DESCRIPTION
ValueError

If the configuration parameters are invalid or missing.
TypeError

If the provided hidden activation function is not a string or function.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
def __init__(self, config):
    """
    Initialize the ErnieIntermediate class with the provided configuration.

    Args:
        self (object): The instance of the ErnieIntermediate class.
        config (object):
            An object containing the configuration parameters.

            - hidden_size (int): The size of the hidden layer.
            - intermediate_size (int): The size of the intermediate layer.
            - hidden_act (str or function): The activation function for the hidden layer.
            If provided as a string, it should be a key in ACT2FN dictionary.

    Returns:
        None.

    Raises:
        ValueError: If the configuration parameters are invalid or missing.
        TypeError: If the provided hidden activation function is not a string or function.
    """
    super().__init__()
    self.dense = nn.Linear(config.hidden_size, config.intermediate_size)
    if isinstance(config.hidden_act, str):
        self.intermediate_act_fn = ACT2FN[config.hidden_act]
    else:
        self.intermediate_act_fn = config.hidden_act

mindnlp.transformers.models.ernie.modeling_ernie.ErnieIntermediate.forward(hidden_states)

Constructs the intermediate layer of the ERNIE model.

PARAMETER DESCRIPTION
self

An instance of the ErnieIntermediate class.

TYPE: ErnieIntermediate

hidden_states

The input hidden states tensor of shape (batch_size, sequence_length, hidden_size). It represents the output from the previous layer of the ERNIE model.

TYPE: Tensor

RETURNS DESCRIPTION
Tensor

mindspore.Tensor: The tensor representing the intermediate hidden states of shape (batch_size, sequence_length, hidden_size). It is the result of applying the intermediate layer operations on the input hidden states.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
def forward(self, hidden_states: mindspore.Tensor) -> mindspore.Tensor:
    """
    Constructs the intermediate layer of the ERNIE model.

    Args:
        self (ErnieIntermediate): An instance of the ErnieIntermediate class.
        hidden_states (mindspore.Tensor): The input hidden states tensor of shape (batch_size, sequence_length, hidden_size).
            It represents the output from the previous layer of the ERNIE model.

    Returns:
        mindspore.Tensor: The tensor representing the intermediate hidden states of shape (batch_size, sequence_length, hidden_size).
            It is the result of applying the intermediate layer operations on the input hidden states.

    Raises:
        None.
    """
    hidden_states = self.dense(hidden_states)
    hidden_states = self.intermediate_act_fn(hidden_states)
    return hidden_states

mindnlp.transformers.models.ernie.modeling_ernie.ErnieLMPredictionHead

Bases: Module

Represents a prediction head for ERNIE Language Model that performs decoding and transformation operations on hidden states.

This class inherits from nn.Module and provides methods for initializing the prediction head and forwarding predictions based on the input hidden states.

ATTRIBUTE DESCRIPTION
transform

ErniePredictionHeadTransform object for transforming hidden states.
decoder

nn.Linear object for decoding hidden states into output predictions.
bias

Parameter object for bias initialization.
METHOD DESCRIPTION
__init__

Initializes the prediction head with the given configuration.
forward

Constructs predictions based on the input hidden states by applying transformation and decoding operations.
Example 
>>> config = get_config()
>>> prediction_head = ErnieLMPredictionHead(config)
>>> hidden_states = get_hidden_states()
>>> predictions = prediction_head.forward(hidden_states)
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
class ErnieLMPredictionHead(nn.Module):

    """
    Represents a prediction head for ERNIE Language Model that performs decoding and transformation operations
    on hidden states.

    This class inherits from nn.Module and provides methods for initializing the prediction head and forwarding
    predictions based on the input hidden states.

    Attributes:
        transform: ErniePredictionHeadTransform object for transforming hidden states.
        decoder: nn.Linear object for decoding hidden states into output predictions.
        bias: Parameter object for bias initialization.

    Methods:
        __init__: Initializes the prediction head with the given configuration.
        forward: Constructs predictions based on the input hidden states by applying
            transformation and decoding operations.

    Example:
        ```python
        >>> config = get_config()
        >>> prediction_head = ErnieLMPredictionHead(config)
        >>> hidden_states = get_hidden_states()
        >>> predictions = prediction_head.forward(hidden_states)
        ```
    """
    def __init__(self, config):
        """
        Initializes an instance of the ErnieLMPredictionHead class.

        Args:
            self: The instance of the ErnieLMPredictionHead class.
            config: An object that holds configuration settings for the ErnieLMPredictionHead.
                It is expected to contain properties like hidden_size, vocab_size, and any other relevant
                configuration parameters.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.transform = ErniePredictionHeadTransform(config)

        # The output weights are the same as the input embeddings, but there is
        # an output-only bias for each token.
        self.decoder = nn.Linear(config.hidden_size, config.vocab_size, bias=False)

        self.bias = Parameter(ops.zeros(config.vocab_size), 'bias')

        # Need a link between the two variables so that the bias is correctly resized with `resize_token_embeddings`
        self.decoder.bias = self.bias

    def forward(self, hidden_states):
        """
        This method 'forward' is part of the class 'ErnieLMPredictionHead' and is responsible for forwarding
        the hidden states using transformation and decoding.

        Args:
            self: Represents the instance of the class. It is implicitly passed and does not need to be provided as
                an argument.

            hidden_states (Tensor): The input hidden states to be processed. It is expected to be a tensor containing
                the initial hidden states.

        Returns:
            Tensor: The processed hidden states after transformation and decoding.

        Raises:
            TypeError: If the input 'hidden_states' is not of type Tensor.
            ValueError: If the input 'hidden_states' is empty or invalid for transformation and decoding.
            RuntimeError: If there is an issue during the transformation or decoding process.
        """
        hidden_states = self.transform(hidden_states)
        hidden_states = self.decoder(hidden_states)
        return hidden_states

mindnlp.transformers.models.ernie.modeling_ernie.ErnieLMPredictionHead.__init__(config)

Initializes an instance of the ErnieLMPredictionHead class.

PARAMETER DESCRIPTION
self

The instance of the ErnieLMPredictionHead class.
config

An object that holds configuration settings for the ErnieLMPredictionHead. It is expected to contain properties like hidden_size, vocab_size, and any other relevant configuration parameters.
RETURNS DESCRIPTION

None.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
def __init__(self, config):
    """
    Initializes an instance of the ErnieLMPredictionHead class.

    Args:
        self: The instance of the ErnieLMPredictionHead class.
        config: An object that holds configuration settings for the ErnieLMPredictionHead.
            It is expected to contain properties like hidden_size, vocab_size, and any other relevant
            configuration parameters.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.transform = ErniePredictionHeadTransform(config)

    # The output weights are the same as the input embeddings, but there is
    # an output-only bias for each token.
    self.decoder = nn.Linear(config.hidden_size, config.vocab_size, bias=False)

    self.bias = Parameter(ops.zeros(config.vocab_size), 'bias')

    # Need a link between the two variables so that the bias is correctly resized with `resize_token_embeddings`
    self.decoder.bias = self.bias

mindnlp.transformers.models.ernie.modeling_ernie.ErnieLMPredictionHead.forward(hidden_states)

This method 'forward' is part of the class 'ErnieLMPredictionHead' and is responsible for forwarding the hidden states using transformation and decoding.

PARAMETER DESCRIPTION
self

Represents the instance of the class. It is implicitly passed and does not need to be provided as an argument.
hidden_states

The input hidden states to be processed. It is expected to be a tensor containing the initial hidden states.

TYPE: Tensor

RETURNS DESCRIPTION
Tensor

The processed hidden states after transformation and decoding.
RAISES DESCRIPTION
TypeError

If the input 'hidden_states' is not of type Tensor.
ValueError

If the input 'hidden_states' is empty or invalid for transformation and decoding.
RuntimeError

If there is an issue during the transformation or decoding process.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
def forward(self, hidden_states):
    """
    This method 'forward' is part of the class 'ErnieLMPredictionHead' and is responsible for forwarding
    the hidden states using transformation and decoding.

    Args:
        self: Represents the instance of the class. It is implicitly passed and does not need to be provided as
            an argument.

        hidden_states (Tensor): The input hidden states to be processed. It is expected to be a tensor containing
            the initial hidden states.

    Returns:
        Tensor: The processed hidden states after transformation and decoding.

    Raises:
        TypeError: If the input 'hidden_states' is not of type Tensor.
        ValueError: If the input 'hidden_states' is empty or invalid for transformation and decoding.
        RuntimeError: If there is an issue during the transformation or decoding process.
    """
    hidden_states = self.transform(hidden_states)
    hidden_states = self.decoder(hidden_states)
    return hidden_states

mindnlp.transformers.models.ernie.modeling_ernie.ErnieLayer

Bases: Module

ErnieLayer is a class representing a layer in the Ernie model. This class inherits from nn.Module and contains methods for initializing the layer and forwarding the layer's feed forward chunk.

ATTRIBUTE DESCRIPTION
chunk_size_feed_forward

The chunk size for the feed forward operation.

TYPE: int

seq_len_dim

The dimension of the sequence length.

TYPE: int

attention

The attention mechanism used in the layer.

TYPE: ErnieAttention

is_decoder

Indicates whether the layer is a decoder model.

TYPE: bool

add_cross_attention

Indicates whether cross attention is added to the layer.

TYPE: bool

crossattention

The cross attention mechanism used in the layer.

TYPE: ErnieAttention

intermediate

The intermediate layer in the Ernie model.

TYPE: ErnieIntermediate

output

The output layer in the Ernie model.

TYPE: ErnieOutput

METHOD DESCRIPTION
__init__

Initializes the ErnieLayer with the provided configuration.
forward

Constructs the layer using the given input tensors and parameters.
feed_forward_chunk

Executes the feed forward operation on the attention output.
RAISES DESCRIPTION
ValueError

If the layer is not instantiated with cross-attention layers when encoder_hidden_states are passed.
RETURNS DESCRIPTION
Tuple

Outputs of the layer's forward method, including the layer output and present key value if the layer is a decoder model.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
class ErnieLayer(nn.Module):

    """ErnieLayer is a class representing a layer in the Ernie model.
    This class inherits from nn.Module and contains methods for initializing the layer and forwarding
    the layer's feed forward chunk.

    Attributes:
        chunk_size_feed_forward (int): The chunk size for the feed forward operation.
        seq_len_dim (int): The dimension of the sequence length.
        attention (ErnieAttention): The attention mechanism used in the layer.
        is_decoder (bool): Indicates whether the layer is a decoder model.
        add_cross_attention (bool): Indicates whether cross attention is added to the layer.
        crossattention (ErnieAttention): The cross attention mechanism used in the layer.
        intermediate (ErnieIntermediate): The intermediate layer in the Ernie model.
        output (ErnieOutput): The output layer in the Ernie model.

    Methods:
        __init__: Initializes the ErnieLayer with the provided configuration.
        forward: Constructs the layer using the given input tensors and parameters.
        feed_forward_chunk(attention_output): Executes the feed forward operation on the attention output.

    Raises:
        ValueError: If the layer is not instantiated with cross-attention layers when `encoder_hidden_states` are passed.

    Returns:
        Tuple: Outputs of the layer's forward method, including the layer output and present key value if the layer is a decoder model.
    """
    def __init__(self, config):
        """
        Initializes an instance of the ErnieLayer class.

        Args:
            self: The instance of the ErnieLayer class.
            config:
                A configuration object containing various settings for the ErnieLayer.

                - Type: object
                - Purpose: Configures the behavior of the ErnieLayer.
                - Restrictions: Must contain the following attributes:

                    - chunk_size_feed_forward: Chunk size for feed-forward operations.
                    - is_decoder: Boolean indicating whether the layer is used as a decoder model.
                    - add_cross_attention: Boolean indicating whether cross attention is added.
                    - position_embedding_type: Optional parameter specifying the position embedding type for cross
                    attention.

        Returns:
            None.

        Raises:
            ValueError: Raised if cross attention is added but the ErnieLayer is not used as a decoder model.
        """
        super().__init__()
        self.chunk_size_feed_forward = config.chunk_size_feed_forward
        self.seq_len_dim = 1
        self.attention = ErnieAttention(config)
        self.is_decoder = config.is_decoder
        self.add_cross_attention = config.add_cross_attention
        if self.add_cross_attention:
            if not self.is_decoder:
                raise ValueError(f"{self} should be used as a decoder model if cross attention is added")
            self.crossattention = ErnieAttention(config, position_embedding_type="absolute")
        self.intermediate = ErnieIntermediate(config)
        self.output = ErnieOutput(config)

    def forward(
        self,
        hidden_states: mindspore.Tensor,
        attention_mask: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        encoder_hidden_states: Optional[mindspore.Tensor] = None,
        encoder_attention_mask: Optional[mindspore.Tensor] = None,
        past_key_value: Optional[Tuple[Tuple[mindspore.Tensor]]] = None,
        output_attentions: Optional[bool] = False,
    ) -> Tuple[mindspore.Tensor]:
        """
        Constructs an ERNIE (Enhanced Representation through kNowledge Integration) layer.

        Args:
            self: The object itself.
            hidden_states (mindspore.Tensor): The input hidden states for the layer.
            attention_mask (Optional[mindspore.Tensor]): Mask for the attention mechanism. Defaults to None.
            head_mask (Optional[mindspore.Tensor]): Mask for the attention heads. Defaults to None.
            encoder_hidden_states (Optional[mindspore.Tensor]): Hidden states from the encoder layer. Defaults to None.
            encoder_attention_mask (Optional[mindspore.Tensor]): Mask for the encoder attention mechanism.
                Defaults to None.
            past_key_value (Optional[Tuple[Tuple[mindspore.Tensor]]]): Cached key and value tensors for fast inference.
                Defaults to None.
            output_attentions (Optional[bool]): Whether to return attentions weights. Defaults to False.

        Returns:
            Tuple[mindspore.Tensor]: A tuple containing the layer output tensor.

        Raises:
            ValueError: If `encoder_hidden_states` are passed, and cross-attention layers are not instantiated
                by setting `config.add_cross_attention=True`.
        """
        # decoder uni-directional self-attention cached key/values tuple is at positions 1,2
        self_attn_past_key_value = past_key_value[:2] if past_key_value is not None else None
        self_attention_outputs = self.attention(
            hidden_states,
            attention_mask,
            head_mask,
            output_attentions=output_attentions,
            past_key_value=self_attn_past_key_value,
        )
        attention_output = self_attention_outputs[0]

        # if decoder, the last output is tuple of self-attn cache
        if self.is_decoder:
            outputs = self_attention_outputs[1:-1]
            present_key_value = self_attention_outputs[-1]
        else:
            outputs = self_attention_outputs[1:]  # add self attentions if we output attention weights

        cross_attn_present_key_value = None
        if self.is_decoder and encoder_hidden_states is not None:
            if not hasattr(self, "crossattention"):
                raise ValueError(
                    f"If `encoder_hidden_states` are passed, {self} has to be instantiated with cross-attention layers"
                    " by setting `config.add_cross_attention=True`"
                )

            # cross_attn cached key/values tuple is at positions 3,4 of past_key_value tuple
            cross_attn_past_key_value = past_key_value[-2:] if past_key_value is not None else None
            cross_attention_outputs = self.crossattention(
                attention_output,
                attention_mask,
                head_mask,
                encoder_hidden_states,
                encoder_attention_mask,
                cross_attn_past_key_value,
                output_attentions,
            )
            attention_output = cross_attention_outputs[0]
            outputs = outputs + cross_attention_outputs[1:-1]  # add cross attentions if we output attention weights

            # add cross-attn cache to positions 3,4 of present_key_value tuple
            cross_attn_present_key_value = cross_attention_outputs[-1]
            present_key_value = present_key_value + cross_attn_present_key_value

        layer_output = apply_chunking_to_forward(
            self.feed_forward_chunk, self.chunk_size_feed_forward, self.seq_len_dim, attention_output
        )
        outputs = (layer_output,) + outputs

        # if decoder, return the attn key/values as the last output
        if self.is_decoder:
            outputs = outputs + (present_key_value,)

        return outputs

    def feed_forward_chunk(self, attention_output):
        """
        This method calculates the feed-forward output for a chunk in the ErnieLayer.

        Args:
            self (object): The instance of the ErnieLayer class.
            attention_output (object): The attention output from the previous layer,
                expected to be a tensor representing the attention scores.

        Returns:
            None: This method does not return any value explicitly but updates the layer_output attribute of
                the ErnieLayer instance.

        Raises:
            None.
        """
        intermediate_output = self.intermediate(attention_output)
        layer_output = self.output(intermediate_output, attention_output)
        return layer_output

mindnlp.transformers.models.ernie.modeling_ernie.ErnieLayer.__init__(config)

Initializes an instance of the ErnieLayer class.

PARAMETER DESCRIPTION
self

The instance of the ErnieLayer class.
config

A configuration object containing various settings for the ErnieLayer.

  • Type: object
  • Purpose: Configures the behavior of the ErnieLayer.

  • Restrictions: Must contain the following attributes:

    • chunk_size_feed_forward: Chunk size for feed-forward operations.
    • is_decoder: Boolean indicating whether the layer is used as a decoder model.
    • add_cross_attention: Boolean indicating whether cross attention is added.
    • position_embedding_type: Optional parameter specifying the position embedding type for cross attention.
RETURNS DESCRIPTION

None.
RAISES DESCRIPTION
ValueError

Raised if cross attention is added but the ErnieLayer is not used as a decoder model.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
def __init__(self, config):
    """
    Initializes an instance of the ErnieLayer class.

    Args:
        self: The instance of the ErnieLayer class.
        config:
            A configuration object containing various settings for the ErnieLayer.

            - Type: object
            - Purpose: Configures the behavior of the ErnieLayer.
            - Restrictions: Must contain the following attributes:

                - chunk_size_feed_forward: Chunk size for feed-forward operations.
                - is_decoder: Boolean indicating whether the layer is used as a decoder model.
                - add_cross_attention: Boolean indicating whether cross attention is added.
                - position_embedding_type: Optional parameter specifying the position embedding type for cross
                attention.

    Returns:
        None.

    Raises:
        ValueError: Raised if cross attention is added but the ErnieLayer is not used as a decoder model.
    """
    super().__init__()
    self.chunk_size_feed_forward = config.chunk_size_feed_forward
    self.seq_len_dim = 1
    self.attention = ErnieAttention(config)
    self.is_decoder = config.is_decoder
    self.add_cross_attention = config.add_cross_attention
    if self.add_cross_attention:
        if not self.is_decoder:
            raise ValueError(f"{self} should be used as a decoder model if cross attention is added")
        self.crossattention = ErnieAttention(config, position_embedding_type="absolute")
    self.intermediate = ErnieIntermediate(config)
    self.output = ErnieOutput(config)

mindnlp.transformers.models.ernie.modeling_ernie.ErnieLayer.feed_forward_chunk(attention_output)

This method calculates the feed-forward output for a chunk in the ErnieLayer.

PARAMETER DESCRIPTION
self

The instance of the ErnieLayer class.

TYPE: object

attention_output

The attention output from the previous layer, expected to be a tensor representing the attention scores.

TYPE: object

RETURNS DESCRIPTION
None

This method does not return any value explicitly but updates the layer_output attribute of the ErnieLayer instance.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
def feed_forward_chunk(self, attention_output):
    """
    This method calculates the feed-forward output for a chunk in the ErnieLayer.

    Args:
        self (object): The instance of the ErnieLayer class.
        attention_output (object): The attention output from the previous layer,
            expected to be a tensor representing the attention scores.

    Returns:
        None: This method does not return any value explicitly but updates the layer_output attribute of
            the ErnieLayer instance.

    Raises:
        None.
    """
    intermediate_output = self.intermediate(attention_output)
    layer_output = self.output(intermediate_output, attention_output)
    return layer_output

mindnlp.transformers.models.ernie.modeling_ernie.ErnieLayer.forward(hidden_states, attention_mask=None, head_mask=None, encoder_hidden_states=None, encoder_attention_mask=None, past_key_value=None, output_attentions=False)

Constructs an ERNIE (Enhanced Representation through kNowledge Integration) layer.

PARAMETER DESCRIPTION
self

The object itself.
hidden_states

The input hidden states for the layer.

TYPE: Tensor

attention_mask

Mask for the attention mechanism. Defaults to None.

TYPE: Optional[Tensor] DEFAULT: None

head_mask

Mask for the attention heads. Defaults to None.

TYPE: Optional[Tensor] DEFAULT: None

encoder_hidden_states

Hidden states from the encoder layer. Defaults to None.

TYPE: Optional[Tensor] DEFAULT: None

encoder_attention_mask

Mask for the encoder attention mechanism. Defaults to None.

TYPE: Optional[Tensor] DEFAULT: None

past_key_value

Cached key and value tensors for fast inference. Defaults to None.

TYPE: Optional[Tuple[Tuple[Tensor]]] DEFAULT: None

output_attentions

Whether to return attentions weights. Defaults to False.

TYPE: Optional[bool] DEFAULT: False

RETURNS DESCRIPTION
Tuple[Tensor]

Tuple[mindspore.Tensor]: A tuple containing the layer output tensor.
RAISES DESCRIPTION
ValueError

If encoder_hidden_states are passed, and cross-attention layers are not instantiated by setting config.add_cross_attention=True.
Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
def forward(
    self,
    hidden_states: mindspore.Tensor,
    attention_mask: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    encoder_hidden_states: Optional[mindspore.Tensor] = None,
    encoder_attention_mask: Optional[mindspore.Tensor] = None,
    past_key_value: Optional[Tuple[Tuple[mindspore.Tensor]]] = None,
    output_attentions: Optional[bool] = False,
) -> Tuple[mindspore.Tensor]:
    """
    Constructs an ERNIE (Enhanced Representation through kNowledge Integration) layer.

    Args:
        self: The object itself.
        hidden_states (mindspore.Tensor): The input hidden states for the layer.
        attention_mask (Optional[mindspore.Tensor]): Mask for the attention mechanism. Defaults to None.
        head_mask (Optional[mindspore.Tensor]): Mask for the attention heads. Defaults to None.
        encoder_hidden_states (Optional[mindspore.Tensor]): Hidden states from the encoder layer. Defaults to None.
        encoder_attention_mask (Optional[mindspore.Tensor]): Mask for the encoder attention mechanism.
            Defaults to None.
        past_key_value (Optional[Tuple[Tuple[mindspore.Tensor]]]): Cached key and value tensors for fast inference.
            Defaults to None.
        output_attentions (Optional[bool]): Whether to return attentions weights. Defaults to False.

    Returns:
        Tuple[mindspore.Tensor]: A tuple containing the layer output tensor.

    Raises:
        ValueError: If `encoder_hidden_states` are passed, and cross-attention layers are not instantiated
            by setting `config.add_cross_attention=True`.
    """
    # decoder uni-directional self-attention cached key/values tuple is at positions 1,2
    self_attn_past_key_value = past_key_value[:2] if past_key_value is not None else None
    self_attention_outputs = self.attention(
        hidden_states,
        attention_mask,
        head_mask,
        output_attentions=output_attentions,
        past_key_value=self_attn_past_key_value,
    )
    attention_output = self_attention_outputs[0]

    # if decoder, the last output is tuple of self-attn cache
    if self.is_decoder:
        outputs = self_attention_outputs[1:-1]
        present_key_value = self_attention_outputs[-1]
    else:
        outputs = self_attention_outputs[1:]  # add self attentions if we output attention weights

    cross_attn_present_key_value = None
    if self.is_decoder and encoder_hidden_states is not None:
        if not hasattr(self, "crossattention"):
            raise ValueError(
                f"If `encoder_hidden_states` are passed, {self} has to be instantiated with cross-attention layers"
                " by setting `config.add_cross_attention=True`"
            )

        # cross_attn cached key/values tuple is at positions 3,4 of past_key_value tuple
        cross_attn_past_key_value = past_key_value[-2:] if past_key_value is not None else None
        cross_attention_outputs = self.crossattention(
            attention_output,
            attention_mask,
            head_mask,
            encoder_hidden_states,
            encoder_attention_mask,
            cross_attn_past_key_value,
            output_attentions,
        )
        attention_output = cross_attention_outputs[0]
        outputs = outputs + cross_attention_outputs[1:-1]  # add cross attentions if we output attention weights

        # add cross-attn cache to positions 3,4 of present_key_value tuple
        cross_attn_present_key_value = cross_attention_outputs[-1]
        present_key_value = present_key_value + cross_attn_present_key_value

    layer_output = apply_chunking_to_forward(
        self.feed_forward_chunk, self.chunk_size_feed_forward, self.seq_len_dim, attention_output
    )
    outputs = (layer_output,) + outputs

    # if decoder, return the attn key/values as the last output
    if self.is_decoder:
        outputs = outputs + (present_key_value,)

    return outputs

mindnlp.transformers.models.ernie.modeling_ernie.ErnieModel

Bases: ErniePreTrainedModel

The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in Attention is all you need by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin.

To behave as an decoder the model needs to be initialized with the is_decoder argument of the configuration set to True. To be used in a Seq2Seq model, the model needs to initialized with both is_decoder argument and add_cross_attention set to True; an encoder_hidden_states is then expected as an input to the forward pass.

Source code in mindnlp/transformers/models/ernie/modeling_ernie.py 
1490
1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
1505
1506
1507
1508
1509
1510
1511
1512
1513
1514
1515
1516
1517
1518
1519
1520
1521
1522
1523
1524
1525
1526
1527
1528
1529
1530
1531
1532
1533
1534
1535
1536
1537
1538
1539
1540
1541
1542
1543
1544
1545
1546
1547
1548
1549
1550
1551
1552
1553
1554
1555
1556
1557
1558
1559
1560
1561
1562
1563
1564
1565
1566
1567
1568
1569
1570
1571
1572
1573
1574
1575
1576
1577
1578
1579
1580
1581
1582
1583
1584
1585
1586
1587
1588
1589
1590
1591
1592
1593
1594
1595
1596
1597
1598
1599
1600
1601
1602
1603
1604
1605
1606
1607
1608
1609
1610
1611
1612
1613
1614
1615
1616
1617
1618
1619
1620
1621
1622
1623
1624
1625
1626
1627
1628
1629
1630
1631
1632
1633
1634
1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
1654
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
1670
1671
1672
1673
1674
1675
1676
1677
1678
1679
1680
1681
1682
1683
1684
1685
1686
1687
1688
1689
1690
1691
1692
1693
1694
1695
1696
1697
1698
1699
1700
class ErnieModel(ErniePreTrainedModel):
    """
    The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of
    cross-attention is added between the self-attention layers, following the architecture described in [Attention is
    all you need](https://arxiv.org/abs/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit,
    Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin.

    To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set
    to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and
    `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass.
    """
    # Copied from transformers.models.bert.modeling_bert.BertModel.__init__ with Bert->Ernie
    def __init__(self, config, add_pooling_layer=True):
        """
        Initializes an instance of the ErnieModel class.

        Args:
            self: The instance of the class.
            config (object): The configuration object containing settings for the Ernie model.
            add_pooling_layer (bool): A flag indicating whether to add a pooling layer to the model. Default is True.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(config)
        self.config = config

        self.embeddings = ErnieEmbeddings(config)
        self.encoder = ErnieEncoder(config)

        self.pooler = ErniePooler(config) if add_pooling_layer else None

        # Initialize weights and apply final processing
        self.post_init()

    # Copied from transformers.models.bert.modeling_bert.BertModel.get_input_embeddings
    def get_input_embeddings(self):
        """
        This method returns the input embeddings for the ErnieModel.

        Args:
            self: The instance of the ErnieModel class.

        Returns:
            The input embeddings for the ErnieModel.

        Raises:
            This method does not raise any exceptions.
        """
        return self.embeddings.word_embeddings

    # Copied from transformers.models.bert.modeling_bert.BertModel.set_input_embeddings
    def set_input_embeddings(self, value):
        """
        Sets the input embeddings for the ErnieModel.

        Args:
            self (ErnieModel): The instance of the ErnieModel class.
            value: The input embeddings to be set for the ErnieModel.
                It should be of type that is compatible with the embeddings.word_embeddings attribute.

        Returns:
            None.

        Raises:
            None.
        """
        self.embeddings.word_embeddings = value

    # Copied from transformers.models.bert.modeling_bert.BertModel._prune_heads
    def _prune_heads(self, heads_to_prune):
        """
        Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base
        class PreTrainedModel
        """
        for layer, heads in heads_to_prune.items():
            self.encoder.layer[layer].attention.prune_heads(heads)

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        token_type_ids: Optional[mindspore.Tensor] = None,
        task_type_ids: Optional[mindspore.Tensor] = None,
        position_ids: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        encoder_hidden_states: Optional[mindspore.Tensor] = None,
        encoder_attention_mask: Optional[mindspore.Tensor] = None,
        past_key_values: Optional[List[mindspore.Tensor]] = None,
        use_cache: Optional[bool] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple[mindspore.Tensor], BaseModelOutputWithPoolingAndCrossAttentions]:
        r"""
        encoder_hidden_states  (`mindspore.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*):
            Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if
            the model is configured as a decoder.
        encoder_attention_mask (`mindspore.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
            Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in
            the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`:

            - 1 for tokens that are **not masked**,
            - 0 for tokens that are **masked**.
        past_key_values (`tuple(tuple(mindspore.Tensor))` of length `config.n_layers` with each tuple having 4 tensors
            of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`):
            Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding.
            If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that
            don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all
            `decoder_input_ids` of shape `(batch_size, sequence_length)`.
        use_cache (`bool`, *optional*):
            If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see
            `past_key_values`).
        """
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        if self.config.is_decoder:
            use_cache = use_cache if use_cache is not None else self.config.use_cache
        else:
            use_cache = False

        if input_ids is not None and inputs_embeds is not None:
            raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
        if input_ids is not None:
            self.warn_if_padding_and_no_attention_mask(input_ids, attention_mask)
            input_shape = input_ids.shape
        elif inputs_embeds is not None:
            input_shape = inputs_embeds.shape[:-1]
        else:
            raise ValueError("You have to specify either input_ids or inputs_embeds")

        batch_size, seq_length = input_shape

        # past_key_values_length
        past_key_values_length = past_key_values[0][0].shape[2] if past_key_values is not None else 0

        if attention_mask is None:
            attention_mask = ops.ones(((batch_size, seq_length + past_key_values_length)))

        if token_type_ids is None:
            if hasattr(self.embeddings, "token_type_ids"):
                buffered_token_type_ids = self.embeddings.token_type_ids[:, :seq_length]
                buffered_token_type_ids_expanded = buffered_token_type_ids.broadcast_to((batch_size, seq_length))
                token_type_ids = buffered_token_type_ids_expanded
            else:
                token_type_ids = ops.zeros(input_shape, dtype=mindspore.int64)

        # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length]
        # ourselves in which case we just need to make it broadcastable to all heads.
        extended_attention_mask: mindspore.Tensor = self.get_extended_attention_mask(attention_mask, input_shape)

        # If a 2D or 3D attention mask is provided for the cross-attention
        # we need to make broadcastable to [batch_size, num_heads, seq_length, seq_length]
        if self.config.is_decoder and encoder_hidden_states is not None:
            encoder_batch_size, encoder_sequence_length, _ = encoder_hidden_states.shape
            encoder_hidden_shape = (encoder_batch_size, encoder_sequence_length)
            if encoder_attention_mask is None:
                encoder_attention_mask = ops.ones(encoder_hidden_shape)
            encoder_extended_attention_mask = self.invert_attention_mask(encoder_attention_mask)
        else:
            encoder_extended_attention_mask = None

        # Prepare head mask if needed
        # 1.0 in head_mask indicate we keep the head
        # attention_probs has shape bsz x n_heads x N x N
        # input head_mask has shape [num_heads] or [num_hidden_layers x num_heads]
        # and head_mask is converted to shape [num_hidden_layers x batch x num_heads x seq_length x seq_length]
        head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)

        embedding_output = self.embeddings(
            input_ids=input_ids,
            position_ids=position_ids,
            token_type_ids=token_type_ids,
            task_type_ids=task_type_ids,
            inputs_embeds=inputs_embeds,
            past_key_values_length=past_key_values_length,
        )
        encoder_outputs = self.encoder(
            embedding_output,
            attention_mask=extended_attention_mask,
            head_mask=head_mask,
            encoder_hidden_states=encoder_hidden_states,
            encoder_attention_mask=encoder_extended_attention_mask,
            past_key_values=past_key_values,
            use_cache=use_cache,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )
        sequence_output = encoder_outputs[0]
        pooled_output = self.pooler(sequence_output) if self.pooler is not None else None

        if not return_dict:
            return (sequence_output, pooled_output) + encoder_outputs[1:]

        return BaseModelOutputWithPoolingAndCrossAttentions(
            last_hidden_state=sequence_output,
            pooler_output=pooled_output,
            past_key_values=encoder_outputs.past_key_values,
            hidden_states=encoder_outputs.hidden_states,
            attentions=encoder_outputs.attentions,
            cross_attentions=encoder_outputs.cross_attentions,
        )

mindnlp.transformers.models.ernie.modeling_ernie.ErnieModel.__init__(config, add_pooling_layer=True)

Initializes an instance of the ErnieModel class.

PARAMETER DESCRIPTION
self

The instance of the class.
config

The configuration object containing settings for the Ernie model.

TYPE: object

add_pooling_layer

A flag indicating whether to add a pooling layer to the model. Default is True.

TYPE: bool DEFAULT: True

RETURNS DESCRIPTION

None.
Source code in mindnlp/transformers/models/ernie/mode