bartpho

mindnlp.transformers.models.bartpho.tokenization_bartpho.BartphoTokenizer

Bases: PreTrainedTokenizer

Adapted from [XLMRobertaTokenizer]. Based on SentencePiece.

This tokenizer inherits from [PreTrainedTokenizer] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

PARAMETER	DESCRIPTION
vocab_file	Path to the vocabulary file. This vocabulary is the pre-trained SentencePiece model available from the multilingual XLM-RoBERTa, also used in mBART, consisting of 250K types. TYPE: `str`
monolingual_vocab_file	Path to the monolingual vocabulary file. This monolingual vocabulary consists of Vietnamese-specialized types extracted from the multilingual vocabulary vocab_file of 250K types. TYPE: `str`
bos_token	The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the cls_token. TYPE: `str`, *optional*, defaults to `"<s>"` DEFAULT: '<s>'
eos_token	The end of sequence token. When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the sep_token. TYPE: `str`, *optional*, defaults to `"</s>"` DEFAULT: '</s>'
sep_token	The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. TYPE: `str`, *optional*, defaults to `"</s>"` DEFAULT: '</s>'
cls_token	The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens. TYPE: `str`, *optional*, defaults to `"<s>"` DEFAULT: '<s>'
unk_token	The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. TYPE: `str`, *optional*, defaults to `"<unk>"` DEFAULT: '<unk>'
pad_token	The token used for padding, for example when batching sequences of different lengths. TYPE: `str`, *optional*, defaults to `"<pad>"` DEFAULT: '<pad>'
mask_token	The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict. TYPE: `str`, *optional*, defaults to `"<mask>"` DEFAULT: '<mask>'
sp_model_kwargs	Will be passed to the SentencePieceProcessor.__init__() method. The Python wrapper for SentencePiece can be used, among other things, to set: enable_sampling: Enable subword regularization. nbest_size: Sampling parameters for unigram. Invalid for BPE-Dropout. nbest_size = {0,1}: No sampling is performed. nbest_size > 1: samples from the nbest_size results. nbest_size < 0: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. alpha: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. TYPE: `dict`, *optional* DEFAULT: None

ATTRIBUTE	DESCRIPTION
sp_model	The SentencePiece processor that is used for every conversion (string, tokens and IDs). TYPE: `SentencePieceProcessor`

Source code in mindnlp/transformers/models/bartpho/tokenization_bartpho.py

class BartphoTokenizer(PreTrainedTokenizer):
    """
    Adapted from [`XLMRobertaTokenizer`]. Based on [SentencePiece](https://github.com/google/sentencepiece).

    This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
    this superclass for more information regarding those methods.

    Args:
        vocab_file (`str`):
            Path to the vocabulary file. This vocabulary is the pre-trained SentencePiece model available from the
            multilingual XLM-RoBERTa, also used in mBART, consisting of 250K types.
        monolingual_vocab_file (`str`):
            Path to the monolingual vocabulary file. This monolingual vocabulary consists of Vietnamese-specialized
            types extracted from the multilingual vocabulary vocab_file of 250K types.
        bos_token (`str`, *optional*, defaults to `"<s>"`):
            The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.

            <Tip>

            When building a sequence using special tokens, this is not the token that is used for the beginning of
            sequence. The token used is the `cls_token`.

            </Tip>

        eos_token (`str`, *optional*, defaults to `"</s>"`):
            The end of sequence token.

            <Tip>

            When building a sequence using special tokens, this is not the token that is used for the end of sequence.
            The token used is the `sep_token`.

            </Tip>

        sep_token (`str`, *optional*, defaults to `"</s>"`):
            The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
            sequence classification or for a text and a question for question answering. It is also used as the last
            token of a sequence built with special tokens.
        cls_token (`str`, *optional*, defaults to `"<s>"`):
            The classifier token which is used when doing sequence classification (classification of the whole sequence
            instead of per-token classification). It is the first token of the sequence when built with special tokens.
        unk_token (`str`, *optional*, defaults to `"<unk>"`):
            The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
            token instead.
        pad_token (`str`, *optional*, defaults to `"<pad>"`):
            The token used for padding, for example when batching sequences of different lengths.
        mask_token (`str`, *optional*, defaults to `"<mask>"`):
            The token used for masking values. This is the token used when training this model with masked language
            modeling. This is the token which the model will try to predict.
        sp_model_kwargs (`dict`, *optional*):
            Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for
            SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things,
            to set:

            - `enable_sampling`: Enable subword regularization.
            - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.
            - `nbest_size = {0,1}`: No sampling is performed.
            - `nbest_size > 1`: samples from the nbest_size results.
            - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
            using forward-filtering-and-backward-sampling algorithm.
            - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
            BPE-dropout.

    Attributes:
        sp_model (`SentencePieceProcessor`):
            The *SentencePiece* processor that is used for every conversion (string, tokens and IDs).
    """
    vocab_files_names = VOCAB_FILES_NAMES
    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
    model_input_names = ["input_ids", "attention_mask"]

    def __init__(
        self,
        vocab_file,
        monolingual_vocab_file,
        bos_token="<s>",
        eos_token="</s>",
        sep_token="</s>",
        cls_token="<s>",
        unk_token="<unk>",
        pad_token="<pad>",
        mask_token="<mask>",
        sp_model_kwargs: Optional[Dict[str, Any]] = None,
        **kwargs,
    ) -> None:
        """
        Initializes a new instance of the BartphoTokenizer class.

        Args:
            self: The instance of the class.
            vocab_file (str): The path to the vocabulary file.
            monolingual_vocab_file (str): The path to the monolingual vocabulary file.
            bos_token (str, optional): The beginning of sentence token. Defaults to '<s>'.
            eos_token (str, optional): The end of sentence token. Defaults to '</s>'.
            sep_token (str, optional): The separator token. Defaults to '</s>'.
            cls_token (str, optional): The classification token. Defaults to '<s>'.
            unk_token (str, optional): The unknown token. Defaults to '<unk>'.
            pad_token (str, optional): The padding token. Defaults to '<pad>'.
            mask_token (str, optional): The masking token. Defaults to '<mask>'.
            sp_model_kwargs (Optional[Dict[str, Any]], optional): Additional keyword arguments for the SentencePieceProcessor. Defaults to None.
            **kwargs: Additional keyword arguments.

        Returns:
            None

        Raises:
            None
        """
        # Mask token behave like a normal word, i.e. include the space before it
        mask_token = AddedToken(mask_token, lstrip=True, rstrip=False) if isinstance(mask_token, str) else mask_token

        self.sp_model_kwargs = {} if sp_model_kwargs is None else sp_model_kwargs

        self.vocab_file = vocab_file
        self.monolingual_vocab_file = monolingual_vocab_file
        self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
        self.sp_model.Load(str(vocab_file))

        # Load the reduced vocab

        # Keep order of special tokens for backward compatibility
        self.fairseq_tokens_to_ids = {}
        cnt = 0
        for token in [bos_token, pad_token, eos_token, unk_token, sep_token, cls_token]:
            if str(token) not in self.fairseq_tokens_to_ids:
                self.fairseq_tokens_to_ids[str(token)] = cnt
                cnt += 1
        with open(monolingual_vocab_file, "r", encoding="utf-8") as f:
            for line in f.readlines():
                token = line.strip().split()[0]
                self.fairseq_tokens_to_ids[token] = len(self.fairseq_tokens_to_ids)
        if str(mask_token) not in self.fairseq_tokens_to_ids:
            self.fairseq_tokens_to_ids[str(mask_token)] = len(self.fairseq_tokens_to_ids)

        self.fairseq_ids_to_tokens = {v: k for k, v in self.fairseq_tokens_to_ids.items()}

        super().__init__(
            bos_token=bos_token,
            eos_token=eos_token,
            unk_token=unk_token,
            sep_token=sep_token,
            cls_token=cls_token,
            pad_token=pad_token,
            mask_token=mask_token,
            sp_model_kwargs=self.sp_model_kwargs,
            **kwargs,
        )

    def __getstate__(self):
        """
        This method '__getstate__' in the class 'BartphoTokenizer' is used to provide the state of the object for pickling.

        Args:
            self: An instance of the BartphoTokenizer class. Represents the current object whose state needs to be retrieved.

        Returns:
            None: This method returns None after setting the 'sp_model' attribute to None and serializing the 'sp_model_proto' attribute.

        Raises:
            None.
        """
        state = self.__dict__.copy()
        state["sp_model"] = None
        state["sp_model_proto"] = self.sp_model.serialized_model_proto()
        return state

    def __setstate__(self, d):
        """
        This method '__setstate__' is defined within the class 'BartphoTokenizer' and is used to restore the 
        object's state from a dictionary representation. 
        It takes two parameters, 'self' which refers to the instance of the class, and 'd' which is a dictionary 
        representing the state to be restored.

        Args:
            self (BartphoTokenizer): The instance of the BartphoTokenizer class.
            d (dict): A dictionary representing the state to be restored. 
                It contains the data necessary to reforward the object's state.

        Returns:
            None.

        Raises:
            None:
                However, potential exceptions that could arise during the execution of this method may include, 
                but are not limited to, those related to the initialization of the 'SentencePieceProcessor' instance or 
                its 'LoadFromSerializedProto' method.
        """
        self.__dict__ = d

        # for backward compatibility
        if not hasattr(self, "sp_model_kwargs"):
            self.sp_model_kwargs = {}

        self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
        self.sp_model.LoadFromSerializedProto(self.sp_model_proto)

    def build_inputs_with_special_tokens(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
    ) -> List[int]:
        """
        Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
        adding special tokens. An BARTPho sequence has the following format:

        - single sequence: `<s> X </s>`
        - pair of sequences: `<s> A </s></s> B </s>`

        Args:
            token_ids_0 (`List[int]`):
                List of IDs to which the special tokens will be added.
            token_ids_1 (`List[int]`, *optional*):
                Optional second list of IDs for sequence pairs.

        Returns:
            `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
        """
        if token_ids_1 is None:
            return [self.cls_token_id] + token_ids_0 + [self.sep_token_id]
        cls = [self.cls_token_id]
        sep = [self.sep_token_id]
        return cls + token_ids_0 + sep + sep + token_ids_1 + sep

    def get_special_tokens_mask(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False
    ) -> List[int]:
        """
        Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
        special tokens using the tokenizer `prepare_for_model` method.

        Args:
            token_ids_0 (`List[int]`):
                List of IDs.
            token_ids_1 (`List[int]`, *optional*):
                Optional second list of IDs for sequence pairs.
            already_has_special_tokens (`bool`, *optional*, defaults to `False`):
                Whether or not the token list is already formatted with special tokens for the model.

        Returns:
            `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
        """
        if already_has_special_tokens:
            return super().get_special_tokens_mask(
                token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True
            )

        if token_ids_1 is None:
            return [1] + ([0] * len(token_ids_0)) + [1]
        return [1] + ([0] * len(token_ids_0)) + [1, 1] + ([0] * len(token_ids_1)) + [1]

    def create_token_type_ids_from_sequences(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
    ) -> List[int]:
        """
        Create a mask from the two sequences passed to be used in a sequence-pair classification task. BARTPho does not
        make use of token type ids, therefore a list of zeros is returned.

        Args:
            token_ids_0 (`List[int]`):
                List of IDs.
            token_ids_1 (`List[int]`, *optional*):
                Optional second list of IDs for sequence pairs.

        Returns:
            `List[int]`: List of zeros.

        """
        sep = [self.sep_token_id]
        cls = [self.cls_token_id]

        if token_ids_1 is None:
            return len(cls + token_ids_0 + sep) * [0]
        return len(cls + token_ids_0 + sep + sep + token_ids_1 + sep) * [0]

    @property
    def vocab_size(self):
        """
        Returns the size of the vocabulary used by the BartphoTokenizer.

        Args:
            self: An instance of the BartphoTokenizer class.

        Returns:
            None

        Raises:
            None
        """
        return len(self.fairseq_ids_to_tokens)

    def get_vocab(self):
        """
        Retrieves the vocabulary dictionary of the BartphoTokenizer instance.

        Args:
            self (BartphoTokenizer): The instance of the BartphoTokenizer class.

        Returns:
            vocab (dict): A dictionary containing the vocabulary of the tokenizer. The keys represent the tokens,
                and the values represent their associated IDs. The dictionary includes both the original vocabulary
                defined by the tokenizer as well as any additional tokens that have been added.

        Raises:
            None.
        """
        vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)}
        vocab.update(self.added_tokens_encoder)
        return vocab

    def _tokenize(self, text: str) -> List[str]:
        """
        Tokenizes the input text using the SentencePiece model.

        Args:
            self (BartphoTokenizer): The instance of the BartphoTokenizer class.
            text (str): The input text to be tokenized.

        Returns:
            List[str]: A list of tokenized strings representing the text.

        Raises:
            None
        """
        return self.sp_model.encode(text, out_type=str)

    def _convert_token_to_id(self, token):
        """Converts a token (str) in an id using the vocab."""
        if token in self.fairseq_tokens_to_ids:
            return self.fairseq_tokens_to_ids[token]
        else:
            return self.unk_token_id

    def _convert_id_to_token(self, index):
        """Converts an index (integer) in a token (str) using the vocab."""
        return self.fairseq_ids_to_tokens[index]

    def convert_tokens_to_string(self, tokens):
        """Converts a sequence of tokens (strings for sub-words) in a single string."""
        out_string = "".join(tokens).replace(SPIECE_UNDERLINE, " ").strip()
        return out_string

    def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
        """
        Saves the vocabulary files to the specified directory.

        Args:
            self (BartphoTokenizer): An instance of the BartphoTokenizer class.
            save_directory (str): The directory where the vocabulary files will be saved.
            filename_prefix (Optional[str], optional): The prefix to be added to the filenames (default: None).

        Returns:
            Tuple[str]: A tuple containing the paths of the saved vocabulary files.

        Raises:
            FileNotFoundError: If the save_directory does not exist.
            IsADirectoryError: If the specified save_directory is not a directory.
        """
        if not os.path.isdir(save_directory):
            logger.error(f"Vocabulary path ({save_directory}) should be a directory")
            return
        out_vocab_file = os.path.join(
            save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
        )
        out_monolingual_vocab_file = os.path.join(
            save_directory,
            (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["monolingual_vocab_file"],
        )

        if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file) and os.path.isfile(self.vocab_file):
            copyfile(self.vocab_file, out_vocab_file)
        elif not os.path.isfile(self.vocab_file):
            with open(out_vocab_file, "wb") as fi:
                content_spiece_model = self.sp_model.serialized_model_proto()
                fi.write(content_spiece_model)

        if os.path.abspath(self.monolingual_vocab_file) != os.path.abspath(
            out_monolingual_vocab_file
        ) and os.path.isfile(self.monolingual_vocab_file):
            copyfile(self.monolingual_vocab_file, out_monolingual_vocab_file)
        elif not os.path.isfile(self.monolingual_vocab_file):
            with open(out_monolingual_vocab_file, "w", encoding="utf-8") as fp:
                for token in self.fairseq_tokens_to_ids:
                    if token not in self.all_special_tokens:
                        fp.write(f"{str(token)} \n")

        return out_vocab_file, out_monolingual_vocab_file

mindnlp.transformers.models.bartpho.tokenization_bartpho.BartphoTokenizer.vocab_size property

Returns the size of the vocabulary used by the BartphoTokenizer.

PARAMETER	DESCRIPTION
self	An instance of the BartphoTokenizer class.

RETURNS	DESCRIPTION
	None

mindnlp.transformers.models.bartpho.tokenization_bartpho.BartphoTokenizer.__getstate__()

This method 'getstate' in the class 'BartphoTokenizer' is used to provide the state of the object for pickling.

PARAMETER	DESCRIPTION
self	An instance of the BartphoTokenizer class. Represents the current object whose state needs to be retrieved.

RETURNS	DESCRIPTION
None	This method returns None after setting the 'sp_model' attribute to None and serializing the 'sp_model_proto' attribute.

Source code in mindnlp/transformers/models/bartpho/tokenization_bartpho.py

def __getstate__(self):
    """
    This method '__getstate__' in the class 'BartphoTokenizer' is used to provide the state of the object for pickling.

    Args:
        self: An instance of the BartphoTokenizer class. Represents the current object whose state needs to be retrieved.

    Returns:
        None: This method returns None after setting the 'sp_model' attribute to None and serializing the 'sp_model_proto' attribute.

    Raises:
        None.
    """
    state = self.__dict__.copy()
    state["sp_model"] = None
    state["sp_model_proto"] = self.sp_model.serialized_model_proto()
    return state

mindnlp.transformers.models.bartpho.tokenization_bartpho.BartphoTokenizer.__init__(vocab_file, monolingual_vocab_file, bos_token='<s>', eos_token='</s>', sep_token='</s>', cls_token='<s>', unk_token='<unk>', pad_token='<pad>', mask_token='<mask>', sp_model_kwargs=None, **kwargs)

Initializes a new instance of the BartphoTokenizer class.

PARAMETER	DESCRIPTION
self	The instance of the class.
vocab_file	The path to the vocabulary file. TYPE: str
monolingual_vocab_file	The path to the monolingual vocabulary file. TYPE: str
bos_token	The beginning of sentence token. Defaults to ''. TYPE: str DEFAULT: '<s>'
eos_token	The end of sentence token. Defaults to ''. TYPE: str DEFAULT: '</s>'
sep_token	The separator token. Defaults to ''. TYPE: str DEFAULT: '</s>'
cls_token	The classification token. Defaults to ''. TYPE: str DEFAULT: '<s>'
unk_token	The unknown token. Defaults to ''. TYPE: str DEFAULT: '<unk>'
pad_token	The padding token. Defaults to ''. TYPE: str DEFAULT: '<pad>'
mask_token	The masking token. Defaults to ''. TYPE: str DEFAULT: '<mask>'
sp_model_kwargs	Additional keyword arguments for the SentencePieceProcessor. Defaults to None. TYPE: Optional[Dict[str, Any]] DEFAULT: None
**kwargs	Additional keyword arguments. DEFAULT: {}

RETURNS	DESCRIPTION
None	None

Source code in mindnlp/transformers/models/bartpho/tokenization_bartpho.py

def __init__(
    self,
    vocab_file,
    monolingual_vocab_file,
    bos_token="<s>",
    eos_token="</s>",
    sep_token="</s>",
    cls_token="<s>",
    unk_token="<unk>",
    pad_token="<pad>",
    mask_token="<mask>",
    sp_model_kwargs: Optional[Dict[str, Any]] = None,
    **kwargs,
) -> None:
    """
    Initializes a new instance of the BartphoTokenizer class.

    Args:
        self: The instance of the class.
        vocab_file (str): The path to the vocabulary file.
        monolingual_vocab_file (str): The path to the monolingual vocabulary file.
        bos_token (str, optional): The beginning of sentence token. Defaults to '<s>'.
        eos_token (str, optional): The end of sentence token. Defaults to '</s>'.
        sep_token (str, optional): The separator token. Defaults to '</s>'.
        cls_token (str, optional): The classification token. Defaults to '<s>'.
        unk_token (str, optional): The unknown token. Defaults to '<unk>'.
        pad_token (str, optional): The padding token. Defaults to '<pad>'.
        mask_token (str, optional): The masking token. Defaults to '<mask>'.
        sp_model_kwargs (Optional[Dict[str, Any]], optional): Additional keyword arguments for the SentencePieceProcessor. Defaults to None.
        **kwargs: Additional keyword arguments.

    Returns:
        None

    Raises:
        None
    """
    # Mask token behave like a normal word, i.e. include the space before it
    mask_token = AddedToken(mask_token, lstrip=True, rstrip=False) if isinstance(mask_token, str) else mask_token

    self.sp_model_kwargs = {} if sp_model_kwargs is None else sp_model_kwargs

    self.vocab_file = vocab_file
    self.monolingual_vocab_file = monolingual_vocab_file
    self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
    self.sp_model.Load(str(vocab_file))

    # Load the reduced vocab

    # Keep order of special tokens for backward compatibility
    self.fairseq_tokens_to_ids = {}
    cnt = 0
    for token in [bos_token, pad_token, eos_token, unk_token, sep_token, cls_token]:
        if str(token) not in self.fairseq_tokens_to_ids:
            self.fairseq_tokens_to_ids[str(token)] = cnt
            cnt += 1
    with open(monolingual_vocab_file, "r", encoding="utf-8") as f:
        for line in f.readlines():
            token = line.strip().split()[0]
            self.fairseq_tokens_to_ids[token] = len(self.fairseq_tokens_to_ids)
    if str(mask_token) not in self.fairseq_tokens_to_ids:
        self.fairseq_tokens_to_ids[str(mask_token)] = len(self.fairseq_tokens_to_ids)

    self.fairseq_ids_to_tokens = {v: k for k, v in self.fairseq_tokens_to_ids.items()}

    super().__init__(
        bos_token=bos_token,
        eos_token=eos_token,
        unk_token=unk_token,
        sep_token=sep_token,
        cls_token=cls_token,
        pad_token=pad_token,
        mask_token=mask_token,
        sp_model_kwargs=self.sp_model_kwargs,
        **kwargs,
    )

mindnlp.transformers.models.bartpho.tokenization_bartpho.BartphoTokenizer.__setstate__(d)

This method 'setstate' is defined within the class 'BartphoTokenizer' and is used to restore the object's state from a dictionary representation. It takes two parameters, 'self' which refers to the instance of the class, and 'd' which is a dictionary representing the state to be restored.

PARAMETER	DESCRIPTION
self	The instance of the BartphoTokenizer class. TYPE: BartphoTokenizer
d	A dictionary representing the state to be restored. It contains the data necessary to reforward the object's state. TYPE: dict

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
None	However, potential exceptions that could arise during the execution of this method may include, but are not limited to, those related to the initialization of the 'SentencePieceProcessor' instance or its 'LoadFromSerializedProto' method.

Source code in mindnlp/transformers/models/bartpho/tokenization_bartpho.py

def __setstate__(self, d):
    """
    This method '__setstate__' is defined within the class 'BartphoTokenizer' and is used to restore the 
    object's state from a dictionary representation. 
    It takes two parameters, 'self' which refers to the instance of the class, and 'd' which is a dictionary 
    representing the state to be restored.

    Args:
        self (BartphoTokenizer): The instance of the BartphoTokenizer class.
        d (dict): A dictionary representing the state to be restored. 
            It contains the data necessary to reforward the object's state.

    Returns:
        None.

    Raises:
        None:
            However, potential exceptions that could arise during the execution of this method may include, 
            but are not limited to, those related to the initialization of the 'SentencePieceProcessor' instance or 
            its 'LoadFromSerializedProto' method.
    """
    self.__dict__ = d

    # for backward compatibility
    if not hasattr(self, "sp_model_kwargs"):
        self.sp_model_kwargs = {}

    self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
    self.sp_model.LoadFromSerializedProto(self.sp_model_proto)

mindnlp.transformers.models.bartpho.tokenization_bartpho.BartphoTokenizer.build_inputs_with_special_tokens(token_ids_0, token_ids_1=None)

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An BARTPho sequence has the following format:

• single sequence: <s> X </s>
• pair of sequences: <s> A </s></s> B </s>

PARAMETER	DESCRIPTION
token_ids_0	List of IDs to which the special tokens will be added. TYPE: `List[int]`
token_ids_1	Optional second list of IDs for sequence pairs. TYPE: `List[int]`, *optional* DEFAULT: None

RETURNS	DESCRIPTION
List[int]	List[int]: List of input IDs with the appropriate special tokens.

Source code in mindnlp/transformers/models/bartpho/tokenization_bartpho.py

def build_inputs_with_special_tokens(
    self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
) -> List[int]:
    """
    Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
    adding special tokens. An BARTPho sequence has the following format:

    - single sequence: `<s> X </s>`
    - pair of sequences: `<s> A </s></s> B </s>`

    Args:
        token_ids_0 (`List[int]`):
            List of IDs to which the special tokens will be added.
        token_ids_1 (`List[int]`, *optional*):
            Optional second list of IDs for sequence pairs.

    Returns:
        `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
    """
    if token_ids_1 is None:
        return [self.cls_token_id] + token_ids_0 + [self.sep_token_id]
    cls = [self.cls_token_id]
    sep = [self.sep_token_id]
    return cls + token_ids_0 + sep + sep + token_ids_1 + sep

mindnlp.transformers.models.bartpho.tokenization_bartpho.BartphoTokenizer.convert_tokens_to_string(tokens)

Converts a sequence of tokens (strings for sub-words) in a single string.

Source code in mindnlp/transformers/models/bartpho/tokenization_bartpho.py

def convert_tokens_to_string(self, tokens):
    """Converts a sequence of tokens (strings for sub-words) in a single string."""
    out_string = "".join(tokens).replace(SPIECE_UNDERLINE, " ").strip()
    return out_string

mindnlp.transformers.models.bartpho.tokenization_bartpho.BartphoTokenizer.create_token_type_ids_from_sequences(token_ids_0, token_ids_1=None)

Create a mask from the two sequences passed to be used in a sequence-pair classification task. BARTPho does not make use of token type ids, therefore a list of zeros is returned.

PARAMETER	DESCRIPTION
token_ids_0	List of IDs. TYPE: `List[int]`
token_ids_1	Optional second list of IDs for sequence pairs. TYPE: `List[int]`, *optional* DEFAULT: None

RETURNS	DESCRIPTION
List[int]	List[int]: List of zeros.

Source code in mindnlp/transformers/models/bartpho/tokenization_bartpho.py

def create_token_type_ids_from_sequences(
    self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
) -> List[int]:
    """
    Create a mask from the two sequences passed to be used in a sequence-pair classification task. BARTPho does not
    make use of token type ids, therefore a list of zeros is returned.

    Args:
        token_ids_0 (`List[int]`):
            List of IDs.
        token_ids_1 (`List[int]`, *optional*):
            Optional second list of IDs for sequence pairs.

    Returns:
        `List[int]`: List of zeros.

    """
    sep = [self.sep_token_id]
    cls = [self.cls_token_id]

    if token_ids_1 is None:
        return len(cls + token_ids_0 + sep) * [0]
    return len(cls + token_ids_0 + sep + sep + token_ids_1 + sep) * [0]

mindnlp.transformers.models.bartpho.tokenization_bartpho.BartphoTokenizer.get_special_tokens_mask(token_ids_0, token_ids_1=None, already_has_special_tokens=False)

Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer prepare_for_model method.

PARAMETER	DESCRIPTION
token_ids_0	List of IDs. TYPE: `List[int]`
token_ids_1	Optional second list of IDs for sequence pairs. TYPE: `List[int]`, *optional* DEFAULT: None
already_has_special_tokens	Whether or not the token list is already formatted with special tokens for the model. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False

RETURNS	DESCRIPTION
List[int]	List[int]: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.

Source code in mindnlp/transformers/models/bartpho/tokenization_bartpho.py

def get_special_tokens_mask(
    self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False
) -> List[int]:
    """
    Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
    special tokens using the tokenizer `prepare_for_model` method.

    Args:
        token_ids_0 (`List[int]`):
            List of IDs.
        token_ids_1 (`List[int]`, *optional*):
            Optional second list of IDs for sequence pairs.
        already_has_special_tokens (`bool`, *optional*, defaults to `False`):
            Whether or not the token list is already formatted with special tokens for the model.

    Returns:
        `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
    """
    if already_has_special_tokens:
        return super().get_special_tokens_mask(
            token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True
        )

    if token_ids_1 is None:
        return [1] + ([0] * len(token_ids_0)) + [1]
    return [1] + ([0] * len(token_ids_0)) + [1, 1] + ([0] * len(token_ids_1)) + [1]

mindnlp.transformers.models.bartpho.tokenization_bartpho.BartphoTokenizer.get_vocab()

Retrieves the vocabulary dictionary of the BartphoTokenizer instance.

PARAMETER	DESCRIPTION
self	The instance of the BartphoTokenizer class. TYPE: BartphoTokenizer

RETURNS	DESCRIPTION
vocab	A dictionary containing the vocabulary of the tokenizer. The keys represent the tokens, and the values represent their associated IDs. The dictionary includes both the original vocabulary defined by the tokenizer as well as any additional tokens that have been added. TYPE: dict

Source code in mindnlp/transformers/models/bartpho/tokenization_bartpho.py

def get_vocab(self):
    """
    Retrieves the vocabulary dictionary of the BartphoTokenizer instance.

    Args:
        self (BartphoTokenizer): The instance of the BartphoTokenizer class.

    Returns:
        vocab (dict): A dictionary containing the vocabulary of the tokenizer. The keys represent the tokens,
            and the values represent their associated IDs. The dictionary includes both the original vocabulary
            defined by the tokenizer as well as any additional tokens that have been added.

    Raises:
        None.
    """
    vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)}
    vocab.update(self.added_tokens_encoder)
    return vocab

mindnlp.transformers.models.bartpho.tokenization_bartpho.BartphoTokenizer.save_vocabulary(save_directory, filename_prefix=None)

Saves the vocabulary files to the specified directory.

PARAMETER	DESCRIPTION
self	An instance of the BartphoTokenizer class. TYPE: BartphoTokenizer
save_directory	The directory where the vocabulary files will be saved. TYPE: str
filename_prefix	The prefix to be added to the filenames (default: None). TYPE: Optional[str] DEFAULT: None

RETURNS	DESCRIPTION
Tuple[str]	Tuple[str]: A tuple containing the paths of the saved vocabulary files.

RAISES	DESCRIPTION
FileNotFoundError	If the save_directory does not exist.
IsADirectoryError	If the specified save_directory is not a directory.

Source code in mindnlp/transformers/models/bartpho/tokenization_bartpho.py

def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
    """
    Saves the vocabulary files to the specified directory.

    Args:
        self (BartphoTokenizer): An instance of the BartphoTokenizer class.
        save_directory (str): The directory where the vocabulary files will be saved.
        filename_prefix (Optional[str], optional): The prefix to be added to the filenames (default: None).

    Returns:
        Tuple[str]: A tuple containing the paths of the saved vocabulary files.

    Raises:
        FileNotFoundError: If the save_directory does not exist.
        IsADirectoryError: If the specified save_directory is not a directory.
    """
    if not os.path.isdir(save_directory):
        logger.error(f"Vocabulary path ({save_directory}) should be a directory")
        return
    out_vocab_file = os.path.join(
        save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
    )
    out_monolingual_vocab_file = os.path.join(
        save_directory,
        (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["monolingual_vocab_file"],
    )

    if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file) and os.path.isfile(self.vocab_file):
        copyfile(self.vocab_file, out_vocab_file)
    elif not os.path.isfile(self.vocab_file):
        with open(out_vocab_file, "wb") as fi:
            content_spiece_model = self.sp_model.serialized_model_proto()
            fi.write(content_spiece_model)

    if os.path.abspath(self.monolingual_vocab_file) != os.path.abspath(
        out_monolingual_vocab_file
    ) and os.path.isfile(self.monolingual_vocab_file):
        copyfile(self.monolingual_vocab_file, out_monolingual_vocab_file)
    elif not os.path.isfile(self.monolingual_vocab_file):
        with open(out_monolingual_vocab_file, "w", encoding="utf-8") as fp:
            for token in self.fairseq_tokens_to_ids:
                if token not in self.all_special_tokens:
                    fp.write(f"{str(token)} \n")

    return out_vocab_file, out_monolingual_vocab_file