barthez

mindnlp.transformers.models.barthez.tokenization_barthez.BarthezTokenizer

Bases: PreTrainedTokenizer

Adapted from [CamembertTokenizer] and [BartTokenizer]. Construct a BARThez tokenizer. Based on SentencePiece.

This tokenizer inherits from [PreTrainedTokenizer] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

PARAMETER	DESCRIPTION
vocab_file	SentencePiece file (generally has a .spm extension) that contains the vocabulary necessary to instantiate a tokenizer. TYPE: `str`
bos_token	The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. defaults to "<s>" When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the cls_token. TYPE: `str`, *optional* DEFAULT: '<s>'
eos_token	The end of sequence token. defaults to "</s>" When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the sep_token. TYPE: `str`, *optional* DEFAULT: '</s>'
sep_token	The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. defaults to "</s>" TYPE: `str`, *optional* DEFAULT: '</s>'
cls_token	The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens. defaults to "<s>" TYPE: `str`, *optional* DEFAULT: '<s>'
unk_token	The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. defaults to "<unk>" TYPE: `str`, *optional* DEFAULT: '<unk>'
pad_token	The token used for padding, for example when batching sequences of different lengths. defaults to "<pad>" TYPE: `str`, *optional* DEFAULT: '<pad>'
mask_token	The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict. defaults to "<mask>" TYPE: `str`, *optional* DEFAULT: '<mask>'
sp_model_kwargs	Will be passed to the SentencePieceProcessor.__init__() method. The Python wrapper for SentencePiece can be used, among other things, to set: enable_sampling: Enable subword regularization. nbest_size: Sampling parameters for unigram. Invalid for BPE-Dropout. nbest_size = {0,1}: No sampling is performed. nbest_size > 1: samples from the nbest_size results. nbest_size < 0: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. alpha: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. TYPE: `dict`, *optional* DEFAULT: None

ATTRIBUTE	DESCRIPTION
sp_model	The SentencePiece processor that is used for every conversion (string, tokens and IDs). TYPE: `SentencePieceProcessor`

Source code in mindnlp/transformers/models/barthez/tokenization_barthez.py

class BarthezTokenizer(PreTrainedTokenizer):
    """
    Adapted from [`CamembertTokenizer`] and [`BartTokenizer`]. Construct a BARThez tokenizer. Based on
    [SentencePiece](https://github.com/google/sentencepiece).

    This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
    this superclass for more information regarding those methods.

    Args:
        vocab_file (`str`):
            [SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
            contains the vocabulary necessary to instantiate a tokenizer.
        bos_token (`str`, *optional*):
            The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
            defaults to `"<s>"`

            <Tip>

            When building a sequence using special tokens, this is not the token that is used for the beginning of
            sequence. The token used is the `cls_token`.

            </Tip>

        eos_token (`str`, *optional*):
            The end of sequence token. defaults to `"</s>"`

            <Tip>

            When building a sequence using special tokens, this is not the token that is used for the end of sequence.
            The token used is the `sep_token`.

            </Tip>

        sep_token (`str`, *optional*):
            The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
            sequence classification or for a text and a question for question answering. It is also used as the last
            token of a sequence built with special tokens. defaults to `"</s>"`
        cls_token (`str`, *optional*):
            The classifier token which is used when doing sequence classification (classification of the whole sequence
            instead of per-token classification). It is the first token of the sequence when built with special tokens.
            defaults to `"<s>"`
        unk_token (`str`, *optional*):
            The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
            token instead. defaults to `"<unk>"`
        pad_token (`str`, *optional*):
            The token used for padding, for example when batching sequences of different lengths. defaults to `"<pad>"`
        mask_token (`str`, *optional*):
            The token used for masking values. This is the token used when training this model with masked language
            modeling. This is the token which the model will try to predict. defaults to `"<mask>"`
        sp_model_kwargs (`dict`, *optional*):
            Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for
            SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things,
            to set:

            - `enable_sampling`: Enable subword regularization.
            - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.

                - `nbest_size = {0,1}`: No sampling is performed.
                - `nbest_size > 1`: samples from the nbest_size results.
                - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
                using forward-filtering-and-backward-sampling algorithm.
           - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
              BPE-dropout.

    Attributes:
        sp_model (`SentencePieceProcessor`):
            The *SentencePiece* processor that is used for every conversion (string, tokens and IDs).
    """
    vocab_files_names = VOCAB_FILES_NAMES
    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
    model_input_names = ["input_ids", "attention_mask"]

    def __init__(
        self,
        vocab_file,
        bos_token="<s>",
        eos_token="</s>",
        sep_token="</s>",
        cls_token="<s>",
        unk_token="<unk>",
        pad_token="<pad>",
        mask_token="<mask>",
        sp_model_kwargs: Optional[Dict[str, Any]] = None,
        **kwargs,
    ) -> None:
        """
        __init__

        Initializes a new instance of the BarthezTokenizer class.

        Args:
            self: The instance of the class.
            vocab_file (str): The path to the vocabulary file.
            bos_token (str, optional): The beginning of sequence token. Defaults to '<s>'.
            eos_token (str, optional): The end of sequence token. Defaults to '</s>'.
            sep_token (str, optional): The separator token. Defaults to '</s>'.
            cls_token (str, optional): The classification token. Defaults to '<s>'.
            unk_token (str, optional): The unknown token. Defaults to '<unk>'.
            pad_token (str, optional): The padding token. Defaults to '<pad>'.
            mask_token (str, optional): The mask token. Defaults to '<mask>'.
            sp_model_kwargs (Optional[Dict[str, Any]], optional): Optional sentence piece model arguments. Defaults to None.

        Returns:
            None.

        Raises:
            TypeError: If the vocab_file is not a valid string.
            TypeError: If any token parameter is not a valid string.
            TypeError: If sp_model_kwargs is not a valid dictionary.
            OSError: If the vocab_file cannot be loaded.
            ValueError: If the sp_model_kwargs are invalid.
        """
        # Mask token behave like a normal word, i.e. include the space before it. Will have normalized=False by default this way
        mask_token = AddedToken(mask_token, lstrip=True, special=True) if isinstance(mask_token, str) else mask_token

        self.sp_model_kwargs = {} if sp_model_kwargs is None else sp_model_kwargs

        self.vocab_file = vocab_file
        self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
        self.sp_model.Load(str(vocab_file))
        super().__init__(
            bos_token=bos_token,
            eos_token=eos_token,
            unk_token=unk_token,
            sep_token=sep_token,
            cls_token=cls_token,
            pad_token=pad_token,
            mask_token=mask_token,
            sp_model_kwargs=self.sp_model_kwargs,
            **kwargs,
        )

    def build_inputs_with_special_tokens(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
    ) -> List[int]:
        """
        Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
        adding special tokens. A BARThez sequence has the following format:

        - single sequence: `<s> X </s>`
        - pair of sequences: `<s> A </s></s> B </s>`

        Args:
            token_ids_0 (`List[int]`):
                List of IDs to which the special tokens will be added.
            token_ids_1 (`List[int]`, *optional*):
                Optional second list of IDs for sequence pairs.

        Returns:
            `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
        """
        if token_ids_1 is None:
            return [self.cls_token_id] + token_ids_0 + [self.sep_token_id]
        cls = [self.cls_token_id]
        sep = [self.sep_token_id]
        return cls + token_ids_0 + sep + sep + token_ids_1 + sep

    def get_special_tokens_mask(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False
    ) -> List[int]:
        """
        Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
        special tokens using the tokenizer `prepare_for_model` method.

        Args:
            token_ids_0 (`List[int]`):
                List of IDs.
            token_ids_1 (`List[int]`, *optional*):
                Optional second list of IDs for sequence pairs.
            already_has_special_tokens (`bool`, *optional*, defaults to `False`):
                Whether or not the token list is already formatted with special tokens for the model.

        Returns:
            `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
        """
        if already_has_special_tokens:
            return super().get_special_tokens_mask(
                token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True
            )

        if token_ids_1 is None:
            return [1] + ([0] * len(token_ids_0)) + [1]
        return [1] + ([0] * len(token_ids_0)) + [1, 1] + ([0] * len(token_ids_1)) + [1]

    def create_token_type_ids_from_sequences(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
    ) -> List[int]:
        """
        Create a mask from the two sequences passed to be used in a sequence-pair classification task.

        Args:
            token_ids_0 (`List[int]`):
                List of IDs.
            token_ids_1 (`List[int]`, *optional*):
                Optional second list of IDs for sequence pairs.

        Returns:
            `List[int]`: List of zeros.
        """
        sep = [self.sep_token_id]
        cls = [self.cls_token_id]

        if token_ids_1 is None:
            return len(cls + token_ids_0 + sep) * [0]
        return len(cls + token_ids_0 + sep + sep + token_ids_1 + sep) * [0]

    @property
    def vocab_size(self):
        """
        Returns the size of the vocabulary used by the BarthezTokenizer instance.

        Args:
            self (BarthezTokenizer): The instance of the BarthezTokenizer class.

        Returns:
            int: The size of the vocabulary used by the BarthezTokenizer instance.

        Raises:
            None.
        """
        return len(self.sp_model)

    def get_vocab(self):
        """
        Get the vocabulary of the BarthezTokenizer.

        Args:
            self (BarthezTokenizer): The instance of the BarthezTokenizer class.
                It represents the tokenizer object.

        Returns:
            dict: A dictionary containing the vocabulary of the tokenizer.
                Keys are tokens and values are corresponding token IDs.

        Raises:
            None.
        """
        vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)}
        vocab.update(self.added_tokens_encoder)
        return vocab

    def _tokenize(self, text: str) -> List[str]:
        """
        This method '_tokenize' in the class 'BarthezTokenizer' tokenizes the input text using the SentencePiece model.

        Args:
            self: An instance of the BarthezTokenizer class.
            text (str): The input text to be tokenized.

        Returns:
            List[str]: A list of strings representing the tokens generated from the input text.

        Raises:
            None
        """
        return self.sp_model.encode(text, out_type=str)

    def _convert_token_to_id(self, token):
        """Converts a token (str) in an id using the vocab."""
        return self.sp_model.PieceToId(token)

    def _convert_id_to_token(self, index):
        """Converts an index (integer) in a token (str) using the vocab."""
        return self.sp_model.IdToPiece(index)

    # Copied from transformers.models.albert.tokenization_albert.AlbertTokenizer.convert_tokens_to_string
    def convert_tokens_to_string(self, tokens):
        """Converts a sequence of tokens (string) in a single string."""
        current_sub_tokens = []
        out_string = ""
        prev_is_special = False
        for token in tokens:
            # make sure that special tokens are not decoded using sentencepiece model
            if token in self.all_special_tokens:
                if not prev_is_special:
                    out_string += " "
                out_string += self.sp_model.decode(current_sub_tokens) + token
                prev_is_special = True
                current_sub_tokens = []
            else:
                current_sub_tokens.append(token)
                prev_is_special = False
        out_string += self.sp_model.decode(current_sub_tokens)
        return out_string.strip()

    def __getstate__(self):
        """
        __getstate__

        Description:
            This method is used to return the state of the BarthezTokenizer object for pickling.

        Args:
            self (object): The instance of the BarthezTokenizer class.

        Returns:
            None: This method returns a value of type None,
                indicating that it does not return any specific data but modifies the state of the object.

        Raises:
            None.
        """
        state = self.__dict__.copy()
        state["sp_model"] = None
        return state

    def __setstate__(self, d):
        """
        Sets the state of the BarthezTokenizer object by restoring its attributes from a dictionary.

        Args:
            self (BarthezTokenizer): The instance of the BarthezTokenizer class.
            d (dict): The dictionary containing the attributes to restore the state of the object.

        Returns:
            None.

        Raises:
            None.
        """
        self.__dict__ = d

        # for backward compatibility
        if not hasattr(self, "sp_model_kwargs"):
            self.sp_model_kwargs = {}

        self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
        self.sp_model.Load(self.vocab_file)

    def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
        """
        Save the vocabulary of the BarthezTokenizer to a specified directory.

        Args:
            self (BarthezTokenizer): An instance of the BarthezTokenizer class.
            save_directory (str): The directory where the vocabulary will be saved.
            filename_prefix (Optional[str], optional): The prefix to be added to the filename. Defaults to None.

        Returns:
            Tuple[str]: A tuple containing the path of the saved vocabulary file.

        Raises:
            OSError: If the save_directory is not a valid directory.
            FileNotFoundError: If the self.vocab_file does not exist.
            IOError: If an error occurs while copying the vocabulary file.
            Exception: If any other exception occurs.

        Note:
            - The save_directory should be a valid directory where the vocabulary file will be saved.
            - The filename_prefix, if provided, will be added as a prefix to the filename.
            - The method either copies the existing vocabulary file or creates a new one if it does not exist.
        """
        if not os.path.isdir(save_directory):
            logger.error(f"Vocabulary path ({save_directory}) should be a directory")
            return
        out_vocab_file = os.path.join(
            save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
        )

        if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file) and os.path.isfile(self.vocab_file):
            copyfile(self.vocab_file, out_vocab_file)
        elif not os.path.isfile(self.vocab_file):
            with open(out_vocab_file, "wb") as fi:
                content_spiece_model = self.sp_model.serialized_model_proto()
                fi.write(content_spiece_model)

        return (out_vocab_file,)

mindnlp.transformers.models.barthez.tokenization_barthez.BarthezTokenizer.vocab_size property

Returns the size of the vocabulary used by the BarthezTokenizer instance.

PARAMETER	DESCRIPTION
self	The instance of the BarthezTokenizer class. TYPE: BarthezTokenizer

RETURNS	DESCRIPTION
int	The size of the vocabulary used by the BarthezTokenizer instance.

mindnlp.transformers.models.barthez.tokenization_barthez.BarthezTokenizer.__getstate__()

getstate

Description

This method is used to return the state of the BarthezTokenizer object for pickling.

PARAMETER	DESCRIPTION
self	The instance of the BarthezTokenizer class. TYPE: object

RETURNS	DESCRIPTION
None	This method returns a value of type None, indicating that it does not return any specific data but modifies the state of the object.

Source code in mindnlp/transformers/models/barthez/tokenization_barthez.py

def __getstate__(self):
    """
    __getstate__

    Description:
        This method is used to return the state of the BarthezTokenizer object for pickling.

    Args:
        self (object): The instance of the BarthezTokenizer class.

    Returns:
        None: This method returns a value of type None,
            indicating that it does not return any specific data but modifies the state of the object.

    Raises:
        None.
    """
    state = self.__dict__.copy()
    state["sp_model"] = None
    return state

mindnlp.transformers.models.barthez.tokenization_barthez.BarthezTokenizer.__init__(vocab_file, bos_token='<s>', eos_token='</s>', sep_token='</s>', cls_token='<s>', unk_token='<unk>', pad_token='<pad>', mask_token='<mask>', sp_model_kwargs=None, **kwargs)

init

Initializes a new instance of the BarthezTokenizer class.

PARAMETER	DESCRIPTION
self	The instance of the class.
vocab_file	The path to the vocabulary file. TYPE: str
bos_token	The beginning of sequence token. Defaults to ''. TYPE: str DEFAULT: '<s>'
eos_token	The end of sequence token. Defaults to ''. TYPE: str DEFAULT: '</s>'
sep_token	The separator token. Defaults to ''. TYPE: str DEFAULT: '</s>'
cls_token	The classification token. Defaults to ''. TYPE: str DEFAULT: '<s>'
unk_token	The unknown token. Defaults to ''. TYPE: str DEFAULT: '<unk>'
pad_token	The padding token. Defaults to ''. TYPE: str DEFAULT: '<pad>'
mask_token	The mask token. Defaults to ''. TYPE: str DEFAULT: '<mask>'
sp_model_kwargs	Optional sentence piece model arguments. Defaults to None. TYPE: Optional[Dict[str, Any]] DEFAULT: None

RETURNS	DESCRIPTION
None	None.

RAISES	DESCRIPTION
TypeError	If the vocab_file is not a valid string.
TypeError	If any token parameter is not a valid string.
TypeError	If sp_model_kwargs is not a valid dictionary.
OSError	If the vocab_file cannot be loaded.
ValueError	If the sp_model_kwargs are invalid.

Source code in mindnlp/transformers/models/barthez/tokenization_barthez.py

def __init__(
    self,
    vocab_file,
    bos_token="<s>",
    eos_token="</s>",
    sep_token="</s>",
    cls_token="<s>",
    unk_token="<unk>",
    pad_token="<pad>",
    mask_token="<mask>",
    sp_model_kwargs: Optional[Dict[str, Any]] = None,
    **kwargs,
) -> None:
    """
    __init__

    Initializes a new instance of the BarthezTokenizer class.

    Args:
        self: The instance of the class.
        vocab_file (str): The path to the vocabulary file.
        bos_token (str, optional): The beginning of sequence token. Defaults to '<s>'.
        eos_token (str, optional): The end of sequence token. Defaults to '</s>'.
        sep_token (str, optional): The separator token. Defaults to '</s>'.
        cls_token (str, optional): The classification token. Defaults to '<s>'.
        unk_token (str, optional): The unknown token. Defaults to '<unk>'.
        pad_token (str, optional): The padding token. Defaults to '<pad>'.
        mask_token (str, optional): The mask token. Defaults to '<mask>'.
        sp_model_kwargs (Optional[Dict[str, Any]], optional): Optional sentence piece model arguments. Defaults to None.

    Returns:
        None.

    Raises:
        TypeError: If the vocab_file is not a valid string.
        TypeError: If any token parameter is not a valid string.
        TypeError: If sp_model_kwargs is not a valid dictionary.
        OSError: If the vocab_file cannot be loaded.
        ValueError: If the sp_model_kwargs are invalid.
    """
    # Mask token behave like a normal word, i.e. include the space before it. Will have normalized=False by default this way
    mask_token = AddedToken(mask_token, lstrip=True, special=True) if isinstance(mask_token, str) else mask_token

    self.sp_model_kwargs = {} if sp_model_kwargs is None else sp_model_kwargs

    self.vocab_file = vocab_file
    self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
    self.sp_model.Load(str(vocab_file))
    super().__init__(
        bos_token=bos_token,
        eos_token=eos_token,
        unk_token=unk_token,
        sep_token=sep_token,
        cls_token=cls_token,
        pad_token=pad_token,
        mask_token=mask_token,
        sp_model_kwargs=self.sp_model_kwargs,
        **kwargs,
    )

mindnlp.transformers.models.barthez.tokenization_barthez.BarthezTokenizer.__setstate__(d)

Sets the state of the BarthezTokenizer object by restoring its attributes from a dictionary.

PARAMETER	DESCRIPTION
self	The instance of the BarthezTokenizer class. TYPE: BarthezTokenizer
d	The dictionary containing the attributes to restore the state of the object. TYPE: dict

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/barthez/tokenization_barthez.py

def __setstate__(self, d):
    """
    Sets the state of the BarthezTokenizer object by restoring its attributes from a dictionary.

    Args:
        self (BarthezTokenizer): The instance of the BarthezTokenizer class.
        d (dict): The dictionary containing the attributes to restore the state of the object.

    Returns:
        None.

    Raises:
        None.
    """
    self.__dict__ = d

    # for backward compatibility
    if not hasattr(self, "sp_model_kwargs"):
        self.sp_model_kwargs = {}

    self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
    self.sp_model.Load(self.vocab_file)

mindnlp.transformers.models.barthez.tokenization_barthez.BarthezTokenizer.build_inputs_with_special_tokens(token_ids_0, token_ids_1=None)

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BARThez sequence has the following format:

• single sequence: <s> X </s>
• pair of sequences: <s> A </s></s> B </s>

PARAMETER	DESCRIPTION
token_ids_0	List of IDs to which the special tokens will be added. TYPE: `List[int]`
token_ids_1	Optional second list of IDs for sequence pairs. TYPE: `List[int]`, *optional* DEFAULT: None

RETURNS	DESCRIPTION
List[int]	List[int]: List of input IDs with the appropriate special tokens.

Source code in mindnlp/transformers/models/barthez/tokenization_barthez.py

def build_inputs_with_special_tokens(
    self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
) -> List[int]:
    """
    Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
    adding special tokens. A BARThez sequence has the following format:

    - single sequence: `<s> X </s>`
    - pair of sequences: `<s> A </s></s> B </s>`

    Args:
        token_ids_0 (`List[int]`):
            List of IDs to which the special tokens will be added.
        token_ids_1 (`List[int]`, *optional*):
            Optional second list of IDs for sequence pairs.

    Returns:
        `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
    """
    if token_ids_1 is None:
        return [self.cls_token_id] + token_ids_0 + [self.sep_token_id]
    cls = [self.cls_token_id]
    sep = [self.sep_token_id]
    return cls + token_ids_0 + sep + sep + token_ids_1 + sep

mindnlp.transformers.models.barthez.tokenization_barthez.BarthezTokenizer.convert_tokens_to_string(tokens)

Converts a sequence of tokens (string) in a single string.

Source code in mindnlp/transformers/models/barthez/tokenization_barthez.py

def convert_tokens_to_string(self, tokens):
    """Converts a sequence of tokens (string) in a single string."""
    current_sub_tokens = []
    out_string = ""
    prev_is_special = False
    for token in tokens:
        # make sure that special tokens are not decoded using sentencepiece model
        if token in self.all_special_tokens:
            if not prev_is_special:
                out_string += " "
            out_string += self.sp_model.decode(current_sub_tokens) + token
            prev_is_special = True
            current_sub_tokens = []
        else:
            current_sub_tokens.append(token)
            prev_is_special = False
    out_string += self.sp_model.decode(current_sub_tokens)
    return out_string.strip()

mindnlp.transformers.models.barthez.tokenization_barthez.BarthezTokenizer.create_token_type_ids_from_sequences(token_ids_0, token_ids_1=None)

Create a mask from the two sequences passed to be used in a sequence-pair classification task.

PARAMETER	DESCRIPTION
token_ids_0	List of IDs. TYPE: `List[int]`
token_ids_1	Optional second list of IDs for sequence pairs. TYPE: `List[int]`, *optional* DEFAULT: None

RETURNS	DESCRIPTION
List[int]	List[int]: List of zeros.

Source code in mindnlp/transformers/models/barthez/tokenization_barthez.py

def create_token_type_ids_from_sequences(
    self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
) -> List[int]:
    """
    Create a mask from the two sequences passed to be used in a sequence-pair classification task.

    Args:
        token_ids_0 (`List[int]`):
            List of IDs.
        token_ids_1 (`List[int]`, *optional*):
            Optional second list of IDs for sequence pairs.

    Returns:
        `List[int]`: List of zeros.
    """
    sep = [self.sep_token_id]
    cls = [self.cls_token_id]

    if token_ids_1 is None:
        return len(cls + token_ids_0 + sep) * [0]
    return len(cls + token_ids_0 + sep + sep + token_ids_1 + sep) * [0]

mindnlp.transformers.models.barthez.tokenization_barthez.BarthezTokenizer.get_special_tokens_mask(token_ids_0, token_ids_1=None, already_has_special_tokens=False)

Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer prepare_for_model method.

PARAMETER	DESCRIPTION
token_ids_0	List of IDs. TYPE: `List[int]`
token_ids_1	Optional second list of IDs for sequence pairs. TYPE: `List[int]`, *optional* DEFAULT: None
already_has_special_tokens	Whether or not the token list is already formatted with special tokens for the model. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False

RETURNS	DESCRIPTION
List[int]	List[int]: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.

Source code in mindnlp/transformers/models/barthez/tokenization_barthez.py

def get_special_tokens_mask(
    self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False
) -> List[int]:
    """
    Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
    special tokens using the tokenizer `prepare_for_model` method.

    Args:
        token_ids_0 (`List[int]`):
            List of IDs.
        token_ids_1 (`List[int]`, *optional*):
            Optional second list of IDs for sequence pairs.
        already_has_special_tokens (`bool`, *optional*, defaults to `False`):
            Whether or not the token list is already formatted with special tokens for the model.

    Returns:
        `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
    """
    if already_has_special_tokens:
        return super().get_special_tokens_mask(
            token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True
        )

    if token_ids_1 is None:
        return [1] + ([0] * len(token_ids_0)) + [1]
    return [1] + ([0] * len(token_ids_0)) + [1, 1] + ([0] * len(token_ids_1)) + [1]

mindnlp.transformers.models.barthez.tokenization_barthez.BarthezTokenizer.get_vocab()

Get the vocabulary of the BarthezTokenizer.

PARAMETER	DESCRIPTION
self	The instance of the BarthezTokenizer class. It represents the tokenizer object. TYPE: BarthezTokenizer

RETURNS	DESCRIPTION
dict	A dictionary containing the vocabulary of the tokenizer. Keys are tokens and values are corresponding token IDs.

Source code in mindnlp/transformers/models/barthez/tokenization_barthez.py

def get_vocab(self):
    """
    Get the vocabulary of the BarthezTokenizer.

    Args:
        self (BarthezTokenizer): The instance of the BarthezTokenizer class.
            It represents the tokenizer object.

    Returns:
        dict: A dictionary containing the vocabulary of the tokenizer.
            Keys are tokens and values are corresponding token IDs.

    Raises:
        None.
    """
    vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)}
    vocab.update(self.added_tokens_encoder)
    return vocab

mindnlp.transformers.models.barthez.tokenization_barthez.BarthezTokenizer.save_vocabulary(save_directory, filename_prefix=None)

Save the vocabulary of the BarthezTokenizer to a specified directory.

PARAMETER	DESCRIPTION
self	An instance of the BarthezTokenizer class. TYPE: BarthezTokenizer
save_directory	The directory where the vocabulary will be saved. TYPE: str
filename_prefix	The prefix to be added to the filename. Defaults to None. TYPE: Optional[str] DEFAULT: None

RETURNS	DESCRIPTION
Tuple[str]	Tuple[str]: A tuple containing the path of the saved vocabulary file.

RAISES	DESCRIPTION
OSError	If the save_directory is not a valid directory.
FileNotFoundError	If the self.vocab_file does not exist.
IOError	If an error occurs while copying the vocabulary file.
Exception	If any other exception occurs.

Note

• The save_directory should be a valid directory where the vocabulary file will be saved.
• The filename_prefix, if provided, will be added as a prefix to the filename.
• The method either copies the existing vocabulary file or creates a new one if it does not exist.

Source code in mindnlp/transformers/models/barthez/tokenization_barthez.py

def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
    """
    Save the vocabulary of the BarthezTokenizer to a specified directory.

    Args:
        self (BarthezTokenizer): An instance of the BarthezTokenizer class.
        save_directory (str): The directory where the vocabulary will be saved.
        filename_prefix (Optional[str], optional): The prefix to be added to the filename. Defaults to None.

    Returns:
        Tuple[str]: A tuple containing the path of the saved vocabulary file.

    Raises:
        OSError: If the save_directory is not a valid directory.
        FileNotFoundError: If the self.vocab_file does not exist.
        IOError: If an error occurs while copying the vocabulary file.
        Exception: If any other exception occurs.

    Note:
        - The save_directory should be a valid directory where the vocabulary file will be saved.
        - The filename_prefix, if provided, will be added as a prefix to the filename.
        - The method either copies the existing vocabulary file or creates a new one if it does not exist.
    """
    if not os.path.isdir(save_directory):
        logger.error(f"Vocabulary path ({save_directory}) should be a directory")
        return
    out_vocab_file = os.path.join(
        save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
    )

    if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file) and os.path.isfile(self.vocab_file):
        copyfile(self.vocab_file, out_vocab_file)
    elif not os.path.isfile(self.vocab_file):
        with open(out_vocab_file, "wb") as fi:
            content_spiece_model = self.sp_model.serialized_model_proto()
            fi.write(content_spiece_model)

    return (out_vocab_file,)

mindnlp.transformers.models.barthez.tokenization_barthez_fast.BarthezTokenizerFast

Bases: PreTrainedTokenizerFast

Adapted from [CamembertTokenizer] and [BartTokenizer]. Construct a "fast" BARThez tokenizer. Based on SentencePiece.

This tokenizer inherits from [PreTrainedTokenizerFast] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

PARAMETER	DESCRIPTION
vocab_file	SentencePiece file (generally has a .spm extension) that contains the vocabulary necessary to instantiate a tokenizer. TYPE: `str` DEFAULT: None
bos_token	The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the cls_token. TYPE: `str`, *optional*, defaults to `"<s>"` DEFAULT: '<s>'
eos_token	The end of sequence token. When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the sep_token. TYPE: `str`, *optional*, defaults to `"</s>"` DEFAULT: '</s>'
sep_token	The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. defaults to "</s>" TYPE: `str`, *optional* DEFAULT: '</s>'
cls_token	The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens. defaults to "<s>" TYPE: `str`, *optional* DEFAULT: '<s>'
unk_token	The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. defaults to "<unk>" TYPE: `str`, *optional* DEFAULT: '<unk>'
pad_token	The token used for padding, for example when batching sequences of different lengths. defaults to "<pad>" TYPE: `str`, *optional* DEFAULT: '<pad>'
mask_token	The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict. defaults to "<mask>" TYPE: `str`, *optional* DEFAULT: '<mask>'
additional_special_tokens	Additional special tokens used by the tokenizer. defaults to ["<s>NOTUSED", "</s>NOTUSED"] TYPE: `List[str]`, *optional*

Source code in mindnlp/transformers/models/barthez/tokenization_barthez_fast.py

class BarthezTokenizerFast(PreTrainedTokenizerFast):
    """
    Adapted from [`CamembertTokenizer`] and [`BartTokenizer`]. Construct a "fast" BARThez tokenizer. Based on
    [SentencePiece](https://github.com/google/sentencepiece).

    This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should
    refer to this superclass for more information regarding those methods.

    Args:
        vocab_file (`str`):
            [SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
            contains the vocabulary necessary to instantiate a tokenizer.
        bos_token (`str`, *optional*, defaults to `"<s>"`):
            The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.

            <Tip>

            When building a sequence using special tokens, this is not the token that is used for the beginning of
            sequence. The token used is the `cls_token`.

            </Tip>

        eos_token (`str`, *optional*, defaults to `"</s>"`):
            The end of sequence token.

            <Tip>

            When building a sequence using special tokens, this is not the token that is used for the end of sequence.
            The token used is the `sep_token`.

            </Tip>

        sep_token (`str`, *optional*):
            The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
            sequence classification or for a text and a question for question answering. It is also used as the last
            token of a sequence built with special tokens. defaults to `"</s>"`
        cls_token (`str`, *optional*):
            The classifier token which is used when doing sequence classification (classification of the whole sequence
            instead of per-token classification). It is the first token of the sequence when built with special tokens.
            defaults to `"<s>"`
        unk_token (`str`, *optional*):
            The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
            token instead. defaults to `"<unk>"`
        pad_token (`str`, *optional*):
            The token used for padding, for example when batching sequences of different lengths. defaults to `"<pad>"`
        mask_token (`str`, *optional*):
            The token used for masking values. This is the token used when training this model with masked language
            modeling. This is the token which the model will try to predict. defaults to `"<mask>"`
        additional_special_tokens (`List[str]`, *optional*):
            Additional special tokens used by the tokenizer. defaults to `["<s>NOTUSED", "</s>NOTUSED"]`
    """
    vocab_files_names = VOCAB_FILES_NAMES
    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
    model_input_names = ["input_ids", "attention_mask"]
    slow_tokenizer_class = BarthezTokenizer

    def __init__(
        self,
        vocab_file=None,
        tokenizer_file=None,
        bos_token="<s>",
        eos_token="</s>",
        sep_token="</s>",
        cls_token="<s>",
        unk_token="<unk>",
        pad_token="<pad>",
        mask_token="<mask>",
        **kwargs,
    ):
        """
        Initialize a BarthezTokenizerFast object.

        Args:
            vocab_file (str): Path to the vocabulary file. Default is None.
            tokenizer_file (str): Path to the tokenizer file. Default is None.
            bos_token (str): Beginning of sentence token. Default is '<s>'.
            eos_token (str): End of sentence token. Default is '</s>'.
            sep_token (str): Separator token. Default is '</s>'.
            cls_token (str): Classification token. Default is '<s>'.
            unk_token (str): Token for unknown words. Default is '<unk>'.
            pad_token (str): Padding token. Default is '<pad>'.
            mask_token (str): Mask token. Default is '<mask>'.

        Returns:
            None.

        Raises:
            TypeError: If mask_token is not a string.
        """
        # Mask token behave like a normal word, i.e. include the space before it
        mask_token = AddedToken(mask_token, lstrip=True, rstrip=False) if isinstance(mask_token, str) else mask_token

        super().__init__(
            vocab_file,
            tokenizer_file=tokenizer_file,
            bos_token=bos_token,
            eos_token=eos_token,
            unk_token=unk_token,
            sep_token=sep_token,
            cls_token=cls_token,
            pad_token=pad_token,
            mask_token=mask_token,
            **kwargs,
        )

        self.vocab_file = vocab_file

    @property
    def can_save_slow_tokenizer(self) -> bool:
        """
        Method to check if the slow tokenizer can be saved.

        Args:
            self (BarthezTokenizerFast): An instance of the BarthezTokenizerFast class.
                Represents the current object to check whether the slow tokenizer can be saved.

        Returns:
            bool: Returns a boolean value indicating whether the slow tokenizer can be saved.
                True if the vocab file exists, False if the vocab file does not exist or is not provided.

        Raises:
            None.
        """
        return os.path.isfile(self.vocab_file) if self.vocab_file else False

    def build_inputs_with_special_tokens(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
    ) -> List[int]:
        """
        Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
        adding special tokens. A BARThez sequence has the following format:

        - single sequence: `<s> X </s>`
        - pair of sequences: `<s> A </s></s> B </s>`

        Args:
            token_ids_0 (`List[int]`):
                List of IDs to which the special tokens will be added.
            token_ids_1 (`List[int]`, *optional*):
                Optional second list of IDs for sequence pairs.

        Returns:
            `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
        """
        if token_ids_1 is None:
            return [self.cls_token_id] + token_ids_0 + [self.sep_token_id]
        cls = [self.cls_token_id]
        sep = [self.sep_token_id]
        return cls + token_ids_0 + sep + sep + token_ids_1 + sep

    def create_token_type_ids_from_sequences(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
    ) -> List[int]:
        """
        Create a mask from the two sequences passed to be used in a sequence-pair classification task.

        Args:
            token_ids_0 (`List[int]`):
                List of IDs.
            token_ids_1 (`List[int]`, *optional*):
                Optional second list of IDs for sequence pairs.

        Returns:
            `List[int]`: List of zeros.
        """
        sep = [self.sep_token_id]
        cls = [self.cls_token_id]

        if token_ids_1 is None:
            return len(cls + token_ids_0 + sep) * [0]
        return len(cls + token_ids_0 + sep + sep + token_ids_1 + sep) * [0]

    def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
        """
        Save the vocabulary for a slow tokenizer.

        Args:
            self (BarthezTokenizerFast): The instance of the BarthezTokenizerFast class.
            save_directory (str): The directory where the vocabulary will be saved.
            filename_prefix (Optional[str], optional): The prefix to be added to the filename (default: None).

        Returns:
            Tuple[str]: A tuple containing the path to the saved vocabulary file.

        Raises:
            ValueError: If the fast tokenizer does not have the necessary information to save the vocabulary for a slow tokenizer.
            OSError: If the provided save_directory is not a valid directory.
            IOError: If there is an error while copying the vocabulary file.

        Note:
            - The fast tokenizer must have the necessary information to save the vocabulary for a slow tokenizer.
            - The save_directory should be a valid directory.
            - The vocabulary file will be copied to the save_directory with an optional filename_prefix.

        Example:
            ```python
            >>> tokenizer = BarthezTokenizerFast()
            >>> tokenizer.save_vocabulary('/path/to/save')
            ('/path/to/save/vocab.txt', )
            ```

        """
        if not self.can_save_slow_tokenizer:
            raise ValueError(
                "Your fast tokenizer does not have the necessary information to save the vocabulary for a slow "
                "tokenizer."
            )

        if not os.path.isdir(save_directory):
            logger.error(f"Vocabulary path ({save_directory}) should be a directory")
            return
        out_vocab_file = os.path.join(
            save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
        )

        if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file):
            copyfile(self.vocab_file, out_vocab_file)

        return (out_vocab_file,)

mindnlp.transformers.models.barthez.tokenization_barthez_fast.BarthezTokenizerFast.can_save_slow_tokenizer: bool property

Method to check if the slow tokenizer can be saved.

PARAMETER	DESCRIPTION
self	An instance of the BarthezTokenizerFast class. Represents the current object to check whether the slow tokenizer can be saved. TYPE: BarthezTokenizerFast

RETURNS	DESCRIPTION
bool	Returns a boolean value indicating whether the slow tokenizer can be saved. True if the vocab file exists, False if the vocab file does not exist or is not provided. TYPE: bool

mindnlp.transformers.models.barthez.tokenization_barthez_fast.BarthezTokenizerFast.__init__(vocab_file=None, tokenizer_file=None, bos_token='<s>', eos_token='</s>', sep_token='</s>', cls_token='<s>', unk_token='<unk>', pad_token='<pad>', mask_token='<mask>', **kwargs)

Initialize a BarthezTokenizerFast object.

PARAMETER	DESCRIPTION
vocab_file	Path to the vocabulary file. Default is None. TYPE: str DEFAULT: None
tokenizer_file	Path to the tokenizer file. Default is None. TYPE: str DEFAULT: None
bos_token	Beginning of sentence token. Default is ''. TYPE: str DEFAULT: '<s>'
eos_token	End of sentence token. Default is ''. TYPE: str DEFAULT: '</s>'
sep_token	Separator token. Default is ''. TYPE: str DEFAULT: '</s>'
cls_token	Classification token. Default is ''. TYPE: str DEFAULT: '<s>'
unk_token	Token for unknown words. Default is ''. TYPE: str DEFAULT: '<unk>'
pad_token	Padding token. Default is ''. TYPE: str DEFAULT: '<pad>'
mask_token	Mask token. Default is ''. TYPE: str DEFAULT: '<mask>'

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
TypeError	If mask_token is not a string.

Source code in mindnlp/transformers/models/barthez/tokenization_barthez_fast.py

def __init__(
    self,
    vocab_file=None,
    tokenizer_file=None,
    bos_token="<s>",
    eos_token="</s>",
    sep_token="</s>",
    cls_token="<s>",
    unk_token="<unk>",
    pad_token="<pad>",
    mask_token="<mask>",
    **kwargs,
):
    """
    Initialize a BarthezTokenizerFast object.

    Args:
        vocab_file (str): Path to the vocabulary file. Default is None.
        tokenizer_file (str): Path to the tokenizer file. Default is None.
        bos_token (str): Beginning of sentence token. Default is '<s>'.
        eos_token (str): End of sentence token. Default is '</s>'.
        sep_token (str): Separator token. Default is '</s>'.
        cls_token (str): Classification token. Default is '<s>'.
        unk_token (str): Token for unknown words. Default is '<unk>'.
        pad_token (str): Padding token. Default is '<pad>'.
        mask_token (str): Mask token. Default is '<mask>'.

    Returns:
        None.

    Raises:
        TypeError: If mask_token is not a string.
    """
    # Mask token behave like a normal word, i.e. include the space before it
    mask_token = AddedToken(mask_token, lstrip=True, rstrip=False) if isinstance(mask_token, str) else mask_token

    super().__init__(
        vocab_file,
        tokenizer_file=tokenizer_file,
        bos_token=bos_token,
        eos_token=eos_token,
        unk_token=unk_token,
        sep_token=sep_token,
        cls_token=cls_token,
        pad_token=pad_token,
        mask_token=mask_token,
        **kwargs,
    )

    self.vocab_file = vocab_file

mindnlp.transformers.models.barthez.tokenization_barthez_fast.BarthezTokenizerFast.build_inputs_with_special_tokens(token_ids_0, token_ids_1=None)

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BARThez sequence has the following format:

• single sequence: <s> X </s>
• pair of sequences: <s> A </s></s> B </s>

PARAMETER	DESCRIPTION
token_ids_0	List of IDs to which the special tokens will be added. TYPE: `List[int]`
token_ids_1	Optional second list of IDs for sequence pairs. TYPE: `List[int]`, *optional* DEFAULT: None

RETURNS	DESCRIPTION
List[int]	List[int]: List of input IDs with the appropriate special tokens.

Source code in mindnlp/transformers/models/barthez/tokenization_barthez_fast.py

def build_inputs_with_special_tokens(
    self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
) -> List[int]:
    """
    Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
    adding special tokens. A BARThez sequence has the following format:

    - single sequence: `<s> X </s>`
    - pair of sequences: `<s> A </s></s> B </s>`

    Args:
        token_ids_0 (`List[int]`):
            List of IDs to which the special tokens will be added.
        token_ids_1 (`List[int]`, *optional*):
            Optional second list of IDs for sequence pairs.

    Returns:
        `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
    """
    if token_ids_1 is None:
        return [self.cls_token_id] + token_ids_0 + [self.sep_token_id]
    cls = [self.cls_token_id]
    sep = [self.sep_token_id]
    return cls + token_ids_0 + sep + sep + token_ids_1 + sep

mindnlp.transformers.models.barthez.tokenization_barthez_fast.BarthezTokenizerFast.create_token_type_ids_from_sequences(token_ids_0, token_ids_1=None)

Create a mask from the two sequences passed to be used in a sequence-pair classification task.

PARAMETER	DESCRIPTION
token_ids_0	List of IDs. TYPE: `List[int]`
token_ids_1	Optional second list of IDs for sequence pairs. TYPE: `List[int]`, *optional* DEFAULT: None

RETURNS	DESCRIPTION
List[int]	List[int]: List of zeros.

Source code in mindnlp/transformers/models/barthez/tokenization_barthez_fast.py

def create_token_type_ids_from_sequences(
    self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
) -> List[int]:
    """
    Create a mask from the two sequences passed to be used in a sequence-pair classification task.

    Args:
        token_ids_0 (`List[int]`):
            List of IDs.
        token_ids_1 (`List[int]`, *optional*):
            Optional second list of IDs for sequence pairs.

    Returns:
        `List[int]`: List of zeros.
    """
    sep = [self.sep_token_id]
    cls = [self.cls_token_id]

    if token_ids_1 is None:
        return len(cls + token_ids_0 + sep) * [0]
    return len(cls + token_ids_0 + sep + sep + token_ids_1 + sep) * [0]

mindnlp.transformers.models.barthez.tokenization_barthez_fast.BarthezTokenizerFast.save_vocabulary(save_directory, filename_prefix=None)

Save the vocabulary for a slow tokenizer.

PARAMETER	DESCRIPTION
self	The instance of the BarthezTokenizerFast class. TYPE: BarthezTokenizerFast
save_directory	The directory where the vocabulary will be saved. TYPE: str
filename_prefix	The prefix to be added to the filename (default: None). TYPE: Optional[str] DEFAULT: None

RETURNS	DESCRIPTION
Tuple[str]	Tuple[str]: A tuple containing the path to the saved vocabulary file.

RAISES	DESCRIPTION
ValueError	If the fast tokenizer does not have the necessary information to save the vocabulary for a slow tokenizer.
OSError	If the provided save_directory is not a valid directory.
IOError	If there is an error while copying the vocabulary file.

Note

• The fast tokenizer must have the necessary information to save the vocabulary for a slow tokenizer.
• The save_directory should be a valid directory.
• The vocabulary file will be copied to the save_directory with an optional filename_prefix.

Example

>>> tokenizer = BarthezTokenizerFast()
>>> tokenizer.save_vocabulary('/path/to/save')
('/path/to/save/vocab.txt', )

Source code in mindnlp/transformers/models/barthez/tokenization_barthez_fast.py

def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
    """
    Save the vocabulary for a slow tokenizer.

    Args:
        self (BarthezTokenizerFast): The instance of the BarthezTokenizerFast class.
        save_directory (str): The directory where the vocabulary will be saved.
        filename_prefix (Optional[str], optional): The prefix to be added to the filename (default: None).

    Returns:
        Tuple[str]: A tuple containing the path to the saved vocabulary file.

    Raises:
        ValueError: If the fast tokenizer does not have the necessary information to save the vocabulary for a slow tokenizer.
        OSError: If the provided save_directory is not a valid directory.
        IOError: If there is an error while copying the vocabulary file.

    Note:
        - The fast tokenizer must have the necessary information to save the vocabulary for a slow tokenizer.
        - The save_directory should be a valid directory.
        - The vocabulary file will be copied to the save_directory with an optional filename_prefix.

    Example:
        ```python
        >>> tokenizer = BarthezTokenizerFast()
        >>> tokenizer.save_vocabulary('/path/to/save')
        ('/path/to/save/vocab.txt', )
        ```

    """
    if not self.can_save_slow_tokenizer:
        raise ValueError(
            "Your fast tokenizer does not have the necessary information to save the vocabulary for a slow "
            "tokenizer."
        )

    if not os.path.isdir(save_directory):
        logger.error(f"Vocabulary path ({save_directory}) should be a directory")
        return
    out_vocab_file = os.path.join(
        save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
    )

    if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file):
        copyfile(self.vocab_file, out_vocab_file)

    return (out_vocab_file,)