bark

mindnlp.transformers.models.bark.configuration_bark.BARK_PRETRAINED_CONFIG_ARCHIVE_MAP = {'suno/bark-small': 'https://hf-mirror.com/suno/bark-small/resolve/main/config.json', 'suno/bark': 'https://hf-mirror.com/suno/bark/resolve/main/config.json'} module-attribute

mindnlp.transformers.models.bark.configuration_bark.BarkCoarseConfig

Bases: BarkSubModelConfig

BarkCoarseConfig is a Python class that represents the configuration settings for the coarse behavior model in the Bark autonomous driving simulation framework. This class inherits from the BarkSubModelConfig class.

The BarkCoarseConfig class provides a set of parameters and options that can be used to configure the behavior of the coarse model. These parameters include settings related to the behavior model itself, such as the desired velocity, acceleration limits, and time horizons, as well as settings for the perception model, such as sensor range and field of view.

ATTRIBUTE	DESCRIPTION
desired_velocity	The desired velocity of the ego vehicle. TYPE: float
max_acceleration	The maximum acceleration limit for the ego vehicle. TYPE: float
min_acceleration	The minimum acceleration limit for the ego vehicle. TYPE: float
horizon_time	The time horizon for the behavior planning. TYPE: float
perception_range	The range of the perception sensor. TYPE: float
field_of_view	The field of view of the perception sensor. TYPE: float

METHOD	DESCRIPTION
__init__	Initializes a new instance of the BarkCoarseConfig class with the specified parameters.
get_desired_velocity	Returns the desired velocity of the ego vehicle.
set_desired_velocity	Sets the desired velocity of the ego vehicle.
get_max_acceleration	Returns the maximum acceleration limit for the ego vehicle.
set_max_acceleration	Sets the maximum acceleration limit for the ego vehicle.
get_min_acceleration	Returns the minimum acceleration limit for the ego vehicle.
set_min_acceleration	Sets the minimum acceleration limit for the ego vehicle.
get_horizon_time	Returns the time horizon for the behavior planning.
set_horizon_time	Sets the time horizon for the behavior planning.
get_perception_range	Returns the range of the perception sensor.
set_perception_range	Sets the range of the perception sensor.
get_field_of_view	Returns the field of view of the perception sensor.
set_field_of_view	Sets the field of view of the perception sensor.

Source code in mindnlp/transformers/models/bark/configuration_bark.py

class BarkCoarseConfig(BarkSubModelConfig):

    """
    BarkCoarseConfig is a Python class that represents the configuration settings for the coarse behavior model
    in the Bark autonomous driving simulation framework.
    This class inherits from the BarkSubModelConfig class.

    The BarkCoarseConfig class provides a set of parameters and options that can be used to configure the behavior of the coarse model.
    These parameters include settings related to the behavior model itself, such as the desired velocity,
    acceleration limits, and time horizons, as well as settings for the perception model,
    such as sensor range and field of view.

    Attributes:
        desired_velocity (float): The desired velocity of the ego vehicle.
        max_acceleration (float): The maximum acceleration limit for the ego vehicle.
        min_acceleration (float): The minimum acceleration limit for the ego vehicle.
        horizon_time (float): The time horizon for the behavior planning.
        perception_range (float): The range of the perception sensor.
        field_of_view (float): The field of view of the perception sensor.

    Methods:
        __init__(self, desired_velocity, max_acceleration, min_acceleration, horizon_time, perception_range, field_of_view):
            Initializes a new instance of the BarkCoarseConfig class with the specified parameters.
        get_desired_velocity(self): Returns the desired velocity of the ego vehicle.
        set_desired_velocity(self, desired_velocity): Sets the desired velocity of the ego vehicle.
        get_max_acceleration(self): Returns the maximum acceleration limit for the ego vehicle.
        set_max_acceleration(self, max_acceleration): Sets the maximum acceleration limit for the ego vehicle.
        get_min_acceleration(self): Returns the minimum acceleration limit for the ego vehicle.
        set_min_acceleration(self, min_acceleration): Sets the minimum acceleration limit for the ego vehicle.
        get_horizon_time(self): Returns the time horizon for the behavior planning.
        set_horizon_time(self, horizon_time): Sets the time horizon for the behavior planning.
        get_perception_range(self): Returns the range of the perception sensor.
        set_perception_range(self, perception_range): Sets the range of the perception sensor.
        get_field_of_view(self): Returns the field of view of the perception sensor.
        set_field_of_view(self, field_of_view): Sets the field of view of the perception sensor.

    """
    model_type = "coarse_acoustics"

mindnlp.transformers.models.bark.configuration_bark.BarkConfig

Bases: PretrainedConfig

This is the configuration class to store the configuration of a [BarkModel]. It is used to instantiate a Bark model according to the specified sub-models configurations, defining the model architecture.

Instantiating a configuration with the defaults will yield a similar configuration to that of the Bark suno/bark architecture.

Configuration objects inherit from [PretrainedConfig] and can be used to control the model outputs. Read the documentation from [PretrainedConfig] for more information.

PARAMETER	DESCRIPTION
semantic_config	Configuration of the underlying semantic sub-model. TYPE: [`BarkSemanticConfig`], *optional* DEFAULT: None
coarse_acoustics_config	Configuration of the underlying coarse acoustics sub-model. TYPE: [`BarkCoarseConfig`], *optional* DEFAULT: None
fine_acoustics_config	Configuration of the underlying fine acoustics sub-model. TYPE: [`BarkFineConfig`], *optional* DEFAULT: None
codec_config	Configuration of the underlying codec sub-model. TYPE: [`AutoConfig`], *optional* DEFAULT: None

Example

>>> from transformers import (
...     BarkSemanticConfig,
...     BarkCoarseConfig,
...     BarkFineConfig,
...     BarkModel,
...     BarkConfig,
...     AutoConfig,
... )
...
>>> # Initializing Bark sub-modules configurations.
>>> semantic_config = BarkSemanticConfig()
>>> coarse_acoustics_config = BarkCoarseConfig()
>>> fine_acoustics_config = BarkFineConfig()
>>> codec_config = AutoConfig.from_pretrained("facebook/encodec_24khz")
...
...
>>> # Initializing a Bark module style configuration
>>> configuration = BarkConfig.from_sub_model_configs(
...     semantic_config, coarse_acoustics_config, fine_acoustics_config, codec_config
... )
...
>>> # Initializing a model (with random weights)
>>> model = BarkModel(configuration)
...
>>> # Accessing the model configuration
>>> configuration = model.config

Source code in mindnlp/transformers/models/bark/configuration_bark.py

class BarkConfig(PretrainedConfig):
    """
    This is the configuration class to store the configuration of a [`BarkModel`]. It is used to instantiate a Bark
    model according to the specified sub-models configurations, defining the model architecture.

    Instantiating a configuration with the defaults will yield a similar configuration to that of the Bark
    [suno/bark](https://hf-mirror.com/suno/bark) architecture.

    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
    documentation from [`PretrainedConfig`] for more information.

    Args:
        semantic_config ([`BarkSemanticConfig`], *optional*):
            Configuration of the underlying semantic sub-model.
        coarse_acoustics_config ([`BarkCoarseConfig`], *optional*):
            Configuration of the underlying coarse acoustics sub-model.
        fine_acoustics_config ([`BarkFineConfig`], *optional*):
            Configuration of the underlying fine acoustics sub-model.
        codec_config ([`AutoConfig`], *optional*):
            Configuration of the underlying codec sub-model.

    Example:
        ```python
        >>> from transformers import (
        ...     BarkSemanticConfig,
        ...     BarkCoarseConfig,
        ...     BarkFineConfig,
        ...     BarkModel,
        ...     BarkConfig,
        ...     AutoConfig,
        ... )
        ...
        >>> # Initializing Bark sub-modules configurations.
        >>> semantic_config = BarkSemanticConfig()
        >>> coarse_acoustics_config = BarkCoarseConfig()
        >>> fine_acoustics_config = BarkFineConfig()
        >>> codec_config = AutoConfig.from_pretrained("facebook/encodec_24khz")
        ...
        ...
        >>> # Initializing a Bark module style configuration
        >>> configuration = BarkConfig.from_sub_model_configs(
        ...     semantic_config, coarse_acoustics_config, fine_acoustics_config, codec_config
        ... )
        ...
        >>> # Initializing a model (with random weights)
        >>> model = BarkModel(configuration)
        ...
        >>> # Accessing the model configuration
        >>> configuration = model.config
        ```
    """
    model_type = "bark"

    def __init__(
        self,
        semantic_config: Dict = None,
        coarse_acoustics_config: Dict = None,
        fine_acoustics_config: Dict = None,
        codec_config: Dict = None,
        initializer_range=0.02,
        **kwargs,
    ):
        """
        Initializes a BarkConfig object with the provided configurations.

        Args:
            self: The instance of the BarkConfig class.
            semantic_config (Dict): Dictionary containing configuration for the semantic model. Defaults to None.
            coarse_acoustics_config (Dict): Dictionary containing configuration for the coarse acoustics model. Defaults to None.
            fine_acoustics_config (Dict): Dictionary containing configuration for the fine acoustics model. Defaults to None.
            codec_config (Dict): Dictionary containing configuration for the codec model. Defaults to None.
            initializer_range (float): Range for weight initialization. Defaults to 0.02.

        Returns:
            None.

        Raises:
            None
        """
        if semantic_config is None:
            semantic_config = {}
            logger.info("semantic_config is None. initializing the semantic model with default values.")

        if coarse_acoustics_config is None:
            coarse_acoustics_config = {}
            logger.info("coarse_acoustics_config is None. initializing the coarse model with default values.")

        if fine_acoustics_config is None:
            fine_acoustics_config = {}
            logger.info("fine_acoustics_config is None. initializing the fine model with default values.")

        if codec_config is None:
            codec_config = {}
            logger.info("codec_config is None. initializing the codec model with default values.")

        self.semantic_config = BarkSemanticConfig(**semantic_config)
        self.coarse_acoustics_config = BarkCoarseConfig(**coarse_acoustics_config)
        self.fine_acoustics_config = BarkFineConfig(**fine_acoustics_config)
        codec_model_type = codec_config["model_type"] if "model_type" in codec_config else "encodec"
        self.codec_config = CONFIG_MAPPING[codec_model_type](**codec_config)

        self.initializer_range = initializer_range

        super().__init__(**kwargs)

    @classmethod
    def from_sub_model_configs(
        cls,
        semantic_config: BarkSemanticConfig,
        coarse_acoustics_config: BarkCoarseConfig,
        fine_acoustics_config: BarkFineConfig,
        codec_config: PretrainedConfig,
        **kwargs,
    ):
        r"""
        Instantiate a [`BarkConfig`] (or a derived class) from bark sub-models configuration.

        Returns:
            [`BarkConfig`]: An instance of a configuration object
        """
        return cls(
            semantic_config=semantic_config.to_dict(),
            coarse_acoustics_config=coarse_acoustics_config.to_dict(),
            fine_acoustics_config=fine_acoustics_config.to_dict(),
            codec_config=codec_config.to_dict(),
            **kwargs,
        )

mindnlp.transformers.models.bark.configuration_bark.BarkConfig.__init__(semantic_config=None, coarse_acoustics_config=None, fine_acoustics_config=None, codec_config=None, initializer_range=0.02, **kwargs)

Initializes a BarkConfig object with the provided configurations.

PARAMETER	DESCRIPTION
self	The instance of the BarkConfig class.
semantic_config	Dictionary containing configuration for the semantic model. Defaults to None. TYPE: Dict DEFAULT: None
coarse_acoustics_config	Dictionary containing configuration for the coarse acoustics model. Defaults to None. TYPE: Dict DEFAULT: None
fine_acoustics_config	Dictionary containing configuration for the fine acoustics model. Defaults to None. TYPE: Dict DEFAULT: None
codec_config	Dictionary containing configuration for the codec model. Defaults to None. TYPE: Dict DEFAULT: None
initializer_range	Range for weight initialization. Defaults to 0.02. TYPE: float DEFAULT: 0.02

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/bark/configuration_bark.py

def __init__(
    self,
    semantic_config: Dict = None,
    coarse_acoustics_config: Dict = None,
    fine_acoustics_config: Dict = None,
    codec_config: Dict = None,
    initializer_range=0.02,
    **kwargs,
):
    """
    Initializes a BarkConfig object with the provided configurations.

    Args:
        self: The instance of the BarkConfig class.
        semantic_config (Dict): Dictionary containing configuration for the semantic model. Defaults to None.
        coarse_acoustics_config (Dict): Dictionary containing configuration for the coarse acoustics model. Defaults to None.
        fine_acoustics_config (Dict): Dictionary containing configuration for the fine acoustics model. Defaults to None.
        codec_config (Dict): Dictionary containing configuration for the codec model. Defaults to None.
        initializer_range (float): Range for weight initialization. Defaults to 0.02.

    Returns:
        None.

    Raises:
        None
    """
    if semantic_config is None:
        semantic_config = {}
        logger.info("semantic_config is None. initializing the semantic model with default values.")

    if coarse_acoustics_config is None:
        coarse_acoustics_config = {}
        logger.info("coarse_acoustics_config is None. initializing the coarse model with default values.")

    if fine_acoustics_config is None:
        fine_acoustics_config = {}
        logger.info("fine_acoustics_config is None. initializing the fine model with default values.")

    if codec_config is None:
        codec_config = {}
        logger.info("codec_config is None. initializing the codec model with default values.")

    self.semantic_config = BarkSemanticConfig(**semantic_config)
    self.coarse_acoustics_config = BarkCoarseConfig(**coarse_acoustics_config)
    self.fine_acoustics_config = BarkFineConfig(**fine_acoustics_config)
    codec_model_type = codec_config["model_type"] if "model_type" in codec_config else "encodec"
    self.codec_config = CONFIG_MAPPING[codec_model_type](**codec_config)

    self.initializer_range = initializer_range

    super().__init__(**kwargs)

mindnlp.transformers.models.bark.configuration_bark.BarkConfig.from_sub_model_configs(semantic_config, coarse_acoustics_config, fine_acoustics_config, codec_config, **kwargs) classmethod

Instantiate a [BarkConfig] (or a derived class) from bark sub-models configuration.

RETURNS	DESCRIPTION
	[BarkConfig]: An instance of a configuration object

Source code in mindnlp/transformers/models/bark/configuration_bark.py

@classmethod
def from_sub_model_configs(
    cls,
    semantic_config: BarkSemanticConfig,
    coarse_acoustics_config: BarkCoarseConfig,
    fine_acoustics_config: BarkFineConfig,
    codec_config: PretrainedConfig,
    **kwargs,
):
    r"""
    Instantiate a [`BarkConfig`] (or a derived class) from bark sub-models configuration.

    Returns:
        [`BarkConfig`]: An instance of a configuration object
    """
    return cls(
        semantic_config=semantic_config.to_dict(),
        coarse_acoustics_config=coarse_acoustics_config.to_dict(),
        fine_acoustics_config=fine_acoustics_config.to_dict(),
        codec_config=codec_config.to_dict(),
        **kwargs,
    )

mindnlp.transformers.models.bark.configuration_bark.BarkFineConfig

Bases: BarkSubModelConfig

BarkFineConfig represents the configuration settings for a fine-tuning model within the Bark framework. This class inherits from BarkSubModelConfig and provides parameters for configuring the fine-tuning process, including options for tying word embeddings, specifying the total number of codes, and the number of codes given.

PARAMETER	DESCRIPTION
tie_word_embeddings	Flag indicating whether to tie word embeddings during fine-tuning. TYPE: bool DEFAULT: True
n_codes_total	The total number of codes used in the fine-tuning model. TYPE: int DEFAULT: 8
n_codes_given	The number of codes given as input to the fine-tuning model. TYPE: int DEFAULT: 1

Inherits from BarkSubModelConfig and initializes the configuration settings for the fine-tuning model based on the provided parameters.

Source code in mindnlp/transformers/models/bark/configuration_bark.py

class BarkFineConfig(BarkSubModelConfig):

    """
    BarkFineConfig represents the configuration settings for a fine-tuning model within the Bark framework.
    This class inherits from BarkSubModelConfig and provides parameters for configuring the fine-tuning process,
    including options for tying word embeddings, specifying the total number of codes, and the number of codes given.

    Parameters:
        tie_word_embeddings (bool): Flag indicating whether to tie word embeddings during fine-tuning.
        n_codes_total (int): The total number of codes used in the fine-tuning model.
        n_codes_given (int): The number of codes given as input to the fine-tuning model.

    Inherits from BarkSubModelConfig and initializes the configuration settings for the fine-tuning model based on the provided parameters.
    """
    model_type = "fine_acoustics"

    def __init__(self, tie_word_embeddings=True, n_codes_total=8, n_codes_given=1, **kwargs):
        """Initializes a new instance of the BarkFineConfig class.

        Args:
            self (BarkFineConfig): The object instance.
            tie_word_embeddings (bool): Whether to tie the word embeddings of the model. Defaults to True.
            n_codes_total (int): The total number of codes. Defaults to 8.
            n_codes_given (int): The number of given codes. Defaults to 1.

        Returns:
            None.

        Raises:
            None.
        """
        self.n_codes_total = n_codes_total
        self.n_codes_given = n_codes_given

        super().__init__(tie_word_embeddings=tie_word_embeddings, **kwargs)

mindnlp.transformers.models.bark.configuration_bark.BarkFineConfig.__init__(tie_word_embeddings=True, n_codes_total=8, n_codes_given=1, **kwargs)

Initializes a new instance of the BarkFineConfig class.

PARAMETER	DESCRIPTION
self	The object instance. TYPE: BarkFineConfig
tie_word_embeddings	Whether to tie the word embeddings of the model. Defaults to True. TYPE: bool DEFAULT: True
n_codes_total	The total number of codes. Defaults to 8. TYPE: int DEFAULT: 8
n_codes_given	The number of given codes. Defaults to 1. TYPE: int DEFAULT: 1

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/bark/configuration_bark.py

def __init__(self, tie_word_embeddings=True, n_codes_total=8, n_codes_given=1, **kwargs):
    """Initializes a new instance of the BarkFineConfig class.

    Args:
        self (BarkFineConfig): The object instance.
        tie_word_embeddings (bool): Whether to tie the word embeddings of the model. Defaults to True.
        n_codes_total (int): The total number of codes. Defaults to 8.
        n_codes_given (int): The number of given codes. Defaults to 1.

    Returns:
        None.

    Raises:
        None.
    """
    self.n_codes_total = n_codes_total
    self.n_codes_given = n_codes_given

    super().__init__(tie_word_embeddings=tie_word_embeddings, **kwargs)

mindnlp.transformers.models.bark.configuration_bark.BarkSemanticConfig

Bases: BarkSubModelConfig

Represents a configuration class for semantic segmentation models in the Bark framework. This class inherits properties and methods from the BarkSubModelConfig class.

ATTRIBUTE	DESCRIPTION
model_name	The name of the semantic segmentation model. TYPE: str
num_classes	The number of classes in the semantic segmentation task. TYPE: int
input_shape	The input shape of the model in the format (height, width, channels). TYPE: tuple
backbone	The backbone architecture used in the model. TYPE: str
pretrained_backbone	Indicates if a pretrained backbone is used. TYPE: bool
normalization	The type of normalization applied to the input data. TYPE: str
loss_function	The loss function used for training the model. TYPE: str
optimizer	The optimizer used during model training. TYPE: str

METHOD	DESCRIPTION
set_model_name	Sets the name of the semantic segmentation model.
set_num_classes	Sets the number of classes in the semantic segmentation task.
set_input_shape	Sets the input shape of the model.
set_backbone	Sets the backbone architecture used in the model.
set_pretrained_backbone	Sets whether a pretrained backbone is used.
set_normalization	Sets the type of normalization applied to the input data.
set_loss_function	Sets the loss function used for training the model.
set_optimizer	Sets the optimizer used during model training.

Source code in mindnlp/transformers/models/bark/configuration_bark.py

class BarkSemanticConfig(BarkSubModelConfig):

    """
    Represents a configuration class for semantic segmentation models in the Bark framework.
    This class inherits properties and methods from the BarkSubModelConfig class.

    Attributes:
        model_name (str): The name of the semantic segmentation model.
        num_classes (int): The number of classes in the semantic segmentation task.
        input_shape (tuple): The input shape of the model in the format (height, width, channels).
        backbone (str): The backbone architecture used in the model.
        pretrained_backbone (bool): Indicates if a pretrained backbone is used.
        normalization (str): The type of normalization applied to the input data.
        loss_function (str): The loss function used for training the model.
        optimizer (str): The optimizer used during model training.

    Methods:
        set_model_name:
            Sets the name of the semantic segmentation model.

        set_num_classes:
            Sets the number of classes in the semantic segmentation task.

        set_input_shape:
            Sets the input shape of the model.

        set_backbone:
            Sets the backbone architecture used in the model.

        set_pretrained_backbone:
            Sets whether a pretrained backbone is used.

        set_normalization:
            Sets the type of normalization applied to the input data.

        set_loss_function:
            Sets the loss function used for training the model.

        set_optimizer:
            Sets the optimizer used during model training.
    """
    model_type = "semantic"

mindnlp.transformers.models.bark.generation_configuration_bark.BarkGenerationConfig

Bases: GenerationConfig

Class that represents a generation configuration for BarkModel.

This class is used to generate speeches with a nested BarkGenerationConfig, which includes BarkSemanticGenerationConfig, BarkCoarseGenerationConfig, and BarkFineGenerationConfig. It inherits from GenerationConfig and allows control over the model generation.

ATTRIBUTE	DESCRIPTION
semantic_config	Semantic generation configuration. TYPE: BarkSemanticGenerationConfig
coarse_acoustics_config	Coarse generation configuration. TYPE: BarkCoarseGenerationConfig
fine_acoustics_config	Fine generation configuration. TYPE: BarkFineGenerationConfig
sample_rate	Sample rate for the generation process. TYPE: int
codebook_size	Vector length for each codebook. TYPE: int

METHOD	DESCRIPTION
__init__	Initializes the BarkGenerationConfig instance with provided configurations or defaults.
from_sub_model_configs	Instantiates a BarkGenerationConfig from Bark sub-models generation configurations.
to_dict	Serializes the instance to a Python dictionary, including all configuration attributes.

Source code in mindnlp/transformers/models/bark/generation_configuration_bark.py

class BarkGenerationConfig(GenerationConfig):

    """
    Class that represents a generation configuration for BarkModel.

    This class is used to generate speeches with a nested BarkGenerationConfig, which includes
    BarkSemanticGenerationConfig, BarkCoarseGenerationConfig, and BarkFineGenerationConfig.
    It inherits from GenerationConfig and allows control over the model generation.

    Attributes:
        semantic_config (BarkSemanticGenerationConfig): Semantic generation configuration.
        coarse_acoustics_config (BarkCoarseGenerationConfig): Coarse generation configuration.
        fine_acoustics_config (BarkFineGenerationConfig): Fine generation configuration.
        sample_rate (int): Sample rate for the generation process.
        codebook_size (int): Vector length for each codebook.

    Methods:
        __init__: Initializes the BarkGenerationConfig instance with provided configurations or defaults.
        from_sub_model_configs: Instantiates a BarkGenerationConfig from Bark sub-models generation configurations.
        to_dict: Serializes the instance to a Python dictionary, including all configuration attributes.
    """
    model_type = "bark"
    is_composition = True

    def __init__(
        self,
        semantic_config: Dict = None,
        coarse_acoustics_config: Dict = None,
        fine_acoustics_config: Dict = None,
        sample_rate=24_000,
        codebook_size=1024,
        **kwargs,
    ):
        """Class that holds a generation configuration for [`BarkModel`].

        The [`BarkModel`] does not have a `generate` method, but uses this class to generate speeches with a nested
        [`BarkGenerationConfig`] which uses [`BarkSemanticGenerationConfig`], [`BarkCoarseGenerationConfig`],
        [`BarkFineGenerationConfig`].

        This configuration inherit from [`GenerationConfig`] and can be used to control the model generation. Read the
        documentation from [`GenerationConfig`] for more information.

        Args:
            semantic_config (`Dict`, *optional*):
                Semantic generation configuration.
            coarse_acoustics_config (`Dict`, *optional*):
                Coarse generation configuration.
            fine_acoustics_config (`Dict`, *optional*):
                Fine generation configuration.
            sample_rate (`int`, *optional*, defaults to 24_000):
                Sample rate.
            codebook_size (`int`, *optional*, defaults to 1024):
                Vector length for each codebook.
        """
        super().__init__()
        if semantic_config is None:
            semantic_config = {}
            logger.info("semantic_config is None. initializing the semantic model with default values.")

        if coarse_acoustics_config is None:
            coarse_acoustics_config = {}
            logger.info("coarse_acoustics_config is None. initializing the coarse model with default values.")

        if fine_acoustics_config is None:
            fine_acoustics_config = {}
            logger.info("fine_acoustics_config is None. initializing the fine model with default values.")

        self.semantic_config = BarkSemanticGenerationConfig(**semantic_config)
        self.coarse_acoustics_config = BarkCoarseGenerationConfig(**coarse_acoustics_config)
        self.fine_acoustics_config = BarkFineGenerationConfig(**fine_acoustics_config)

        self.sample_rate = sample_rate
        self.codebook_size = codebook_size

    @classmethod
    def from_sub_model_configs(
        cls,
        semantic_config: BarkSemanticGenerationConfig,
        coarse_acoustics_config: BarkCoarseGenerationConfig,
        fine_acoustics_config: BarkFineGenerationConfig,
        **kwargs,
    ):
        r"""
        Instantiate a [`BarkGenerationConfig`] (or a derived class) from bark sub-models generation configuration.

        Returns:
            [`BarkGenerationConfig`]: An instance of a configuration object
        """
        return cls(
            semantic_config=semantic_config.to_dict(),
            coarse_acoustics_config=coarse_acoustics_config.to_dict(),
            fine_acoustics_config=fine_acoustics_config.to_dict(),
            **kwargs,
        )

    def to_dict(self):
        """
        Serializes this instance to a Python dictionary. Override the default [`~PretrainedConfig.to_dict`].

        Returns:
            `Dict[str, any]`: Dictionary of all the attributes that make up this configuration instance,
        """
        output = copy.deepcopy(self.__dict__)

        output["semantic_config"] = self.semantic_config.to_dict()
        output["coarse_acoustics_config"] = self.coarse_acoustics_config.to_dict()
        output["fine_acoustics_config"] = self.fine_acoustics_config.to_dict()

        output["model_type"] = self.__class__.model_type
        return output

mindnlp.transformers.models.bark.generation_configuration_bark.BarkGenerationConfig.__init__(semantic_config=None, coarse_acoustics_config=None, fine_acoustics_config=None, sample_rate=24000, codebook_size=1024, **kwargs)

Class that holds a generation configuration for [BarkModel].

The [BarkModel] does not have a generate method, but uses this class to generate speeches with a nested [BarkGenerationConfig] which uses [BarkSemanticGenerationConfig], [BarkCoarseGenerationConfig], [BarkFineGenerationConfig].

This configuration inherit from [GenerationConfig] and can be used to control the model generation. Read the documentation from [GenerationConfig] for more information.

PARAMETER	DESCRIPTION
semantic_config	Semantic generation configuration. TYPE: `Dict`, *optional* DEFAULT: None
coarse_acoustics_config	Coarse generation configuration. TYPE: `Dict`, *optional* DEFAULT: None
fine_acoustics_config	Fine generation configuration. TYPE: `Dict`, *optional* DEFAULT: None
sample_rate	Sample rate. TYPE: `int`, *optional*, defaults to 24_000 DEFAULT: 24000
codebook_size	Vector length for each codebook. TYPE: `int`, *optional*, defaults to 1024 DEFAULT: 1024

Source code in mindnlp/transformers/models/bark/generation_configuration_bark.py

def __init__(
    self,
    semantic_config: Dict = None,
    coarse_acoustics_config: Dict = None,
    fine_acoustics_config: Dict = None,
    sample_rate=24_000,
    codebook_size=1024,
    **kwargs,
):
    """Class that holds a generation configuration for [`BarkModel`].

    The [`BarkModel`] does not have a `generate` method, but uses this class to generate speeches with a nested
    [`BarkGenerationConfig`] which uses [`BarkSemanticGenerationConfig`], [`BarkCoarseGenerationConfig`],
    [`BarkFineGenerationConfig`].

    This configuration inherit from [`GenerationConfig`] and can be used to control the model generation. Read the
    documentation from [`GenerationConfig`] for more information.

    Args:
        semantic_config (`Dict`, *optional*):
            Semantic generation configuration.
        coarse_acoustics_config (`Dict`, *optional*):
            Coarse generation configuration.
        fine_acoustics_config (`Dict`, *optional*):
            Fine generation configuration.
        sample_rate (`int`, *optional*, defaults to 24_000):
            Sample rate.
        codebook_size (`int`, *optional*, defaults to 1024):
            Vector length for each codebook.
    """
    super().__init__()
    if semantic_config is None:
        semantic_config = {}
        logger.info("semantic_config is None. initializing the semantic model with default values.")

    if coarse_acoustics_config is None:
        coarse_acoustics_config = {}
        logger.info("coarse_acoustics_config is None. initializing the coarse model with default values.")

    if fine_acoustics_config is None:
        fine_acoustics_config = {}
        logger.info("fine_acoustics_config is None. initializing the fine model with default values.")

    self.semantic_config = BarkSemanticGenerationConfig(**semantic_config)
    self.coarse_acoustics_config = BarkCoarseGenerationConfig(**coarse_acoustics_config)
    self.fine_acoustics_config = BarkFineGenerationConfig(**fine_acoustics_config)

    self.sample_rate = sample_rate
    self.codebook_size = codebook_size

mindnlp.transformers.models.bark.generation_configuration_bark.BarkGenerationConfig.from_sub_model_configs(semantic_config, coarse_acoustics_config, fine_acoustics_config, **kwargs) classmethod

Instantiate a [BarkGenerationConfig] (or a derived class) from bark sub-models generation configuration.

RETURNS	DESCRIPTION
	[BarkGenerationConfig]: An instance of a configuration object

Source code in mindnlp/transformers/models/bark/generation_configuration_bark.py

@classmethod
def from_sub_model_configs(
    cls,
    semantic_config: BarkSemanticGenerationConfig,
    coarse_acoustics_config: BarkCoarseGenerationConfig,
    fine_acoustics_config: BarkFineGenerationConfig,
    **kwargs,
):
    r"""
    Instantiate a [`BarkGenerationConfig`] (or a derived class) from bark sub-models generation configuration.

    Returns:
        [`BarkGenerationConfig`]: An instance of a configuration object
    """
    return cls(
        semantic_config=semantic_config.to_dict(),
        coarse_acoustics_config=coarse_acoustics_config.to_dict(),
        fine_acoustics_config=fine_acoustics_config.to_dict(),
        **kwargs,
    )

mindnlp.transformers.models.bark.generation_configuration_bark.BarkGenerationConfig.to_dict()

Serializes this instance to a Python dictionary. Override the default [~PretrainedConfig.to_dict].

RETURNS	DESCRIPTION
	Dict[str, any]: Dictionary of all the attributes that make up this configuration instance,

Source code in mindnlp/transformers/models/bark/generation_configuration_bark.py

def to_dict(self):
    """
    Serializes this instance to a Python dictionary. Override the default [`~PretrainedConfig.to_dict`].

    Returns:
        `Dict[str, any]`: Dictionary of all the attributes that make up this configuration instance,
    """
    output = copy.deepcopy(self.__dict__)

    output["semantic_config"] = self.semantic_config.to_dict()
    output["coarse_acoustics_config"] = self.coarse_acoustics_config.to_dict()
    output["fine_acoustics_config"] = self.fine_acoustics_config.to_dict()

    output["model_type"] = self.__class__.model_type
    return output

mindnlp.transformers.models.bark.modeling_bark.BARK_PRETRAINED_MODEL_ARCHIVE_LIST = ['suno/bark-small', 'suno/bark'] module-attribute

mindnlp.transformers.models.bark.modeling_bark.BarkFineModel

Bases: BarkPreTrainedModel

BarkFineModel is a model for generating fine acoustics tokens from input coarse acoustics tokens and optional prompts, building on the BarkPreTrainedModel base class.

This class provides methods for resizing token embeddings, tying weights between input and output embeddings, and generating fine acoustics tokens based on input coarse acoustics tokens and generation configurations.

ATTRIBUTE	DESCRIPTION
config	Configuration object containing model settings.

METHOD	DESCRIPTION
resize_token_embeddings	Resizes the input token embeddings matrix of the model, taking care of tying weights embeddings afterwards if necessary.
tie_weights	Ties the weights between the input embeddings list and the output embeddings list.
generate	Generates fine acoustics tokens from input coarse acoustics tokens and optional speaker prompts, following specified generation configurations.
_resize_token_embeddings	Helper method to resize the token embeddings matrix.
get_input_embeddings	Returns the input embeddings layers.
set_input_embeddings	Sets new input embeddings layers.
get_output_embeddings	Returns the output embeddings layers.
set_output_embeddings	Sets new output embeddings layers.
forward	forwards the model for a specific codebook index, handling input tokens, masks, and labels accordingly.

Source code in mindnlp/transformers/models/bark/modeling_bark.py

class BarkFineModel(BarkPreTrainedModel):

    """
    BarkFineModel is a model for generating fine acoustics tokens from input coarse acoustics tokens and optional prompts,
    building on the BarkPreTrainedModel base class.

    This class provides methods for resizing token embeddings, tying weights between input and output embeddings, and
    generating fine acoustics tokens based on input coarse acoustics tokens and generation configurations.

    Attributes:
        config: Configuration object containing model settings.

    Methods:
        resize_token_embeddings:
            Resizes the input token embeddings matrix of the model, taking care of tying weights embeddings afterwards
            if necessary.

        tie_weights():
            Ties the weights between the input embeddings list and the output embeddings list.

        generate:
            Generates fine acoustics tokens from input coarse acoustics tokens and optional speaker prompts,
            following specified generation configurations.

        _resize_token_embeddings:
            Helper method to resize the token embeddings matrix.

        get_input_embeddings() -> nn.ModuleList:
            Returns the input embeddings layers.

        set_input_embeddings(new_embeddings):
            Sets new input embeddings layers.

        get_output_embeddings() -> nn.ModuleList:
            Returns the output embeddings layers.

        set_output_embeddings(new_output_embeddings):
            Sets new output embeddings layers.

        forward:
            forwards the model for a specific codebook index, handling input tokens, masks, and labels accordingly.
    """
    base_model_prefix = "fine_acoustics"
    config_class = BarkFineConfig
    main_input_name = "codebook_idx"

    def __init__(self, config):
        """
        Initializes a BarkFineModel object.

        Args:
            self (BarkFineModel): The instance of the BarkFineModel class.
            config (Config):
                An object containing configuration parameters for the model.
                Parameters:

                - input_vocab_size (int): The size of the input vocabulary.
                - hidden_size (int): The size of the hidden layers.
                - block_size (int): The size of the blocks in the model.
                - dropout (float): The dropout rate.
                - num_layers (int): The number of layers in the model.
                - output_vocab_size (int): The size of the output vocabulary.
                - n_codes_total (int): The total number of codes used.
                - n_codes_given (int): The number of codes given.

        Returns:
            None.

        Raises:
            None.
        """
        # non-causal gpt-like model with one embedding layer and one lm_head for each codebook of Encodec
        super().__init__(config)
        self.config = config

        # initialize a modified non causal GPT-like model
        # note that for there is one embedding layer and one lm_head for each codebook of Encodec
        self.input_embeds_layers = nn.ModuleList(
            [nn.Embedding(config.input_vocab_size, config.hidden_size) for _ in range(config.n_codes_total)]
        )
        self.position_embeds_layer = nn.Embedding(config.block_size, config.hidden_size)

        self.drop = nn.Dropout(p=config.dropout)

        self.layers = nn.ModuleList([BarkBlock(config, is_causal=False) for _ in range(config.num_layers)])

        self.layernorm_final = nn.LayerNorm(config.hidden_size)

        self.lm_heads = nn.ModuleList(
            [
                nn.Linear(config.hidden_size, config.output_vocab_size, bias=False)
                for _ in range(config.n_codes_given, config.n_codes_total)
            ]
        )
        self.gradient_checkpointing = False
        self.n_codes_total = config.n_codes_total

        # Initialize weights and apply final processing
        self.post_init()

    def get_input_embeddings(self):
        """
        This method retrieves the input embeddings for the BarkFineModel.

        Args:
            self (BarkFineModel): The instance of the BarkFineModel class.

        Returns:
            None: This method returns the input embeddings layers for the BarkFineModel.

        Raises:
            None.
        """
        # one embedding layers for each codebook
        return self.input_embeds_layers

    def set_input_embeddings(self, new_embeddings):
        """
        Sets the input embeddings for the BarkFineModel.

        Args:
            self (BarkFineModel): The instance of the BarkFineModel class.
            new_embeddings (object): The new embeddings to set for the input_embeds_layers attribute.

        Returns:
            None.

        Raises:
            None.
        """
        # one embedding layers for each codebook
        self.input_embeds_layers = new_embeddings

    def get_output_embeddings(self):
        """
        This method is defined in the class 'BarkFineModel' and is used to retrieve the output embeddings of the model.

        Args:
            self: An instance of the 'BarkFineModel' class.

        Returns:
            None.

        Raises:
            None.
        """
        # one lm_head for each codebook
        return self.lm_heads

    def set_output_embeddings(self, new_output_embeddings):
        """
        Method to set new output embeddings for the BarkFineModel.

        Args:
            self (BarkFineModel): The instance of the BarkFineModel class.
            new_output_embeddings (object): New output embeddings to be set for the model.

        Returns:
            None.

        Raises:
            None.
        """
        # one lm_head for each codebook
        self.lm_heads = new_output_embeddings

    def _resize_token_embeddings(self, new_num_tokens, pad_to_multiple_of=None):
        """
        Resize the token embeddings for the BarkFineModel.

        Args:
            self (BarkFineModel): The instance of the BarkFineModel class.
            new_num_tokens (int): The new number of tokens to resize the embeddings to.
            pad_to_multiple_of (int or None): If provided, the embeddings will be padded to be a multiple of this value.

        Returns:
            None: The method updates the token embeddings of the model in place.

        Raises:
            TypeError: If new_num_tokens is not an integer.
            ValueError: If new_num_tokens is less than or equal to 0.
            ValueError: If pad_to_multiple_of is provided but is not an integer.
            ValueError: If pad_to_multiple_of is less than or equal to 0.
        """
        old_embeddings_list = self.get_input_embeddings()
        new_embeddings_list = nn.ModuleList(
            [
                self._get_resized_embeddings(old_embeddings, new_num_tokens, pad_to_multiple_of)
                for old_embeddings in old_embeddings_list
            ]
        )
        self.set_input_embeddings(new_embeddings_list)
        new_num_tokens = new_embeddings_list[0].weight.shape[0]

        # if word embeddings are not tied, make sure that lm head is resized as well
        if self.get_output_embeddings() is not None and not self.config.tie_word_embeddings:
            old_lm_head_list = self.get_output_embeddings()
            new_lm_head_list = nn.ModuleList(
                [self._get_resized_lm_head(old_lm_head, new_num_tokens) for old_lm_head in old_lm_head_list]
            )
            self.set_output_embeddings(new_lm_head_list)

        return self.get_input_embeddings()

    def resize_token_embeddings(
        self, new_num_tokens: Optional[int] = None, pad_to_multiple_of: Optional[int] = None
    ) -> nn.Embedding:
        """
        Resizes input token embeddings matrix of the model if `new_num_tokens != config.vocab_size`.

        Takes care of tying weights embeddings afterwards if the model class has a `tie_weights()` method.

        Arguments:
            new_num_tokens (`int`, *optional*):
                The number of new tokens in the embedding matrix. Increasing the size will add newly initialized
                vectors at the end. Reducing the size will remove vectors from the end. If not provided or `None`, just
                returns a pointer to the input tokens `nn.Embedding` module of the model without doing anything.
            pad_to_multiple_of (`int`, *optional*):
                If set will pad the embedding matrix to a multiple of the provided value.

                This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability
                `>= 7.5` (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128. For more
                details about this, or help on choosing the correct value for resizing, refer to this guide:
                https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc

        Returns:
            `nn.Embedding`: Pointer to the input tokens Embeddings Module of the model.
        """
        model_embeds = self._resize_token_embeddings(new_num_tokens, pad_to_multiple_of)
        if new_num_tokens is None and pad_to_multiple_of is None:
            return model_embeds

        # Update base model and current model config
        self.config.output_vocab_size = model_embeds[0].weight.shape[0]
        self.config.vocab_size = model_embeds[0].weight.shape[0]
        self.output_vocab_size = model_embeds[0].weight.shape[0]
        self.vocab_size = model_embeds[0].weight.shape[0]

        # Tie weights again if needed
        self.tie_weights()

        return model_embeds

    def tie_weights(self):
        """
        Tie the weights between the input embeddings list and the output embeddings list.
        """
        if getattr(self.config, "tie_word_embeddings", True):
            self._tied_weights_keys = []
            output_embeddings = self.get_output_embeddings()
            input_embeddings = self.get_input_embeddings()

            for i in range(self.config.n_codes_total - self.config.n_codes_given):
                # self.input_embeds_layers[i + 1].weight = self.lm_heads[i].weight
                self._tie_or_clone_weights(output_embeddings[i], input_embeddings[i + 1])
                self._tied_weights_keys.append(f"lm_heads.{i}.weight")

        for module in self.modules():
            if hasattr(module, "_tie_weights"):
                module._tie_weights()

    def forward(
        self,
        codebook_idx: int,  # an additionnal idx corresponding to the id of the codebook that will be predicted
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        position_ids: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        labels: Optional[mindspore.Tensor] = None,
        input_embeds: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple[mindspore.Tensor], MaskedLMOutput]:
        """
        forward and process the input data for the BarkFineModel.

        Args:
            self (BarkFineModel): The instance of the BarkFineModel class.
            codebook_idx (int): Index of the codebook to predict.
            input_ids (Optional[mindspore.Tensor], optional): Input tensor containing the tokenized input sequence. Defaults to None.
            attention_mask (Optional[mindspore.Tensor], optional): Tensor indicating which tokens should be attended to. Defaults to None.
            position_ids (Optional[mindspore.Tensor], optional): Tensor containing the position indices of each input token. Defaults to None.
            head_mask (Optional[mindspore.Tensor], optional): Tensor specifying which attention heads to mask. Defaults to None.
            labels (Optional[mindspore.Tensor], optional): Tensor containing the labels for the masked language modeling task. Defaults to None.
            input_embeds (Optional[mindspore.Tensor], optional): Tensor containing the input embeddings. Defaults to None.
            output_attentions (Optional[bool], optional): Whether to output attention weights. Defaults to None.
            output_hidden_states (Optional[bool], optional): Whether to output hidden states. Defaults to None.
            return_dict (Optional[bool], optional): Whether to return a dictionary instead of a tuple. Defaults to None.

        Returns:
            Union[Tuple[mindspore.Tensor], MaskedLMOutput]:
                If `return_dict` is False, returns a tuple containing the following:

                - None: Placeholder for loss value (None for this method).
                - logits (mindspore.Tensor): Predicted logits for masked language modeling task.
                - all_hidden_states (Tuple[mindspore.Tensor]): Tuple of hidden states for each layer.
                - all_self_attentions (Tuple[mindspore.Tensor]): Tuple of attention weights for each layer.

            If `return_dict` is True, returns a MaskedLMOutput object containing the following attributes:

                - loss (None): Placeholder for loss value (None for this method).
                - logits (mindspore.Tensor): Predicted logits for masked language modeling task.
                - hidden_states (Tuple[mindspore.Tensor]): Tuple of hidden states for each layer.
                - attentions (Tuple[mindspore.Tensor]): Tuple of attention weights for each layer.

        Raises:
            ValueError: If codebook_idx is 0, as it should be predicted by the coarse model.
            ValueError: If both input_ids and input_embeds are specified.
            ValueError: If neither input_ids nor input_embeds are specified.
            ValueError: If batch_size is not defined or less than or equal to 0.
            NotImplementedError: If labels are provided, as training is not implemented yet.
        """
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        if codebook_idx == 0:
            raise ValueError("Cannot predict 0th codebook - 0th codebook should be predicted by the coarse model")

        if input_ids is not None and input_embeds is not None:
            raise ValueError("You cannot specify both input_ids and input_embeds at the same time")

        if input_ids is None and input_embeds is None:
            raise ValueError("You have to specify either input_ids or input_embeds")

        if input_ids is not None:
            # the input_embeddings are the sum of the j previous codebooks embeddings before
            # the current codebook_idx codebook

            # forward the GPT model itself
            input_embeds = [
                input_embeds_layer(input_ids[:, :, i]).unsqueeze(-1)
                for i, input_embeds_layer in enumerate(self.input_embeds_layers)
            ]  # token embeddings of shape (b, t, n_embd)
            input_embeds = ops.cat(input_embeds, dim=-1)
            input_embeds = input_embeds[:, :, :, : codebook_idx + 1].sum(axis=-1)

        input_shape = input_embeds.shape[:-1]
        batch_size = input_embeds.shape[0]
        seq_length = input_shape[1]

        if position_ids is None:
            position_ids = ops.arange(0, seq_length, dtype=mindspore.int64)
            position_ids = position_ids.unsqueeze(0)  # shape (1, seq_length)

        position_embeds = self.position_embeds_layer(position_ids)  # position embeddings of shape (1, t, n_embd)

        # Attention mask.
        if attention_mask is not None:
            if batch_size <= 0:
                raise ValueError("batch_size has to be defined and > 0")
            # [bsz, to_seq_length] -> [bsz, 1, 1, to_seq_length]
            # from_seq_length is 1 to easily broadcast
            attention_mask = _prepare_4d_attention_mask(attention_mask, input_embeds.dtype, tgt_len=1)

        head_mask = self.get_head_mask(head_mask, self.config.num_layers)

        hidden_states = self.drop(input_embeds + position_embeds)
        output_shape = input_shape + (hidden_states.shape[-1],)

        all_self_attentions = () if output_attentions else None
        all_hidden_states = () if output_hidden_states else None

        for i, block in enumerate(self.layers):
            if output_hidden_states:
                all_hidden_states = all_hidden_states + (hidden_states,)

            outputs = block(
                hidden_states,
                attention_mask=attention_mask,
                head_mask=head_mask[i],
                output_attentions=output_attentions,
            )

            hidden_states = outputs[0]

            if output_attentions:
                all_self_attentions = all_self_attentions + (outputs[1],)

        hidden_states = self.layernorm_final(hidden_states)
        hidden_states = hidden_states.view(output_shape)

        # Add last hidden state
        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        logits = self.lm_heads[codebook_idx - self.config.n_codes_given](hidden_states)

        loss = None
        if labels is not None:
            raise NotImplementedError("Training is not implemented yet")

        if not return_dict:
            return tuple(v for v in [None, logits, all_hidden_states, all_self_attentions] if v is not None)

        return MaskedLMOutput(
            loss=loss,
            logits=logits,
            hidden_states=all_hidden_states,
            attentions=all_self_attentions,
        )

    def generate(
        self,
        coarse_output: mindspore.Tensor,
        semantic_generation_config: BarkSemanticGenerationConfig = None,
        coarse_generation_config: BarkCoarseGenerationConfig = None,
        fine_generation_config: BarkFineGenerationConfig = None,
        codebook_size: int = 1024,
        history_prompt: Optional[Dict[str, mindspore.Tensor]] = None,
        **kwargs,
    ) -> mindspore.Tensor:
        """
        Generates fine acoustics tokens from input coarse acoustics tokens and an additional optional `Bark` speaker
        prompt.

        Args:
            coarse_output (`mindspore.Tensor` of shape (batch_size, seq_len)):
                Input coarse acoustics ids, i.e the output of `BarkCoarseModel.generate`.
            semantic_generation_config (`BarkSemanticGenerationConfig`):
                Generation config indicating how to generate the semantic tokens.
            coarse_generation_config (`BarkCoarseGenerationConfig`):
                Generation config indicating how to generate the coarse tokens.
            fine_generation_config (`BarkFineGenerationConfig`):
                Generation config indicating how to generate the fine tokens.
            codebook_size (`int`, *optional*, defaults to 1024):
                Codebook channel size, i.e. the size of the output vocabulary per codebook channel.
            history_prompt (`Optional[Dict[str,mindspore.Tensor]]`, *optional*):
                Optional `Bark` speaker prompt.

        Returns:
            mindspore.Tensor: Output fine acoustics tokens.
        """
        if semantic_generation_config is None:
            raise ValueError("`semantic_generation_config` has to be provided")

        if coarse_generation_config is None:
            raise ValueError("`coarse_generation_config` has to be provided")

        if fine_generation_config is None:
            raise ValueError("`fine_generation_config` has to be provided")

        # since we don't really use GenerationConfig through the fine model (autoencoder)
        # and since only temperature is used from the classic GenerationConfig parameters
        # manually impose the kwargs priority over the generation config
        temperature = kwargs.get("temperature", fine_generation_config.temperature)

        max_fine_history_length = fine_generation_config.max_fine_history_length
        max_fine_input_length = fine_generation_config.max_fine_input_length

        # shape: (batch, n_coarse_codebooks * seq_len)
        # new_shape: (batch, seq_len, n_coarse_codebooks)
        coarse_output = coarse_output.view(coarse_output.shape[0], -1, coarse_generation_config.n_coarse_codebooks)

        # brings ids into the range [0, codebook_size -1]
        coarse_output = ops.remainder(coarse_output - semantic_generation_config.semantic_vocab_size, codebook_size)
        batch_size = coarse_output.shape[0]

        if history_prompt is not None:
            x_fine_history = ops.repeat_interleave(history_prompt["fine_prompt"].T[None], batch_size, dim=0).astype(mindspore.int64)
            # swapaxes to get to shape (seq_len, n_fine_codebooks)
        else:
            x_fine_history = None

        n_coarse = coarse_generation_config.n_coarse_codebooks

        # pad the last 6th codebooks
        fine_input = F.pad(
            coarse_output,
            (0, fine_generation_config.n_fine_codebooks - n_coarse),
            "constant",
            codebook_size,
        )

        # prepend history if available (max max_fine_history_length)
        if x_fine_history is not None:
            fine_input = ops.cat([x_fine_history[:, -max_fine_history_length:, :], fine_input], dim=1)

            # len of the fine_history that has been added to fine_input
            n_history = x_fine_history[:, -max_fine_history_length:, :].shape[1]
        else:
            n_history = 0

        n_remove_from_end = 0
        # need to pad if too short (since non-causal model)
        if fine_input.shape[1] < max_fine_input_length:
            n_remove_from_end = max_fine_input_length - fine_input.shape[1]
            fine_input = F.pad(fine_input, (0, 0, 0, n_remove_from_end), mode="constant", value=codebook_size)

        # we can be lazy about fractional loop and just keep overwriting codebooks.
        # seems that coarse_output.shape[1] - (max_fine_input_length - n_history) is equal to minus n_remove_from_end
        # So if we needed to pad because too short, n_loops is always 1 (because n_remove_from_end > 0)
        # If not, we loop over at least twice.

        n_loops = (coarse_output.shape[1] - (max_fine_input_length - n_history)) / max_fine_history_length
        n_loops = int(np.ceil(n_loops))
        n_loops = max(0, n_loops) + 1

        for n_outer in range(n_loops):
            start_idx = min([n_outer * max_fine_history_length, fine_input.shape[1] - max_fine_input_length])

            start_fill_idx = min(
                [n_history + n_outer * max_fine_history_length, fine_input.shape[1] - max_fine_history_length]
            )
            rel_start_fill_idx = start_fill_idx - start_idx
            input_buffer = fine_input[:, start_idx : start_idx + max_fine_input_length, :]
            for n_inner in range(n_coarse, fine_generation_config.n_fine_codebooks):
                logits = self.forward(n_inner, input_buffer).logits
                if temperature is None or temperature == 1.0:
                    relevant_logits = logits[:, rel_start_fill_idx:, :codebook_size]
                    codebook_preds = ops.argmax(relevant_logits, -1)
                else:
                    relevant_logits = logits[:, :, :codebook_size] / temperature
                    # apply softmax
                    probs = ops.softmax(relevant_logits, dim=-1)[:, rel_start_fill_idx:max_fine_input_length]
                    # reshape to 2D: (batch_size, seq_len, codebook_size) -> (batch_size*seq_len, codebook_size)
                    probs = probs.reshape((-1, codebook_size))
                    # multinomial then reshape : (batch_size*seq_len)-> (batch_size,seq_len)
                    codebook_preds = ops.multinomial(probs, num_samples=1).view(batch_size, -1)
                codebook_preds = codebook_preds.to(mindspore.int32)
                input_buffer[:, rel_start_fill_idx:, n_inner] = codebook_preds
                del logits, codebook_preds

            # transfer into fine_input
            for n_inner in range(n_coarse, fine_generation_config.n_fine_codebooks):
                fine_input[
                    :, start_fill_idx : start_fill_idx + (max_fine_input_length - rel_start_fill_idx), n_inner
                ] = input_buffer[:, rel_start_fill_idx:, n_inner]
            del input_buffer

        fine_input = fine_input.swapaxes(1, 2)[:, :, n_history:]
        if n_remove_from_end > 0:
            fine_input = fine_input[:, :, :-n_remove_from_end]

        if fine_input.shape[-1] != coarse_output.shape[-2]:
            raise ValueError("input and output should have the same seq_len")

        return fine_input

mindnlp.transformers.models.bark.modeling_bark.BarkFineModel.__init__(config)

Initializes a BarkFineModel object.

PARAMETER	DESCRIPTION
self	The instance of the BarkFineModel class. TYPE: BarkFineModel
config	An object containing configuration parameters for the model. Parameters: input_vocab_size (int): The size of the input vocabulary. hidden_size (int): The size of the hidden layers. block_size (int): The size of the blocks in the model. dropout (float): The dropout rate. num_layers (int): The number of layers in the model. output_vocab_size (int): The size of the output vocabulary. n_codes_total (int): The total number of codes used. n_codes_given (int): The number of codes given. TYPE: Config

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/bark/modeling_bark.py

def __init__(self, config):
    """
    Initializes a BarkFineModel object.

    Args:
        self (BarkFineModel): The instance of the BarkFineModel class.
        config (Config):
            An object containing configuration parameters for the model.
            Parameters:

            - input_vocab_size (int): The size of the input vocabulary.
            - hidden_size (int): The size of the hidden layers.
            - block_size (int): The size of the blocks in the model.
            - dropout (float): The dropout rate.
            - num_layers (int): The number of layers in the model.
            - output_vocab_size (int): The size of the output vocabulary.
            - n_codes_total (int): The total number of codes used.
            - n_codes_given (int): The number of codes given.

    Returns:
        None.

    Raises:
        None.
    """
    # non-causal gpt-like model with one embedding layer and one lm_head for each codebook of Encodec
    super().__init__(config)
    self.config = config

    # initialize a modified non causal GPT-like model
    # note that for there is one embedding layer and one lm_head for each codebook of Encodec
    self.input_embeds_layers = nn.ModuleList(
        [nn.Embedding(config.input_vocab_size, config.hidden_size) for _ in range(config.n_codes_total)]
    )
    self.position_embeds_layer = nn.Embedding(config.block_size, config.hidden_size)

    self.drop = nn.Dropout(p=config.dropout)

    self.layers = nn.ModuleList([BarkBlock(config, is_causal=False) for _ in range(config.num_layers)])

    self.layernorm_final = nn.LayerNorm(config.hidden_size)

    self.lm_heads = nn.ModuleList(
        [
            nn.Linear(config.hidden_size, config.output_vocab_size, bias=False)
            for _ in range(config.n_codes_given, config.n_codes_total)
        ]
    )
    self.gradient_checkpointing = False
    self.n_codes_total = config.n_codes_total

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.bark.modeling_bark.BarkFineModel.forward(codebook_idx, input_ids=None, attention_mask=None, position_ids=None, head_mask=None, labels=None, input_embeds=None, output_attentions=None, output_hidden_states=None, return_dict=None)

forward and process the input data for the BarkFineModel.

PARAMETER	DESCRIPTION
self	The instance of the BarkFineModel class. TYPE: BarkFineModel
codebook_idx	Index of the codebook to predict. TYPE: int
input_ids	Input tensor containing the tokenized input sequence. Defaults to None. TYPE: Optional[Tensor] DEFAULT: None
attention_mask	Tensor indicating which tokens should be attended to. Defaults to None. TYPE: Optional[Tensor] DEFAULT: None
position_ids	Tensor containing the position indices of each input token. Defaults to None. TYPE: Optional[Tensor] DEFAULT: None
head_mask	Tensor specifying which attention heads to mask. Defaults to None. TYPE: Optional[Tensor] DEFAULT: None
labels	Tensor containing the labels for the masked language modeling task. Defaults to None. TYPE: Optional[Tensor] DEFAULT: None
input_embeds	Tensor containing the input embeddings. Defaults to None. TYPE: Optional[Tensor] DEFAULT: None
output_attentions	Whether to output attention weights. Defaults to None. TYPE: Optional[bool] DEFAULT: None
output_hidden_states	Whether to output hidden states. Defaults to None. TYPE: Optional[bool] DEFAULT: None
return_dict	Whether to return a dictionary instead of a tuple. Defaults to None. TYPE: Optional[bool] DEFAULT: None

RETURNS	DESCRIPTION
Union[Tuple[Tensor], MaskedLMOutput]	Union[Tuple[mindspore.Tensor], MaskedLMOutput]: If return_dict is False, returns a tuple containing the following: None: Placeholder for loss value (None for this method). logits (mindspore.Tensor): Predicted logits for masked language modeling task. all_hidden_states (Tuple[mindspore.Tensor]): Tuple of hidden states for each layer. all_self_attentions (Tuple[mindspore.Tensor]): Tuple of attention weights for each layer.
Union[Tuple[Tensor], MaskedLMOutput]	If return_dict is True, returns a MaskedLMOutput object containing the following attributes: loss (None): Placeholder for loss value (None for this method). logits (mindspore.Tensor): Predicted logits for masked language modeling task. hidden_states (Tuple[mindspore.Tensor]): Tuple of hidden states for each layer. attentions (Tuple[mindspore.Tensor]): Tuple of attention weights for each layer.

RAISES	DESCRIPTION
ValueError	If codebook_idx is 0, as it should be predicted by the coarse model.
ValueError	If both input_ids and input_embeds are specified.
ValueError	If neither input_ids nor input_embeds are specified.
ValueError	If batch_size is not defined or less than or equal to 0.
NotImplementedError	If labels are provided, as training is not implemented yet.

Source code in mindnlp/transformers/models/bark/modeling_bark.py

def forward(
    self,
    codebook_idx: int,  # an additionnal idx corresponding to the id of the codebook that will be predicted
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    position_ids: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    labels: Optional[mindspore.Tensor] = None,
    input_embeds: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple[mindspore.Tensor], MaskedLMOutput]:
    """
    forward and process the input data for the BarkFineModel.

    Args:
        self (BarkFineModel): The instance of the BarkFineModel class.
        codebook_idx (int): Index of the codebook to predict.
        input_ids (Optional[mindspore.Tensor], optional): Input tensor containing the tokenized input sequence. Defaults to None.
        attention_mask (Optional[mindspore.Tensor], optional): Tensor indicating which tokens should be attended to. Defaults to None.
        position_ids (Optional[mindspore.Tensor], optional): Tensor containing the position indices of each input token. Defaults to None.
        head_mask (Optional[mindspore.Tensor], optional): Tensor specifying which attention heads to mask. Defaults to None.
        labels (Optional[mindspore.Tensor], optional): Tensor containing the labels for the masked language modeling task. Defaults to None.
        input_embeds (Optional[mindspore.Tensor], optional): Tensor containing the input embeddings. Defaults to None.
        output_attentions (Optional[bool], optional): Whether to output attention weights. Defaults to None.
        output_hidden_states (Optional[bool], optional): Whether to output hidden states. Defaults to None.
        return_dict (Optional[bool], optional): Whether to return a dictionary instead of a tuple. Defaults to None.

    Returns:
        Union[Tuple[mindspore.Tensor], MaskedLMOutput]:
            If `return_dict` is False, returns a tuple containing the following:

            - None: Placeholder for loss value (None for this method).
            - logits (mindspore.Tensor): Predicted logits for masked language modeling task.
            - all_hidden_states (Tuple[mindspore.Tensor]): Tuple of hidden states for each layer.
            - all_self_attentions (Tuple[mindspore.Tensor]): Tuple of attention weights for each layer.

        If `return_dict` is True, returns a MaskedLMOutput object containing the following attributes:

            - loss (None): Placeholder for loss value (None for this method).
            - logits (mindspore.Tensor): Predicted logits for masked language modeling task.
            - hidden_states (Tuple[mindspore.Tensor]): Tuple of hidden states for each layer.
            - attentions (Tuple[mindspore.Tensor]): Tuple of attention weights for each layer.

    Raises:
        ValueError: If codebook_idx is 0, as it should be predicted by the coarse model.
        ValueError: If both input_ids and input_embeds are specified.
        ValueError: If neither input_ids nor input_embeds are specified.
        ValueError: If batch_size is not defined or less than or equal to 0.
        NotImplementedError: If labels are provided, as training is not implemented yet.
    """
    output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
    output_hidden_states = (
        output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
    )
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    if codebook_idx == 0:
        raise ValueError("Cannot predict 0th codebook - 0th codebook should be predicted by the coarse model")

    if input_ids is not None and input_embeds is not None:
        raise ValueError("You cannot specify both input_ids and input_embeds at the same time")

    if input_ids is None and input_embeds is None:
        raise ValueError("You have to specify either input_ids or input_embeds")

    if input_ids is not None:
        # the input_embeddings are the sum of the j previous codebooks embeddings before
        # the current codebook_idx codebook

        # forward the GPT model itself
        input_embeds = [
            input_embeds_layer(input_ids[:, :, i]).unsqueeze(-1)
            for i, input_embeds_layer in enumerate(self.input_embeds_layers)
        ]  # token embeddings of shape (b, t, n_embd)
        input_embeds = ops.cat(input_embeds, dim=-1)
        input_embeds = input_embeds[:, :, :, : codebook_idx + 1].sum(axis=-1)

    input_shape = input_embeds.shape[:-1]
    batch_size = input_embeds.shape[0]
    seq_length = input_shape[1]

    if position_ids is None:
        position_ids = ops.arange(0, seq_length, dtype=mindspore.int64)
        position_ids = position_ids.unsqueeze(0)  # shape (1, seq_length)

    position_embeds = self.position_embeds_layer(position_ids)  # position embeddings of shape (1, t, n_embd)

    # Attention mask.
    if attention_mask is not None:
        if batch_size <= 0:
            raise ValueError("batch_size has to be defined and > 0")
        # [bsz, to_seq_length] -> [bsz, 1, 1, to_seq_length]
        # from_seq_length is 1 to easily broadcast
        attention_mask = _prepare_4d_attention_mask(attention_mask, input_embeds.dtype, tgt_len=1)

    head_mask = self.get_head_mask(head_mask, self.config.num_layers)

    hidden_states = self.drop(input_embeds + position_embeds)
    output_shape = input_shape + (hidden_states.shape[-1],)

    all_self_attentions = () if output_attentions else None
    all_hidden_states = () if output_hidden_states else None

    for i, block in enumerate(self.layers):
        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        outputs = block(
            hidden_states,
            attention_mask=attention_mask,
            head_mask=head_mask[i],
            output_attentions=output_attentions,
        )

        hidden_states = outputs[0]

        if output_attentions:
            all_self_attentions = all_self_attentions + (outputs[1],)

    hidden_states = self.layernorm_final(hidden_states)
    hidden_states = hidden_states.view(output_shape)

    # Add last hidden state
    if output_hidden_states:
        all_hidden_states = all_hidden_states + (hidden_states,)

    logits = self.lm_heads[codebook_idx - self.config.n_codes_given](hidden_states)

    loss = None
    if labels is not None:
        raise NotImplementedError("Training is not implemented yet")

    if not return_dict:
        return tuple(v for v in [None, logits, all_hidden_states, all_self_attentions] if v is not None)

    return MaskedLMOutput(
        loss=loss,
        logits=logits,
        hidden_states=all_hidden_states,
        attentions=all_self_attentions,
    )

mindnlp.transformers.models.bark.modeling_bark.BarkFineModel.generate(coarse_output, semantic_generation_config=None, coarse_generation_config=None, fine_generation_config=None, codebook_size=1024, history_prompt=None, **kwargs)

Generates fine acoustics tokens from input coarse acoustics tokens and an additional optional Bark speaker prompt.

PARAMETER	DESCRIPTION
coarse_output	Input coarse acoustics ids, i.e the output of BarkCoarseModel.generate. TYPE: `mindspore.Tensor` of shape (batch_size, seq_len
semantic_generation_config	Generation config indicating how to generate the semantic tokens. TYPE: `BarkSemanticGenerationConfig` DEFAULT: None
coarse_generation_config	Generation config indicating how to generate the coarse tokens. TYPE: `BarkCoarseGenerationConfig` DEFAULT: None
fine_generation_config	Generation config indicating how to generate the fine tokens. TYPE: `BarkFineGenerationConfig` DEFAULT: None
codebook_size	Codebook channel size, i.e. the size of the output vocabulary per codebook channel. TYPE: `int`, *optional*, defaults to 1024 DEFAULT: 1024
history_prompt	Optional Bark speaker prompt. TYPE: `Optional[Dict[str,mindspore.Tensor]]`, *optional* DEFAULT: None

RETURNS	DESCRIPTION
Tensor	mindspore.Tensor: Output fine acoustics tokens.

Source code in mindnlp/transformers/models/bark/modeling_bark.py

def generate(
    self,
    coarse_output: mindspore.Tensor,
    semantic_generation_config: BarkSemanticGenerationConfig = None,
    coarse_generation_config: BarkCoarseGenerationConfig = None,
    fine_generation_config: BarkFineGenerationConfig = None,
    codebook_size: int = 1024,
    history_prompt: Optional[Dict[str, mindspore.Tensor]] = None,
    **kwargs,
) -> mindspore.Tensor:
    """
    Generates fine acoustics tokens from input coarse acoustics tokens and an additional optional `Bark` speaker
    prompt.

    Args:
        coarse_output (`mindspore.Tensor` of shape (batch_size, seq_len)):
            Input coarse acoustics ids, i.e the output of `BarkCoarseModel.generate`.
        semantic_generation_config (`BarkSemanticGenerationConfig`):
            Generation config indicating how to generate the semantic tokens.
        coarse_generation_config (`BarkCoarseGenerationConfig`):
            Generation config indicating how to generate the coarse tokens.
        fine_generation_config (`BarkFineGenerationConfig`):
            Generation config indicating how to generate the fine tokens.
        codebook_size (`int`, *optional*, defaults to 1024):
            Codebook channel size, i.e. the size of the output vocabulary per codebook channel.
        history_prompt (`Optional[Dict[str,mindspore.Tensor]]`, *optional*):
            Optional `Bark` speaker prompt.

    Returns:
        mindspore.Tensor: Output fine acoustics tokens.
    """
    if semantic_generation_config is None:
        raise ValueError("`semantic_generation_config` has to be provided")

    if coarse_generation_config is None:
        raise ValueError("`coarse_generation_config` has to be provided")

    if fine_generation_config is None:
        raise ValueError("`fine_generation_config` has to be provided")

    # since we don't really use GenerationConfig through the fine model (autoencoder)
    # and since only temperature is used from the classic GenerationConfig parameters
    # manually impose the kwargs priority over the generation config
    temperature = kwargs.get("temperature", fine_generation_config.temperature)

    max_fine_history_length = fine_generation_config.max_fine_history_length
    max_fine_input_length = fine_generation_config.max_fine_input_length

    # shape: (batch, n_coarse_codebooks * seq_len)
    # new_shape: (batch, seq_len, n_coarse_codebooks)
    coarse_output = coarse_output.view(coarse_output.shape[0], -1, coarse_generation_config.n_coarse_codebooks)

    # brings ids into the range [0, codebook_size -1]
    coarse_output = ops.remainder(coarse_output - semantic_generation_config.semantic_vocab_size, codebook_size)
    batch_size = coarse_output.shape[0]

    if history_prompt is not None:
        x_fine_history = ops.repeat_interleave(history_prompt["fine_prompt"].T[None], batch_size, dim=0).astype(mindspore.int64)
        # swapaxes to get to shape (seq_len, n_fine_codebooks)
    else:
        x_fine_history = None

    n_coarse = coarse_generation_config.n_coarse_codebooks

    # pad the last 6th codebooks
    fine_input = F.pad(
        coarse_output,
        (0, fine_generation_config.n_fine_codebooks - n_coarse),
        "constant",
        codebook_size,
    )

    # prepend history if available (max max_fine_history_length)
    if x_fine_history is not None:
        fine_input = ops.cat([x_fine_history[:, -max_fine_history_length:, :], fine_input], dim=1)

        # len of the fine_history that has been added to fine_input
        n_history = x_fine_history[:, -max_fine_history_length:, :].shape[1]
    else:
        n_history = 0

    n_remove_from_end = 0
    # need to pad if too short (since non-causal model)
    if fine_input.shape[1] < max_fine_input_length:
        n_remove_from_end = max_fine_input_length - fine_input.shape[1]
        fine_input = F.pad(fine_input, (0, 0, 0, n_remove_from_end), mode="constant", value=codebook_size)

    # we can be lazy about fractional loop and just keep overwriting codebooks.
    # seems that coarse_output.shape[1] - (max_fine_input_length - n_history) is equal to minus n_remove_from_end
    # So if we needed to pad because too short, n_loops is always 1 (because n_remove_from_end > 0)
    # If not, we loop over at least twice.

    n_loops = (coarse_output.shape[1] - (max_fine_input_length - n_history)) / max_fine_history_length
    n_loops = int(np.ceil(n_loops))
    n_loops = max(0, n_loops) + 1

    for n_outer in range(n_loops):
        start_idx = min([n_outer * max_fine_history_length, fine_input.shape[1] - max_fine_input_length])

        start_fill_idx = min(
            [n_history + n_outer * max_fine_history_length, fine_input.shape[1] - max_fine_history_length]
        )
        rel_start_fill_idx = start_fill_idx - start_idx
        input_buffer = fine_input[:, start_idx : start_idx + max_fine_input_length, :]
        for n_inner in range(n_coarse, fine_generation_config.n_fine_codebooks):
            logits = self.forward(n_inner, input_buffer).logits
            if temperature is None or temperature == 1.0:
                relevant_logits = logits[:, rel_start_fill_idx:, :codebook_size]
                codebook_preds = ops.argmax(relevant_logits, -1)
            else:
                relevant_logits = logits[:, :, :codebook_size] / temperature
                # apply softmax
                probs = ops.softmax(relevant_logits, dim=-1)[:, rel_start_fill_idx:max_fine_input_length]
                # reshape to 2D: (batch_size, seq_len, codebook_size) -> (batch_size*seq_len, codebook_size)
                probs = probs.reshape((-1, codebook_size))
                # multinomial then reshape : (batch_size*seq_len)-> (batch_size,seq_len)
                codebook_preds = ops.multinomial(probs, num_samples=1).view(batch_size, -1)
            codebook_preds = codebook_preds.to(mindspore.int32)
            input_buffer[:, rel_start_fill_idx:, n_inner] = codebook_preds
            del logits, codebook_preds

        # transfer into fine_input
        for n_inner in range(n_coarse, fine_generation_config.n_fine_codebooks):
            fine_input[
                :, start_fill_idx : start_fill_idx + (max_fine_input_length - rel_start_fill_idx), n_inner
            ] = input_buffer[:, rel_start_fill_idx:, n_inner]
        del input_buffer

    fine_input = fine_input.swapaxes(1, 2)[:, :, n_history:]
    if n_remove_from_end > 0:
        fine_input = fine_input[:, :, :-n_remove_from_end]

    if fine_input.shape[-1] != coarse_output.shape[-2]:
        raise ValueError("input and output should have the same seq_len")

    return fine_input

mindnlp.transformers.models.bark.modeling_bark.BarkFineModel.get_input_embeddings()

This method retrieves the input embeddings for the BarkFineModel.

PARAMETER	DESCRIPTION
self	The instance of the BarkFineModel class. TYPE: BarkFineModel

RETURNS	DESCRIPTION
None	This method returns the input embeddings layers for the BarkFineModel.

Source code in mindnlp/transformers/models/bark/modeling_bark.py

def get_input_embeddings(self):
    """
    This method retrieves the input embeddings for the BarkFineModel.

    Args:
        self (BarkFineModel): The instance of the BarkFineModel class.

    Returns:
        None: This method returns the input embeddings layers for the BarkFineModel.

    Raises:
        None.
    """
    # one embedding layers for each codebook
    return self.input_embeds_layers

mindnlp.transformers.models.bark.modeling_bark.BarkFineModel.get_output_embeddings()

This method is defined in the class 'BarkFineModel' and is used to retrieve the output embeddings of the model.

PARAMETER	DESCRIPTION
self	An instance of the 'BarkFineModel' class.

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/bark/modeling_bark.py

def get_output_embeddings(self):
    """
    This method is defined in the class 'BarkFineModel' and is used to retrieve the output embeddings of the model.

    Args:
        self: An instance of the 'BarkFineModel' class.

    Returns:
        None.

    Raises:
        None.
    """
    # one lm_head for each codebook
    return self.lm_heads

mindnlp.transformers.models.bark.modeling_bark.BarkFineModel.resize_token_embeddings(new_num_tokens=None, pad_to_multiple_of=None)

Resizes input token embeddings matrix of the model if new_num_tokens != config.vocab_size.

Takes care of tying weights embeddings afterwards if the model class has a tie_weights() method.

PARAMETER	DESCRIPTION
new_num_tokens	The number of new tokens in the embedding matrix. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end. If not provided or None, just returns a pointer to the input tokens nn.Embedding module of the model without doing anything. TYPE: `int`, *optional* DEFAULT: None
pad_to_multiple_of	If set will pad the embedding matrix to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128. For more details about this, or help on choosing the correct value for resizing, refer to this guide: https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc TYPE: `int`, *optional* DEFAULT: None

RETURNS	DESCRIPTION
Embedding	nn.Embedding: Pointer to the input tokens Embeddings Module of the model.

Source code in mindnlp/transformers/models/bark/modeling_bark.py

def resize_token_embeddings(
    self, new_num_tokens: Optional[int] = None, pad_to_multiple_of: Optional[int] = None
) -> nn.Embedding:
    """
    Resizes input token embeddings matrix of the model if `new_num_tokens != config.vocab_size`.

    Takes care of tying weights embeddings afterwards if the model class has a `tie_weights()` method.

    Arguments:
        new_num_tokens (`int`, *optional*):
            The number of new tokens in the embedding matrix. Increasing the size will add newly initialized
            vectors at the end. Reducing the size will remove vectors from the end. If not provided or `None`, just
            returns a pointer to the input tokens `nn.Embedding` module of the model without doing anything.
        pad_to_multiple_of (`int`, *optional*):
            If set will pad the embedding matrix to a multiple of the provided value.

            This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability
            `>= 7.5` (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128. For more
            details about this, or help on choosing the correct value for resizing, refer to this guide:
            https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc

    Returns:
        `nn.Embedding`: Pointer to the input tokens Embeddings Module of the model.
    """
    model_embeds = self._resize_token_embeddings(new_num_tokens, pad_to_multiple_of)
    if new_num_tokens is None and pad_to_multiple_of is None:
        return model_embeds

    # Update base model and current model config
    self.config.output_vocab_size = model_embeds[0].weight.shape[0]
    self.config.vocab_size = model_embeds[0].weight.shape[0]
    self.output_vocab_size = model_embeds[0].weight.shape[0]
    self.vocab_size = model_embeds[0].weight.shape[0]

    # Tie weights again if needed
    self.tie_weights()

    return model_embeds

mindnlp.transformers.models.bark.modeling_bark.BarkFineModel.set_input_embeddings(new_embeddings)

Sets the input embeddings for the BarkFineModel.

PARAMETER	DESCRIPTION
self	The instance of the BarkFineModel class. TYPE: BarkFineModel
new_embeddings	The new embeddings to set for the input_embeds_layers attribute. TYPE: object

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/bark/modeling_bark.py

def set_input_embeddings(self, new_embeddings):
    """
    Sets the input embeddings for the BarkFineModel.

    Args:
        self (BarkFineModel): The instance of the BarkFineModel class.
        new_embeddings (object): The new embeddings to set for the input_embeds_layers attribute.

    Returns:
        None.

    Raises:
        None.
    """
    # one embedding layers for each codebook
    self.input_embeds_layers = new_embeddings

mindnlp.transformers.models.bark.modeling_bark.BarkFineModel.set_output_embeddings(new_output_embeddings)

Method to set new output embeddings for the BarkFineModel.

PARAMETER	DESCRIPTION
self	The instance of the BarkFineModel class. TYPE: BarkFineModel
new_output_embeddings	New output embeddings to be set for the model. TYPE: object

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/bark/modeling_bark.py

def set_output_embeddings(self, new_output_embeddings):
    """
    Method to set new output embeddings for the BarkFineModel.

    Args:
        self (BarkFineModel): The instance of the BarkFineModel class.
        new_output_embeddings (object): New output embeddings to be set for the model.

    Returns:
        None.

    Raises:
        None.
    """
    # one lm_head for each codebook
    self.lm_heads = new_output_embeddings

mindnlp.transformers.models.bark.modeling_bark.BarkFineModel.tie_weights()

Tie the weights between the input embeddings list and the output embeddings list.

Source code in mindnlp/transformers/models/bark/modeling_bark.py

def tie_weights(self):
    """
    Tie the weights between the input embeddings list and the output embeddings list.
    """
    if getattr(self.config, "tie_word_embeddings", True):
        self._tied_weights_keys = []
        output_embeddings = self.get_output_embeddings()
        input_embeddings = self.get_input_embeddings()

        for i in range(self.config.n_codes_total - self.config.n_codes_given):
            # self.input_embeds_layers[i + 1].weight = self.lm_heads[i].weight
            self._tie_or_clone_weights(output_embeddings[i], input_embeddings[i + 1])
            self._tied_weights_keys.append(f"lm_heads.{i}.weight")

    for module in self.modules():
        if hasattr(module, "_tie_weights"):
            module._tie_weights()

mindnlp.transformers.models.bark.modeling_bark.BarkSemanticModel

Bases: BarkCausalModel

Represents a semantic model for generating text semantic tokens from an input prompt and an optional Bark speaker prompt.

This class inherits from BarkCausalModel and provides a method to generate output semantic tokens based on the input prompt and generation configuration.

ATTRIBUTE	DESCRIPTION
input_embeds_layer	The layer used for input embeddings. TYPE: Layer
config	Configuration settings for the semantic model. TYPE: Config

Source code in mindnlp/transformers/models/bark/modeling_bark.py

class BarkSemanticModel(BarkCausalModel):

    """
        Represents a semantic model for generating text semantic tokens from an input prompt and an optional `Bark` speaker prompt.

        This class inherits from BarkCausalModel and provides a method to generate output semantic tokens
        based on the input prompt and generation configuration.

        Attributes:
            input_embeds_layer (Layer): The layer used for input embeddings.
            config (Config): Configuration settings for the semantic model.
        """
    base_model_prefix = "semantic"
    config_class = BarkSemanticConfig

    def generate(
        self,
        input_ids: mindspore.Tensor,
        semantic_generation_config: BarkSemanticGenerationConfig = None,
        history_prompt: Optional[Dict[str, mindspore.Tensor]] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        **kwargs,
    ) -> mindspore.Tensor:
        """
        Generates text semantic tokens from an input prompt and an additional optional `Bark` speaker prompt.

        Args:
            input_ids (`Optional[mindspore.Tensor]` of shape (batch_size, seq_len), *optional*):
                Input ids, i.e tokenized input sentences. Will be truncated up to
                semantic_generation_config.max_input_semantic_length tokens. Note that the output audios will be as
                long as the longest generation among the batch.
            semantic_generation_config (`BarkSemanticGenerationConfig`):
                Generation config indicating how to generate the semantic tokens.
            history_prompt (`Optional[Dict[str,mindspore.Tensor]]`, *optional*):
                Optional `Bark` speaker prompt.
            attention_mask (`Optional[mindspore.Tensor]`, *optional*):
                Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:

                - 1 for tokens that are **not masked**,
                - 0 for tokens that are **masked**.

                [What are attention masks?](../glossary#attention-mask)

        Returns:
            mindspore.Tensor: Output semantic tokens.
        """
        if semantic_generation_config is None:
            raise ValueError("`semantic_generation_config` has to be provided")

        batch_size = input_ids.shape[0]

        max_input_semantic_length = semantic_generation_config.max_input_semantic_length

        input_ids = input_ids + semantic_generation_config.text_encoding_offset

        if attention_mask is not None:
            input_ids = input_ids.masked_fill((1 - attention_mask).bool(), semantic_generation_config.text_pad_token)

        if history_prompt is not None:
            semantic_history = history_prompt["semantic_prompt"][-max_input_semantic_length:]
            semantic_history = F.pad(
                semantic_history,
                (0, max_input_semantic_length - len(semantic_history)),
                value=semantic_generation_config.semantic_pad_token,
                mode="constant",
            )
        else:
            semantic_history = mindspore.Tensor(
                [semantic_generation_config.semantic_pad_token] * max_input_semantic_length, dtype=mindspore.int32
            )

        semantic_history = ops.repeat_interleave(semantic_history[None], batch_size, dim=0)

        infer_array = mindspore.Tensor(
            [[semantic_generation_config.semantic_infer_token]] * batch_size, dtype=mindspore.int32
        )

        input_embeds = ops.cat(
            [
                self.input_embeds_layer(input_ids[:, :max_input_semantic_length])
                + self.input_embeds_layer(semantic_history[:, : max_input_semantic_length + 1]),
                self.input_embeds_layer(infer_array),
            ],
            dim=1,
        )

        tokens_to_suppress = list(
            range(semantic_generation_config.semantic_vocab_size, semantic_generation_config.semantic_pad_token)
        )
        tokens_to_suppress.extend(
            list(range(semantic_generation_config.semantic_pad_token + 1, self.config.output_vocab_size))
        )

        suppress_tokens_logits_processor = SuppressTokensLogitsProcessor(tokens_to_suppress)

        min_eos_p = kwargs.get("min_eos_p", semantic_generation_config.min_eos_p)
        early_stopping_logits_processor = BarkEosPrioritizerLogitsProcessor(
            eos_token_id=semantic_generation_config.eos_token_id, min_eos_p=min_eos_p
        )

        # pass input_ids in order to stay consistent with the transformers generate method even though it is not used
        # (except to get the input seq_len - that's why we keep the first 257 tokens)
        semantic_output = super().generate(
            ops.ones(batch_size, max_input_semantic_length + 1, dtype=mindspore.int64),
            input_embeds=input_embeds,
            logits_processor=[suppress_tokens_logits_processor, early_stopping_logits_processor],
            generation_config=semantic_generation_config,
            **kwargs,
        )  # size: 10048

        # take the generated semantic tokens
        semantic_output = semantic_output[:, max_input_semantic_length + 1 :]

        return semantic_output

mindnlp.transformers.models.bark.modeling_bark.BarkSemanticModel.generate(input_ids, semantic_generation_config=None, history_prompt=None, attention_mask=None, **kwargs)

Generates text semantic tokens from an input prompt and an additional optional Bark speaker prompt.

PARAMETER	DESCRIPTION
input_ids	Input ids, i.e tokenized input sentences. Will be truncated up to semantic_generation_config.max_input_semantic_length tokens. Note that the output audios will be as long as the longest generation among the batch. TYPE: `Optional[mindspore.Tensor]` of shape (batch_size, seq_len), *optional*
semantic_generation_config	Generation config indicating how to generate the semantic tokens. TYPE: `BarkSemanticGenerationConfig` DEFAULT: None
history_prompt	Optional Bark speaker prompt. TYPE: `Optional[Dict[str,mindspore.Tensor]]`, *optional* DEFAULT: None
attention_mask	Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]: 1 for tokens that are not masked, 0 for tokens that are masked. What are attention masks? TYPE: `Optional[mindspore.Tensor]`, *optional* DEFAULT: None

RETURNS	DESCRIPTION
Tensor	mindspore.Tensor: Output semantic tokens.

Source code in mindnlp/transformers/models/bark/modeling_bark.py

def generate(
    self,
    input_ids: mindspore.Tensor,
    semantic_generation_config: BarkSemanticGenerationConfig = None,
    history_prompt: Optional[Dict[str, mindspore.Tensor]] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    **kwargs,
) -> mindspore.Tensor:
    """
    Generates text semantic tokens from an input prompt and an additional optional `Bark` speaker prompt.

    Args:
        input_ids (`Optional[mindspore.Tensor]` of shape (batch_size, seq_len), *optional*):
            Input ids, i.e tokenized input sentences. Will be truncated up to
            semantic_generation_config.max_input_semantic_length tokens. Note that the output audios will be as
            long as the longest generation among the batch.
        semantic_generation_config (`BarkSemanticGenerationConfig`):
            Generation config indicating how to generate the semantic tokens.
        history_prompt (`Optional[Dict[str,mindspore.Tensor]]`, *optional*):
            Optional `Bark` speaker prompt.
        attention_mask (`Optional[mindspore.Tensor]`, *optional*):
            Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:

            - 1 for tokens that are **not masked**,
            - 0 for tokens that are **masked**.

            [What are attention masks?](../glossary#attention-mask)

    Returns:
        mindspore.Tensor: Output semantic tokens.
    """
    if semantic_generation_config is None:
        raise ValueError("`semantic_generation_config` has to be provided")

    batch_size = input_ids.shape[0]

    max_input_semantic_length = semantic_generation_config.max_input_semantic_length

    input_ids = input_ids + semantic_generation_config.text_encoding_offset

    if attention_mask is not None:
        input_ids = input_ids.masked_fill((1 - attention_mask).bool(), semantic_generation_config.text_pad_token)

    if history_prompt is not None:
        semantic_history = history_prompt["semantic_prompt"][-max_input_semantic_length:]
        semantic_history = F.pad(
            semantic_history,
            (0, max_input_semantic_length - len(semantic_history)),
            value=semantic_generation_config.semantic_pad_token,
            mode="constant",
        )
    else:
        semantic_history = mindspore.Tensor(
            [semantic_generation_config.semantic_pad_token] * max_input_semantic_length, dtype=mindspore.int32
        )

    semantic_history = ops.repeat_interleave(semantic_history[None], batch_size, dim=0)

    infer_array = mindspore.Tensor(
        [[semantic_generation_config.semantic_infer_token]] * batch_size, dtype=mindspore.int32
    )

    input_embeds = ops.cat(
        [
            self.input_embeds_layer(input_ids[:, :max_input_semantic_length])
            + self.input_embeds_layer(semantic_history[:, : max_input_semantic_length + 1]),
            self.input_embeds_layer(infer_array),
        ],
        dim=1,
    )

    tokens_to_suppress = list(
        range(semantic_generation_config.semantic_vocab_size, semantic_generation_config.semantic_pad_token)
    )
    tokens_to_suppress.extend(
        list(range(semantic_generation_config.semantic_pad_token + 1, self.config.output_vocab_size))
    )

    suppress_tokens_logits_processor = SuppressTokensLogitsProcessor(tokens_to_suppress)

    min_eos_p = kwargs.get("min_eos_p", semantic_generation_config.min_eos_p)
    early_stopping_logits_processor = BarkEosPrioritizerLogitsProcessor(
        eos_token_id=semantic_generation_config.eos_token_id, min_eos_p=min_eos_p
    )

    # pass input_ids in order to stay consistent with the transformers generate method even though it is not used
    # (except to get the input seq_len - that's why we keep the first 257 tokens)
    semantic_output = super().generate(
        ops.ones(batch_size, max_input_semantic_length + 1, dtype=mindspore.int64),
        input_embeds=input_embeds,
        logits_processor=[suppress_tokens_logits_processor, early_stopping_logits_processor],
        generation_config=semantic_generation_config,
        **kwargs,
    )  # size: 10048

    # take the generated semantic tokens
    semantic_output = semantic_output[:, max_input_semantic_length + 1 :]

    return semantic_output

mindnlp.transformers.models.bark.modeling_bark.BarkCoarseModel

Bases: BarkCausalModel

Represents a model for generating coarse acoustics tokens from input text semantic tokens and an optional Bark speaker prompt.

This class inherits from BarkCausalModel and includes methods for preprocessing histories and generating coarse acoustics tokens based on provided configurations and inputs.

METHOD	DESCRIPTION
preprocess_histories	Preprocesses optional Bark speaker prompts before generating coarse acoustics tokens. Returns processed semantic and coarse speaker prompts.
generate	Generates coarse acoustics tokens based on input text semantic tokens, generation configurations, and optional speaker prompts. Returns the output coarse acoustics tokens.

PARAMETER	DESCRIPTION
semantic_output	Input text semantic ids. TYPE: Tensor
semantic_generation_config	Generation config for semantic tokens. TYPE: BarkSemanticGenerationConfig
coarse_generation_config	Generation config for coarse tokens. TYPE: BarkCoarseGenerationConfig
codebook_size	Size of the output vocabulary per codebook channel. TYPE: int
history_prompt	Optional Bark speaker prompt. TYPE: Optional[Dict[str, Tensor]]
return_output_lengths	Whether to return the output lengths. TYPE: bool

RETURNS	DESCRIPTION
	Conditional return: By default: mindspore.Tensor: Output coarse acoustics tokens. If return_output_lengths=True: Tuple(mindspore.Tensor, mindspore.Tensor): Output coarse acoustics tokens and the length of each sample in the batch.

Source code in mindnlp/transformers/models/bark/modeling_bark.py

class BarkCoarseModel(BarkCausalModel):

    """
    Represents a model for generating coarse acoustics tokens from input text semantic tokens and an optional `Bark` speaker prompt.

    This class inherits from BarkCausalModel and includes methods for preprocessing histories and generating coarse acoustics tokens based on provided configurations and inputs.

    Methods:
        preprocess_histories(max_coarse_history, semantic_to_coarse_ratio, batch_size, semantic_generation_config, codebook_size, history_prompt=None):
            Preprocesses optional `Bark` speaker prompts before generating coarse acoustics tokens. Returns processed semantic and coarse speaker prompts.

        generate(semantic_output, semantic_generation_config, coarse_generation_config, codebook_size=1024, history_prompt=None, return_output_lengths=None, **kwargs):
            Generates coarse acoustics tokens based on input text semantic tokens, generation configurations, and optional speaker prompts. Returns the output coarse acoustics tokens.

    Args:
        semantic_output (mindspore.Tensor): Input text semantic ids.
        semantic_generation_config (BarkSemanticGenerationConfig): Generation config for semantic tokens.
        coarse_generation_config (BarkCoarseGenerationConfig): Generation config for coarse tokens.
        codebook_size (int, optional): Size of the output vocabulary per codebook channel.
        history_prompt (Optional[Dict[str, mindspore.Tensor]], optional): Optional `Bark` speaker prompt.
        return_output_lengths (bool, optional): Whether to return the output lengths.

    Returns:
        Conditional return:
            By default:

            - mindspore.Tensor: Output coarse acoustics tokens.

            If return_output_lengths=True:

            - Tuple(mindspore.Tensor, mindspore.Tensor):
            Output coarse acoustics tokens and the length of each sample in the batch.
    """
    base_model_prefix = "coarse_acoustics"
    config_class = BarkCoarseConfig

    def preprocess_histories(
        self,
        max_coarse_history: int,
        semantic_to_coarse_ratio: int,
        batch_size: int,
        semantic_generation_config: int,
        codebook_size: int,
        history_prompt: Optional[Dict[str, mindspore.Tensor]] = None,
    ):
        """
        Preprocess the optional `Bark` speaker prompts before `self.generate`.

        Args:
            max_coarse_history (`int`):
                Maximum size of coarse tokens used.
            semantic_to_coarse_ratio (`int`):
                Ratio of semantic to coarse frequency
            batch_size (`int`):
                Batch size, i.e the number of samples.
            semantic_generation_config (`BarkSemanticGenerationConfig`):
                Generation config indicating how to generate the semantic tokens.
            codebook_size (`int`):
                Codebook channel size, i.e. the size of the output vocabulary per codebook channel.
            history_prompt (`Optional[Dict[str,mindspore.Tensor]]`):
                Optional `Bark` speaker prompt.

        Returns:
            `tuple(mindspore.Tensor)`:

                - **x_semantic_history** (`mindspore.Tensor` -- Processed semantic speaker prompt.
                - **x_coarse_history** (`mindspore.Tensor`) -- Processed coarse speaker prompt.
        """
        if history_prompt is not None:
            x_semantic_history = ops.repeat_interleave(history_prompt["semantic_prompt"][None], batch_size, dim=0)
            # clone to avoid modifying history_prompt.coarse_prompt
            x_coarse_history = history_prompt["coarse_prompt"].copy()

            # offset x_coarse_history
            if codebook_size is not None:
                for n in range(1, x_coarse_history.shape[0]):
                    # offset
                    x_coarse_history[n, :] += codebook_size * n

            # flatten x_coarse_history
            x_coarse_history = ops.swapaxes(x_coarse_history, 0, 1).view(-1)

            x_coarse_history = x_coarse_history + semantic_generation_config.semantic_vocab_size

            x_coarse_history = ops.repeat_interleave(x_coarse_history[None], batch_size, dim=0)
            # e.g: after SEMANTIC_VOCAB_SIZE (10000), 1024 tokens dedicated to first codebook, 1024 next tokens
            # dedicated to second codebook.

            max_semantic_history = int(np.floor(max_coarse_history / semantic_to_coarse_ratio))
            # trim histories correctly
            n_semantic_hist_provided = min(
                [
                    max_semantic_history,
                    x_semantic_history.shape[1] - x_semantic_history.shape[1] % 2,
                    int(np.floor(x_coarse_history.shape[1] / semantic_to_coarse_ratio)),
                ]
            )

            n_coarse_hist_provided = int(round(n_semantic_hist_provided * semantic_to_coarse_ratio))

            x_semantic_history = x_semantic_history[:, -n_semantic_hist_provided:].long()
            x_coarse_history = x_coarse_history[:, -n_coarse_hist_provided:].long()
            # bit of a hack for time alignment (sounds better) - from Bark original implementation
            x_coarse_history = x_coarse_history[:, :-2]

        else:
            # shape: (batch_size, 0)
            x_semantic_history = mindspore.tensor(mindspore._c_expression.Tensor([[]] * batch_size, mindspore.int64))
            x_coarse_history = mindspore.tensor(mindspore._c_expression.Tensor([[]] * batch_size, mindspore.int64))
        return x_semantic_history, x_coarse_history

    def generate(
        self,
        semantic_output: mindspore.Tensor,
        semantic_generation_config: BarkSemanticGenerationConfig = None,
        coarse_generation_config: BarkCoarseGenerationConfig = None,
        codebook_size: int = 1024,
        history_prompt: Optional[Dict[str, mindspore.Tensor]] = None,
        return_output_lengths: Optional[bool] = None,
        **kwargs,
    ) -> Union[mindspore.Tensor, Tuple[mindspore.Tensor, mindspore.Tensor]]:
        """
        Generates coarse acoustics tokens from input text semantic tokens and an additional optional `Bark` speaker
        prompt.

        Args:
            semantic_output (`mindspore.Tensor` of shape (batch_size, seq_len), *optional*):
                Input text semantic ids, i.e the output of `BarkSemanticModel.generate`.
            semantic_generation_config (`BarkSemanticGenerationConfig`):
                Generation config indicating how to generate the semantic tokens.
            coarse_generation_config (`BarkCoarseGenerationConfig`):
                Generation config indicating how to generate the coarse tokens.
            codebook_size (`int`, *optional*, defaults to 1024):
                Codebook channel size, i.e. the size of the output vocabulary per codebook channel.
            history_prompt (`Optional[Dict[str,mindspore.Tensor]]`, *optional*):
                Optional `Bark` speaker prompt.
            return_output_lengths (`bool`, *optional*):
                Whether or not to return the output lengths. Useful when batching.

        Returns:
            Conditional return:
                By default:

                - mindspore.Tensor: Output coarse acoustics tokens.
                If `return_output_lengths=True`:

                - `Tuple(mindspore.Tensor, mindspore.Tensor): The output coarse acoustics tokens, and the
                length of each sample of the batch.
        """
        if semantic_generation_config is None:
            raise ValueError("`semantic_generation_config` has to be provided")

        if coarse_generation_config is None:
            raise ValueError("`coarse_generation_config` has to be provided")

        max_coarse_input_length = coarse_generation_config.max_coarse_input_length
        max_coarse_history = coarse_generation_config.max_coarse_history
        sliding_window_len = coarse_generation_config.sliding_window_len

        # replace semantic_pad_token (eos_tok and pad_tok here) with coarse_semantic_pad_token i.e the pad_token
        # used in the next model
        semantic_output = semantic_output.masked_fill(
            semantic_output == semantic_generation_config.semantic_pad_token,
            coarse_generation_config.coarse_semantic_pad_token,
        )

        semantic_to_coarse_ratio = (
            coarse_generation_config.coarse_rate_hz
            / semantic_generation_config.semantic_rate_hz
            * coarse_generation_config.n_coarse_codebooks
        )
        max_semantic_history = int(np.floor(max_coarse_history / semantic_to_coarse_ratio))

        output_lengths = (semantic_output != coarse_generation_config.coarse_semantic_pad_token).sum(1)
        output_lengths = ops.floor(
            output_lengths * semantic_to_coarse_ratio / coarse_generation_config.n_coarse_codebooks
        )
        output_lengths = ops.round(output_lengths * coarse_generation_config.n_coarse_codebooks).int()

        max_generated_len = ops.max(output_lengths).item()

        batch_size = semantic_output.shape[0]

        x_semantic_history, x_coarse = self.preprocess_histories(
            history_prompt=history_prompt,
            max_coarse_history=max_coarse_history,
            semantic_to_coarse_ratio=semantic_to_coarse_ratio,
            batch_size=batch_size,
            semantic_generation_config=semantic_generation_config,
            codebook_size=codebook_size,
        )
        base_semantic_idx = x_semantic_history.shape[1]

        semantic_output = ops.hstack([x_semantic_history, semantic_output])

        n_window_steps = int(np.ceil(max_generated_len / sliding_window_len))

        total_generated_len = 0

        len_coarse_history = x_coarse.shape[1]

        for _ in range(n_window_steps):
            semantic_idx = base_semantic_idx + int(round(total_generated_len / semantic_to_coarse_ratio))

            # pad from right side
            input_coarse = semantic_output[:, int(np.max([0, semantic_idx - max_semantic_history])) :]
            input_coarse = input_coarse[:, :max_coarse_input_length]
            input_coarse = F.pad(
                input_coarse,
                (0, max_coarse_input_length - input_coarse.shape[-1]),
                "constant",
                coarse_generation_config.coarse_semantic_pad_token,
            )

            if 0 in x_coarse.shape:
                input_coarse = ops.hstack(
                    [
                        input_coarse,
                        mindspore.Tensor([[coarse_generation_config.coarse_infer_token]] * batch_size),
                    ]
                )
            else:
                input_coarse = ops.hstack(
                    [
                        input_coarse,
                        mindspore.Tensor([[coarse_generation_config.coarse_infer_token]] * batch_size),
                        x_coarse[:, -max_coarse_history:],
                    ]
                )

            alternatingLogitsProcessor = AlternatingCodebooksLogitsProcessor(
                input_coarse.shape[1],
                semantic_generation_config.semantic_vocab_size,
                codebook_size,
            )

            output_coarse = super().generate(
                input_coarse,
                logits_processor=[alternatingLogitsProcessor],
                max_new_tokens=min(sliding_window_len, max_generated_len - total_generated_len),
                generation_config=coarse_generation_config,
                **kwargs,
            )

            input_coarse_len = input_coarse.shape[1]

            x_coarse = ops.hstack([x_coarse, output_coarse[:, input_coarse_len:]])
            total_generated_len = x_coarse.shape[1] - len_coarse_history

            del output_coarse

        coarse_output = x_coarse[:, len_coarse_history:]

        if return_output_lengths:
            return coarse_output, output_lengths

        return coarse_output

mindnlp.transformers.models.bark.modeling_bark.BarkCoarseModel.generate(semantic_output, semantic_generation_config=None, coarse_generation_config=None, codebook_size=1024, history_prompt=None, return_output_lengths=None, **kwargs)

Generates coarse acoustics tokens from input text semantic tokens and an additional optional Bark speaker prompt.

PARAMETER	DESCRIPTION
semantic_output	Input text semantic ids, i.e the output of BarkSemanticModel.generate. TYPE: `mindspore.Tensor` of shape (batch_size, seq_len), *optional*
semantic_generation_config	Generation config indicating how to generate the semantic tokens. TYPE: `BarkSemanticGenerationConfig` DEFAULT: None
coarse_generation_config	Generation config indicating how to generate the coarse tokens. TYPE: `BarkCoarseGenerationConfig` DEFAULT: None
codebook_size	Codebook channel size, i.e. the size of the output vocabulary per codebook channel. TYPE: `int`, *optional*, defaults to 1024 DEFAULT: 1024
history_prompt	Optional Bark speaker prompt. TYPE: `Optional[Dict[str,mindspore.Tensor]]`, *optional* DEFAULT: None
return_output_lengths	Whether or not to return the output lengths. Useful when batching. TYPE: `bool`, *optional* DEFAULT: None

RETURNS	DESCRIPTION
Union[Tensor, Tuple[Tensor, Tensor]]	Conditional return: By default: mindspore.Tensor: Output coarse acoustics tokens. If return_output_lengths=True: `Tuple(mindspore.Tensor, mindspore.Tensor): The output coarse acoustics tokens, and the length of each sample of the batch.

Source code in mindnlp/transformers/models/bark/modeling_bark.py

def generate(
    self,
    semantic_output: mindspore.Tensor,
    semantic_generation_config: BarkSemanticGenerationConfig = None,
    coarse_generation_config: BarkCoarseGenerationConfig = None,
    codebook_size: int = 1024,
    history_prompt: Optional[Dict[str, mindspore.Tensor]] = None,
    return_output_lengths: Optional[bool] = None,
    **kwargs,
) -> Union[mindspore.Tensor, Tuple[mindspore.Tensor, mindspore.Tensor]]:
    """
    Generates coarse acoustics tokens from input text semantic tokens and an additional optional `Bark` speaker
    prompt.

    Args:
        semantic_output (`mindspore.Tensor` of shape (batch_size, seq_len), *optional*):
            Input text semantic ids, i.e the output of `BarkSemanticModel.generate`.
        semantic_generation_config (`BarkSemanticGenerationConfig`):
            Generation config indicating how to generate the semantic tokens.
        coarse_generation_config (`BarkCoarseGenerationConfig`):
            Generation config indicating how to generate the coarse tokens.
        codebook_size (`int`, *optional*, defaults to 1024):
            Codebook channel size, i.e. the size of the output vocabulary per codebook channel.
        history_prompt (`Optional[Dict[str,mindspore.Tensor]]`, *optional*):
            Optional `Bark` speaker prompt.
        return_output_lengths (`bool`, *optional*):
            Whether or not to return the output lengths. Useful when batching.

    Returns:
        Conditional return:
            By default:

            - mindspore.Tensor: Output coarse acoustics tokens.
            If `return_output_lengths=True`:

            - `Tuple(mindspore.Tensor, mindspore.Tensor): The output coarse acoustics tokens, and the
            length of each sample of the batch.
    """
    if semantic_generation_config is None:
        raise ValueError("`semantic_generation_config` has to be provided")

    if coarse_generation_config is None:
        raise ValueError("`coarse_generation_config` has to be provided")

    max_coarse_input_length = coarse_generation_config.max_coarse_input_length
    max_coarse_history = coarse_generation_config.max_coarse_history
    sliding_window_len = coarse_generation_config.sliding_window_len

    # replace semantic_pad_token (eos_tok and pad_tok here) with coarse_semantic_pad_token i.e the pad_token
    # used in the next model
    semantic_output = semantic_output.masked_fill(
        semantic_output == semantic_generation_config.semantic_pad_token,
        coarse_generation_config.coarse_semantic_pad_token,
    )

    semantic_to_coarse_ratio = (
        coarse_generation_config.coarse_rate_hz
        / semantic_generation_config.semantic_rate_hz
        * coarse_generation_config.n_coarse_codebooks
    )
    max_semantic_history = int(np.floor(max_coarse_history / semantic_to_coarse_ratio))

    output_lengths = (semantic_output != coarse_generation_config.coarse_semantic_pad_token).sum(1)
    output_lengths = ops.floor(
        output_lengths * semantic_to_coarse_ratio / coarse_generation_config.n_coarse_codebooks
    )
    output_lengths = ops.round(output_lengths * coarse_generation_config.n_coarse_codebooks).int()

    max_generated_len = ops.max(output_lengths).item()

    batch_size = semantic_output.shape[0]

    x_semantic_history, x_coarse = self.preprocess_histories(
        history_prompt=history_prompt,
        max_coarse_history=max_coarse_history,
        semantic_to_coarse_ratio=semantic_to_coarse_ratio,
        batch_size=batch_size,
        semantic_generation_config=semantic_generation_config,
        codebook_size=codebook_size,
    )
    base_semantic_idx = x_semantic_history.shape[1]

    semantic_output = ops.hstack([x_semantic_history, semantic_output])

    n_window_steps = int(np.ceil(max_generated_len / sliding_window_len))

    total_generated_len = 0

    len_coarse_history = x_coarse.shape[1]

    for _ in range(n_window_steps):
        semantic_idx = base_semantic_idx + int(round(total_generated_len / semantic_to_coarse_ratio))

        # pad from right side
        input_coarse = semantic_output[:, int(np.max([0, semantic_idx - max_semantic_history])) :]
        input_coarse = input_coarse[:, :max_coarse_input_length]
        input_coarse = F.pad(
            input_coarse,
            (0, max_coarse_input_length - input_coarse.shape[-1]),
            "constant",
            coarse_generation_config.coarse_semantic_pad_token,
        )

        if 0 in x_coarse.shape:
            input_coarse = ops.hstack(
                [
                    input_coarse,
                    mindspore.Tensor([[coarse_generation_config.coarse_infer_token]] * batch_size),
                ]
            )
        else:
            input_coarse = ops.hstack(
                [
                    input_coarse,
                    mindspore.Tensor([[coarse_generation_config.coarse_infer_token]] * batch_size),
                    x_coarse[:, -max_coarse_history:],
                ]
            )

        alternatingLogitsProcessor = AlternatingCodebooksLogitsProcessor(
            input_coarse.shape[1],
            semantic_generation_config.semantic_vocab_size,
            codebook_size,
        )

        output_coarse = super().generate(
            input_coarse,
            logits_processor=[alternatingLogitsProcessor],
            max_new_tokens=min(sliding_window_len, max_generated_len - total_generated_len),
            generation_config=coarse_generation_config,
            **kwargs,
        )

        input_coarse_len = input_coarse.shape[1]

        x_coarse = ops.hstack([x_coarse, output_coarse[:, input_coarse_len:]])
        total_generated_len = x_coarse.shape[1] - len_coarse_history

        del output_coarse

    coarse_output = x_coarse[:, len_coarse_history:]

    if return_output_lengths:
        return coarse_output, output_lengths

    return coarse_output

mindnlp.transformers.models.bark.modeling_bark.BarkCoarseModel.preprocess_histories(max_coarse_history, semantic_to_coarse_ratio, batch_size, semantic_generation_config, codebook_size, history_prompt=None)

Preprocess the optional Bark speaker prompts before self.generate.

PARAMETER	DESCRIPTION
max_coarse_history	Maximum size of coarse tokens used. TYPE: `int`
semantic_to_coarse_ratio	Ratio of semantic to coarse frequency TYPE: `int`
batch_size	Batch size, i.e the number of samples. TYPE: `int`
semantic_generation_config	Generation config indicating how to generate the semantic tokens. TYPE: `BarkSemanticGenerationConfig`
codebook_size	Codebook channel size, i.e. the size of the output vocabulary per codebook channel. TYPE: `int`
history_prompt	Optional Bark speaker prompt. TYPE: `Optional[Dict[str,mindspore.Tensor]]` DEFAULT: None

RETURNS	DESCRIPTION
	tuple(mindspore.Tensor): x_semantic_history (mindspore.Tensor -- Processed semantic speaker prompt. x_coarse_history (mindspore.Tensor) -- Processed coarse speaker prompt.

Source code in mindnlp/transformers/models/bark/modeling_bark.py

def preprocess_histories(
    self,
    max_coarse_history: int,
    semantic_to_coarse_ratio: int,
    batch_size: int,
    semantic_generation_config: int,
    codebook_size: int,
    history_prompt: Optional[Dict[str, mindspore.Tensor]] = None,
):
    """
    Preprocess the optional `Bark` speaker prompts before `self.generate`.

    Args:
        max_coarse_history (`int`):
            Maximum size of coarse tokens used.
        semantic_to_coarse_ratio (`int`):
            Ratio of semantic to coarse frequency
        batch_size (`int`):
            Batch size, i.e the number of samples.
        semantic_generation_config (`BarkSemanticGenerationConfig`):
            Generation config indicating how to generate the semantic tokens.
        codebook_size (`int`):
            Codebook channel size, i.e. the size of the output vocabulary per codebook channel.
        history_prompt (`Optional[Dict[str,mindspore.Tensor]]`):
            Optional `Bark` speaker prompt.

    Returns:
        `tuple(mindspore.Tensor)`:

            - **x_semantic_history** (`mindspore.Tensor` -- Processed semantic speaker prompt.
            - **x_coarse_history** (`mindspore.Tensor`) -- Processed coarse speaker prompt.
    """
    if history_prompt is not None:
        x_semantic_history = ops.repeat_interleave(history_prompt["semantic_prompt"][None], batch_size, dim=0)
        # clone to avoid modifying history_prompt.coarse_prompt
        x_coarse_history = history_prompt["coarse_prompt"].copy()

        # offset x_coarse_history
        if codebook_size is not None:
            for n in range(1, x_coarse_history.shape[0]):
                # offset
                x_coarse_history[n, :] += codebook_size * n

        # flatten x_coarse_history
        x_coarse_history = ops.swapaxes(x_coarse_history, 0, 1).view(-1)

        x_coarse_history = x_coarse_history + semantic_generation_config.semantic_vocab_size

        x_coarse_history = ops.repeat_interleave(x_coarse_history[None], batch_size, dim=0)
        # e.g: after SEMANTIC_VOCAB_SIZE (10000), 1024 tokens dedicated to first codebook, 1024 next tokens
        # dedicated to second codebook.

        max_semantic_history = int(np.floor(max_coarse_history / semantic_to_coarse_ratio))
        # trim histories correctly
        n_semantic_hist_provided = min(
            [
                max_semantic_history,
                x_semantic_history.shape[1] - x_semantic_history.shape[1] % 2,
                int(np.floor(x_coarse_history.shape[1] / semantic_to_coarse_ratio)),
            ]
        )

        n_coarse_hist_provided = int(round(n_semantic_hist_provided * semantic_to_coarse_ratio))

        x_semantic_history = x_semantic_history[:, -n_semantic_hist_provided:].long()
        x_coarse_history = x_coarse_history[:, -n_coarse_hist_provided:].long()
        # bit of a hack for time alignment (sounds better) - from Bark original implementation
        x_coarse_history = x_coarse_history[:, :-2]

    else:
        # shape: (batch_size, 0)
        x_semantic_history = mindspore.tensor(mindspore._c_expression.Tensor([[]] * batch_size, mindspore.int64))
        x_coarse_history = mindspore.tensor(mindspore._c_expression.Tensor([[]] * batch_size, mindspore.int64))
    return x_semantic_history, x_coarse_history

mindnlp.transformers.models.bark.modeling_bark.BarkModel

Bases: BarkPreTrainedModel

BarkModel

This class represents a Bark model that is used for generating audio from an input prompt and an optional speaker prompt. It is a subclass of BarkPreTrainedModel.

METHOD	DESCRIPTION
__init__	Initializes the BarkModel instance.
codec_decode	Turns quantized audio codes into an audio array using the encodec.
generate	Generates audio from an input prompt and an optional speaker prompt.

ATTRIBUTE	DESCRIPTION
semantic	An instance of BarkSemanticModel.
coarse_acoustics	An instance of BarkCoarseModel.
fine_acoustics	An instance of BarkFineModel.
codec_model	An instance of the AutoModel class.
config	The configuration object for the BarkModel.

Example

>>> from transformers import AutoProcessor, BarkModel
...
>>> processor = AutoProcessor.from_pretrained("suno/bark-small")
>>> model = BarkModel.from_pretrained("suno/bark-small")
...
>>> # To add a voice preset, you can pass `voice_preset` to `BarkProcessor.__call__(...)`
>>> voice_preset = "v2/en_speaker_6"
...
>>> inputs = processor("Hello, my dog is cute, I need him in my life", voice_preset=voice_preset)
...
>>> audio_array = model.generate(**inputs, semantic_max_new_tokens=100)
>>> audio_array = audio_array.cpu().numpy().squeeze()

Source code in mindnlp/transformers/models/bark/modeling_bark.py

class BarkModel(BarkPreTrainedModel):
    """
    BarkModel

    This class represents a Bark model that is used for generating audio from an input prompt and an optional speaker prompt. It is a subclass of BarkPreTrainedModel.

    Methods:
        __init__: Initializes the BarkModel instance.
        codec_decode: Turns quantized audio codes into an audio array using the encodec.
        generate:
            Generates audio from an input prompt and an optional speaker prompt.

    Attributes:
        semantic: An instance of BarkSemanticModel.
        coarse_acoustics: An instance of BarkCoarseModel.
        fine_acoustics: An instance of BarkFineModel.
        codec_model: An instance of the AutoModel class.
        config: The configuration object for the BarkModel.

    Example:
        ```python
        >>> from transformers import AutoProcessor, BarkModel
        ...
        >>> processor = AutoProcessor.from_pretrained("suno/bark-small")
        >>> model = BarkModel.from_pretrained("suno/bark-small")
        ...
        >>> # To add a voice preset, you can pass `voice_preset` to `BarkProcessor.__call__(...)`
        >>> voice_preset = "v2/en_speaker_6"
        ...
        >>> inputs = processor("Hello, my dog is cute, I need him in my life", voice_preset=voice_preset)
        ...
        >>> audio_array = model.generate(**inputs, semantic_max_new_tokens=100)
        >>> audio_array = audio_array.cpu().numpy().squeeze()
        ```

    """
    config_class = BarkConfig

    def __init__(self, config):
        """
        Initializes a new instance of BarkModel.

        Args:
            self (BarkModel): The current instance of the BarkModel class.
            config (dict): A dictionary containing configuration settings for the BarkModel.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(config)

        self.semantic = BarkSemanticModel(config.semantic_config)
        self.coarse_acoustics = BarkCoarseModel(config.coarse_acoustics_config)
        self.fine_acoustics = BarkFineModel(config.fine_acoustics_config)

        self.codec_model = AutoModel.from_config(config.codec_config)

        self.config = config

    def codec_decode(self, fine_output, output_lengths=None):
        """Turn quantized audio codes into audio array using encodec."""
        fine_output = fine_output.swapaxes(0, 1)
        emb = self.codec_model.quantizer.decode(fine_output)

        if output_lengths is not None:
            # encodec uses LSTMs which behaves differently with appended padding
            # decoding with encodec takes around 0.1% of the total generation time
            # to keep generation quality, we break batching
            out = [sample[:, :l].unsqueeze(0) for (sample, l) in zip(emb, output_lengths)]
            audio_arr = [self.codec_model.decoder(sample).squeeze() for sample in out]
        else:
            out = self.codec_model.decoder(emb)
            audio_arr = out.squeeze(1)  # squeeze the codebook dimension

        return audio_arr

    def generate(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        history_prompt: Optional[Dict[str, mindspore.Tensor]] = None,
        return_output_lengths: Optional[bool] = None,
        **kwargs,
    ) -> mindspore.Tensor:
        """
        Generates audio from an input prompt and an additional optional `Bark` speaker prompt.

        Args:
            input_ids (`Optional[mindspore.Tensor]` of shape (batch_size, seq_len), *optional*):
                Input ids. Will be truncated up to 256 tokens. Note that the output audios will be as long as the
                longest generation among the batch.
            history_prompt (`Optional[Dict[str,mindspore.Tensor]]`, *optional*):
                Optional `Bark` speaker prompt. Note that for now, this model takes only one speaker prompt per batch.
            kwargs (*optional*):
                Remaining dictionary of keyword arguments. Keyword arguments are of two types:

                - Without a prefix, they will be entered as `**kwargs` for the `generate` method of each sub-model.
                - With a *semantic_*, *coarse_*, *fine_* prefix, they will be input for the `generate` method of the
                semantic, coarse and fine respectively. It has the priority over the keywords without a prefix.

                This means you can, for example, specify a generation strategy for all sub-models except one.
            return_output_lengths (`bool`, *optional*):
                Whether or not to return the waveform lengths. Useful when batching.

        Returns:
            mindspore.Tensor:
                By default:

                - **audio_waveform** (`mindspore.Tensor` of shape (batch_size, seq_len)): Generated audio waveform.

                When `return_output_lengths=True`:

                - Returns a tuple made of:
                - **audio_waveform** (`mindspore.Tensor` of shape (batch_size, seq_len)): Generated audio waveform.
                - **output_lengths** (`mindspore.Tensor` of shape (batch_size)): The length of each waveform in the batch

        Example:
            ```python
            >>> from transformers import AutoProcessor, BarkModel
            ...
            >>> processor = AutoProcessor.from_pretrained("suno/bark-small")
            >>> model = BarkModel.from_pretrained("suno/bark-small")
            ...
            >>> # To add a voice preset, you can pass `voice_preset` to `BarkProcessor.__call__(...)`
            >>> voice_preset = "v2/en_speaker_6"
            ...
            >>> inputs = processor("Hello, my dog is cute, I need him in my life", voice_preset=voice_preset)
            ...
            >>> audio_array = model.generate(**inputs, semantic_max_new_tokens=100)
            >>> audio_array = audio_array.cpu().numpy().squeeze()
            ```
        """
        # TODO (joao):workaround until nested generation config is compatible with PreTrained Model
        # todo: dict
        semantic_generation_config = BarkSemanticGenerationConfig(**self.generation_config.semantic_config)
        coarse_generation_config = BarkCoarseGenerationConfig(**self.generation_config.coarse_acoustics_config)
        fine_generation_config = BarkFineGenerationConfig(**self.generation_config.fine_acoustics_config)

        kwargs_semantic = {
            # if "attention_mask" is set, it should not be passed to CoarseModel and FineModel
            "attention_mask": kwargs.pop("attention_mask", None),
            "min_eos_p": kwargs.pop("min_eos_p", None),
        }
        kwargs_coarse = {}
        kwargs_fine = {}
        for key, value in kwargs.items():
            if key.startswith("semantic_"):
                key = key[len("semantic_") :]
                kwargs_semantic[key] = value
            elif key.startswith("coarse_"):
                key = key[len("coarse_") :]
                kwargs_coarse[key] = value
            elif key.startswith("fine_"):
                key = key[len("fine_") :]
                kwargs_fine[key] = value
            else:
                # If the key is already in a specific config, then it's been set with a
                # submodules specific value and we don't override
                if key not in kwargs_semantic:
                    kwargs_semantic[key] = value
                if key not in kwargs_coarse:
                    kwargs_coarse[key] = value
                if key not in kwargs_fine:
                    kwargs_fine[key] = value

        # 1. Generate from the semantic model
        semantic_output = self.semantic.generate(
            input_ids,
            history_prompt=history_prompt,
            semantic_generation_config=semantic_generation_config,
            **kwargs_semantic,
        )

        # 2. Generate from the coarse model
        coarse_output = self.coarse_acoustics.generate(
            semantic_output,
            history_prompt=history_prompt,
            semantic_generation_config=semantic_generation_config,
            coarse_generation_config=coarse_generation_config,
            codebook_size=self.generation_config.codebook_size,
            return_output_lengths=return_output_lengths,
            **kwargs_coarse,
        )

        output_lengths = None
        if return_output_lengths:
            coarse_output, output_lengths = coarse_output
            # (batch_size, seq_len*coarse_codebooks) -> (batch_size, seq_len)
            output_lengths = output_lengths // coarse_generation_config.n_coarse_codebooks

        # 3. "generate" from the fine model
        output = self.fine_acoustics.generate(
            coarse_output,
            history_prompt=history_prompt,
            semantic_generation_config=semantic_generation_config,
            coarse_generation_config=coarse_generation_config,
            fine_generation_config=fine_generation_config,
            codebook_size=self.generation_config.codebook_size,
            **kwargs_fine,
        )

        if getattr(self, "fine_acoustics_hook", None) is not None:
            # Manually offload fine_acoustics to CPU
            # and load codec_model to GPU
            # since bark doesn't use codec_model forward pass
            self.fine_acoustics_hook.offload()
            self.codec_model = self.codec_model

        # 4. Decode the output and generate audio array
        audio = self.codec_decode(output, output_lengths)

        if getattr(self, "codec_model_hook", None) is not None:
            # Offload codec_model to CPU
            self.codec_model_hook.offload()

        # if return_output_lengths:
        #     output_lengths = [len(sample) for sample in audio]
        #     audio = nn.utils.rnn.pad_sequence(audio, batch_first=True, padding_value=0)
        #     return audio, output_lengths

        return audio

mindnlp.transformers.models.bark.modeling_bark.BarkModel.__init__(config)

Initializes a new instance of BarkModel.

PARAMETER	DESCRIPTION
self	The current instance of the BarkModel class. TYPE: BarkModel
config	A dictionary containing configuration settings for the BarkModel. TYPE: dict

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/bark/modeling_bark.py

def __init__(self, config):
    """
    Initializes a new instance of BarkModel.

    Args:
        self (BarkModel): The current instance of the BarkModel class.
        config (dict): A dictionary containing configuration settings for the BarkModel.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(config)

    self.semantic = BarkSemanticModel(config.semantic_config)
    self.coarse_acoustics = BarkCoarseModel(config.coarse_acoustics_config)
    self.fine_acoustics = BarkFineModel(config.fine_acoustics_config)

    self.codec_model = AutoModel.from_config(config.codec_config)

    self.config = config

mindnlp.transformers.models.bark.modeling_bark.BarkModel.codec_decode(fine_output, output_lengths=None)

Turn quantized audio codes into audio array using encodec.

Source code in mindnlp/transformers/models/bark/modeling_bark.py

def codec_decode(self, fine_output, output_lengths=None):
    """Turn quantized audio codes into audio array using encodec."""
    fine_output = fine_output.swapaxes(0, 1)
    emb = self.codec_model.quantizer.decode(fine_output)

    if output_lengths is not None:
        # encodec uses LSTMs which behaves differently with appended padding
        # decoding with encodec takes around 0.1% of the total generation time
        # to keep generation quality, we break batching
        out = [sample[:, :l].unsqueeze(0) for (sample, l) in zip(emb, output_lengths)]
        audio_arr = [self.codec_model.decoder(sample).squeeze() for sample in out]
    else:
        out = self.codec_model.decoder(emb)
        audio_arr = out.squeeze(1)  # squeeze the codebook dimension

    return audio_arr

mindnlp.transformers.models.bark.modeling_bark.BarkModel.generate(input_ids=None, history_prompt=None, return_output_lengths=None, **kwargs)

Generates audio from an input prompt and an additional optional Bark speaker prompt.

PARAMETER	DESCRIPTION
input_ids	Input ids. Will be truncated up to 256 tokens. Note that the output audios will be as long as the longest generation among the batch. TYPE: `Optional[mindspore.Tensor]` of shape (batch_size, seq_len), *optional* DEFAULT: None
history_prompt	Optional Bark speaker prompt. Note that for now, this model takes only one speaker prompt per batch. TYPE: `Optional[Dict[str,mindspore.Tensor]]`, *optional* DEFAULT: None
kwargs	Remaining dictionary of keyword arguments. Keyword arguments are of two types: Without a prefix, they will be entered as **kwargs for the generate method of each sub-model. With a semantic_, coarse_, fine_ prefix, they will be input for the generate method of the semantic, coarse and fine respectively. It has the priority over the keywords without a prefix. This means you can, for example, specify a generation strategy for all sub-models except one. TYPE: *optional* DEFAULT: {}
return_output_lengths	Whether or not to return the waveform lengths. Useful when batching. TYPE: `bool`, *optional* DEFAULT: None

RETURNS	DESCRIPTION
Tensor	mindspore.Tensor: By default: audio_waveform (mindspore.Tensor of shape (batch_size, seq_len)): Generated audio waveform. When return_output_lengths=True: Returns a tuple made of: audio_waveform (mindspore.Tensor of shape (batch_size, seq_len)): Generated audio waveform. output_lengths (mindspore.Tensor of shape (batch_size)): The length of each waveform in the batch

Example

>>> from transformers import AutoProcessor, BarkModel
...
>>> processor = AutoProcessor.from_pretrained("suno/bark-small")
>>> model = BarkModel.from_pretrained("suno/bark-small")
...
>>> # To add a voice preset, you can pass `voice_preset` to `BarkProcessor.__call__(...)`
>>> voice_preset = "v2/en_speaker_6"
...
>>> inputs = processor("Hello, my dog is cute, I need him in my life", voice_preset=voice_preset)
...
>>> audio_array = model.generate(**inputs, semantic_max_new_tokens=100)
>>> audio_array = audio_array.cpu().numpy().squeeze()

Source code in mindnlp/transformers/models/bark/modeling_bark.py

def generate(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    history_prompt: Optional[Dict[str, mindspore.Tensor]] = None,
    return_output_lengths: Optional[bool] = None,
    **kwargs,
) -> mindspore.Tensor:
    """
    Generates audio from an input prompt and an additional optional `Bark` speaker prompt.

    Args:
        input_ids (`Optional[mindspore.Tensor]` of shape (batch_size, seq_len), *optional*):
            Input ids. Will be truncated up to 256 tokens. Note that the output audios will be as long as the
            longest generation among the batch.
        history_prompt (`Optional[Dict[str,mindspore.Tensor]]`, *optional*):
            Optional `Bark` speaker prompt. Note that for now, this model takes only one speaker prompt per batch.
        kwargs (*optional*):
            Remaining dictionary of keyword arguments. Keyword arguments are of two types:

            - Without a prefix, they will be entered as `**kwargs` for the `generate` method of each sub-model.
            - With a *semantic_*, *coarse_*, *fine_* prefix, they will be input for the `generate` method of the
            semantic, coarse and fine respectively. It has the priority over the keywords without a prefix.

            This means you can, for example, specify a generation strategy for all sub-models except one.
        return_output_lengths (`bool`, *optional*):
            Whether or not to return the waveform lengths. Useful when batching.

    Returns:
        mindspore.Tensor:
            By default:

            - **audio_waveform** (`mindspore.Tensor` of shape (batch_size, seq_len)): Generated audio waveform.

            When `return_output_lengths=True`:

            - Returns a tuple made of:
            - **audio_waveform** (`mindspore.Tensor` of shape (batch_size, seq_len)): Generated audio waveform.
            - **output_lengths** (`mindspore.Tensor` of shape (batch_size)): The length of each waveform in the batch

    Example:
        ```python
        >>> from transformers import AutoProcessor, BarkModel
        ...
        >>> processor = AutoProcessor.from_pretrained("suno/bark-small")
        >>> model = BarkModel.from_pretrained("suno/bark-small")
        ...
        >>> # To add a voice preset, you can pass `voice_preset` to `BarkProcessor.__call__(...)`
        >>> voice_preset = "v2/en_speaker_6"
        ...
        >>> inputs = processor("Hello, my dog is cute, I need him in my life", voice_preset=voice_preset)
        ...
        >>> audio_array = model.generate(**inputs, semantic_max_new_tokens=100)
        >>> audio_array = audio_array.cpu().numpy().squeeze()
        ```
    """
    # TODO (joao):workaround until nested generation config is compatible with PreTrained Model
    # todo: dict
    semantic_generation_config = BarkSemanticGenerationConfig(**self.generation_config.semantic_config)
    coarse_generation_config = BarkCoarseGenerationConfig(**self.generation_config.coarse_acoustics_config)
    fine_generation_config = BarkFineGenerationConfig(**self.generation_config.fine_acoustics_config)

    kwargs_semantic = {
        # if "attention_mask" is set, it should not be passed to CoarseModel and FineModel
        "attention_mask": kwargs.pop("attention_mask", None),
        "min_eos_p": kwargs.pop("min_eos_p", None),
    }
    kwargs_coarse = {}
    kwargs_fine = {}
    for key, value in kwargs.items():
        if key.startswith("semantic_"):
            key = key[len("semantic_") :]
            kwargs_semantic[key] = value
        elif key.startswith("coarse_"):
            key = key[len("coarse_") :]
            kwargs_coarse[key] = value
        elif key.startswith("fine_"):
            key = key[len("fine_") :]
            kwargs_fine[key] = value
        else:
            # If the key is already in a specific config, then it's been set with a
            # submodules specific value and we don't override
            if key not in kwargs_semantic:
                kwargs_semantic[key] = value
            if key not in kwargs_coarse:
                kwargs_coarse[key] = value
            if key not in kwargs_fine:
                kwargs_fine[key] = value

    # 1. Generate from the semantic model
    semantic_output = self.semantic.generate(
        input_ids,
        history_prompt=history_prompt,
        semantic_generation_config=semantic_generation_config,
        **kwargs_semantic,
    )

    # 2. Generate from the coarse model
    coarse_output = self.coarse_acoustics.generate(
        semantic_output,
        history_prompt=history_prompt,
        semantic_generation_config=semantic_generation_config,
        coarse_generation_config=coarse_generation_config,
        codebook_size=self.generation_config.codebook_size,
        return_output_lengths=return_output_lengths,
        **kwargs_coarse,
    )

    output_lengths = None
    if return_output_lengths:
        coarse_output, output_lengths = coarse_output
        # (batch_size, seq_len*coarse_codebooks) -> (batch_size, seq_len)
        output_lengths = output_lengths // coarse_generation_config.n_coarse_codebooks

    # 3. "generate" from the fine model
    output = self.fine_acoustics.generate(
        coarse_output,
        history_prompt=history_prompt,
        semantic_generation_config=semantic_generation_config,
        coarse_generation_config=coarse_generation_config,
        fine_generation_config=fine_generation_config,
        codebook_size=self.generation_config.codebook_size,
        **kwargs_fine,
    )

    if getattr(self, "fine_acoustics_hook", None) is not None:
        # Manually offload fine_acoustics to CPU
        # and load codec_model to GPU
        # since bark doesn't use codec_model forward pass
        self.fine_acoustics_hook.offload()
        self.codec_model = self.codec_model

    # 4. Decode the output and generate audio array
    audio = self.codec_decode(output, output_lengths)

    if getattr(self, "codec_model_hook", None) is not None:
        # Offload codec_model to CPU
        self.codec_model_hook.offload()

    # if return_output_lengths:
    #     output_lengths = [len(sample) for sample in audio]
    #     audio = nn.utils.rnn.pad_sequence(audio, batch_first=True, padding_value=0)
    #     return audio, output_lengths

    return audio

mindnlp.transformers.models.bark.modeling_bark.BarkPreTrainedModel

Bases: PreTrainedModel

An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models.

Source code in mindnlp/transformers/models/bark/modeling_bark.py

class BarkPreTrainedModel(PreTrainedModel):
    """
    An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained
    models.
    """
    config_class = BarkConfig
    supports_gradient_checkpointing = False

    def _init_weights(self, cell):
        """Initialize the weights"""
        if isinstance(cell, nn.Linear):
            # Slightly different from the TF version which uses truncated_normal for initialization
            # cf https://github.com/pytorch/pytorch/pull/5617
            cell.weight.set_data(initializer(Normal(self.config.initializer_range),
                                                    cell.weight.shape, cell.weight.dtype))
            if cell.bias is not None:
                cell.bias.set_data(initializer('zeros', cell.bias.shape, cell.bias.dtype))
        elif isinstance(cell, nn.Embedding):
            weight = np.random.normal(0.0, self.config.initializer_range, cell.weight.shape)
            if cell.padding_idx:
                weight[cell.padding_idx] = 0

            cell.weight.set_data(Tensor(weight, cell.weight.dtype))
        elif isinstance(cell, nn.LayerNorm):
            cell.weight.set_data(initializer('ones', cell.weight.shape, cell.weight.dtype))
            cell.bias.set_data(initializer('zeros', cell.bias.shape, cell.bias.dtype))

mindnlp.transformers.models.bark.modeling_bark.BarkCausalModel

Bases: BarkPreTrainedModel

The BarkCausalModel class is a subclass of BarkPreTrainedModel and represents a model for causal language modeling using the Bark framework.

ATTRIBUTE	DESCRIPTION
`config`	An instance of the BarkConfig class containing the model configuration.
`input_embeds_layer`	An embedding layer for the input vocabulary.
`position_embeds_layer`	An embedding layer for the position indices.
`drop`	A dropout layer.
`layers`	A list of BarkBlock layers for the model.
`layernorm_final`	A layer normalization module for the final hidden states.
`lm_head`	A dense layer for generating the output vocabulary logits.
`gradient_checkpointing`	A boolean indicating whether gradient checkpointing is enabled.

METHOD	DESCRIPTION
`__init__	Initializes the BarkCausalModel instance.
`get_input_embeddings	Returns the input embedding layer.
`set_input_embeddings	Sets the input embedding layer.
`prepare_inputs_for_generation	Prepares the inputs for generation.
`_reorder_cache	Reorders the cache for beam search or beam sampling.

Note

This docstring provides an overview of the class and its methods. For detailed information on each method, please refer to the corresponding method's docstring.

Source code in mindnlp/transformers/models/bark/modeling_bark.py

class BarkCausalModel(BarkPreTrainedModel):

    """
    The `BarkCausalModel` class is a subclass of `BarkPreTrainedModel` and represents a model
    for causal language modeling using the Bark framework.

    Attributes:
        `config`: An instance of the `BarkConfig` class containing the model configuration.
        `input_embeds_layer`: An embedding layer for the input vocabulary.
        `position_embeds_layer`: An embedding layer for the position indices.
        `drop`: A dropout layer.
        `layers`: A list of `BarkBlock` layers for the model.
        `layernorm_final`: A layer normalization module for the final hidden states.
        `lm_head`: A dense layer for generating the output vocabulary logits.
        `gradient_checkpointing`: A boolean indicating whether gradient checkpointing is enabled.

    Methods:
        `__init__(self, config)`: Initializes the `BarkCausalModel` instance.
        `get_input_embeddings(self)`: Returns the input embedding layer.
        `set_input_embeddings(self, new_embeddings)`: Sets the input embedding layer.
        `prepare_inputs_for_generation(self, input_ids, past_key_values=None, **kwargs)`: Prepares the inputs for generation.
        `forward(self, input_ids, past_key_values=None, attention_mask=None, position_ids=None, head_mask=None, labels=None, input_embeds=None, use_cache=None, output_attentions=None,
            output_hidden_states=None, return_dict=None)`: forwards the model output based on the provided inputs.
        `_reorder_cache(past_key_values, beam_idx)`: Reorders the cache for beam search or beam sampling.

    Note:
        This docstring provides an overview of the class and its methods.
        For detailed information on each method, please refer to the corresponding method's docstring.
    """
    config_class = BarkSubModelConfig

    def __init__(self, config):
        """
        Initializes an instance of the BarkCausalModel class.

        Args:
            self: The instance of the class.
            config (object):
                An object containing configuration parameters for the model.

                - input_vocab_size (int): The size of the input vocabulary.
                - hidden_size (int): The size of the hidden state.
                - block_size (int): The size of the block.
                - dropout (float): The dropout probability.
                - num_layers (int): The number of layers.
                - bias (bool): Whether to apply bias in BarkLayerNorm.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(config)
        self.config = config

        # initialize as an autoregressive GPT-like model
        self.input_embeds_layer = nn.Embedding(config.input_vocab_size, config.hidden_size)
        self.position_embeds_layer = nn.Embedding(config.block_size, config.hidden_size)

        self.drop = nn.Dropout(p=config.dropout)

        self.layers = nn.ModuleList([BarkBlock(config, is_causal=True) for _ in range(config.num_layers)])

        self.layernorm_final = BarkLayerNorm(config.hidden_size, bias=config.bias)

        self.lm_head = nn.Linear(config.hidden_size, config.output_vocab_size, bias=False)
        self.gradient_checkpointing = False

        # Initialize weights and apply final processing
        self.post_init()

    def get_input_embeddings(self):
        """
        This method retrieves the input embeddings from the BarkCausalModel.

        Args:
            self (BarkCausalModel): The instance of the BarkCausalModel class.

        Returns:
            None: This method returns the input embeddings layer of the BarkCausalModel.

        Raises:
            This method does not raise any exceptions.
        """
        return self.input_embeds_layer

    def set_input_embeddings(self, new_embeddings):
        """
        Set input embeddings for the BarkCausalModel.

        Args:
            self (BarkCausalModel): The instance of BarkCausalModel.
            new_embeddings (any): The new input embeddings to be set for the model. It can be of any type.

        Returns:
            None.

        Raises:
            None.
        """
        self.input_embeds_layer = new_embeddings

    def prepare_inputs_for_generation(self, input_ids, past_key_values=None, **kwargs):
        """
        This method prepares inputs for generation in the BarkCausalModel class.

        Args:
            self (object): The instance of the class.
            input_ids (tensor): The input tensor containing the tokenized input sequence.
            past_key_values (tuple, optional): The past key values for fast decoding. Defaults to None.

        Returns:
            dict or None: A dictionary containing the prepared input values for generation,
            including the input_ids, input_embeds, past_key_values, use_cache, position_ids, and attention_mask.
            Returns None if input_embeds is not provided and use_cache is False.

        Raises:
            ValueError: If the input_ids shape is incompatible with past_key_values.
            ValueError: If the input_embeds shape is incompatible with use_cache.
            TypeError: If the attention_mask and position_ids shapes are not compatible.
            RuntimeError: If there are issues with calculating position_ids based on attention_mask.
        """
        input_embeds = kwargs.get("input_embeds", None)

        attention_mask = kwargs.get("attention_mask", None)
        position_ids = kwargs.get("position_ids", None)

        if past_key_values is not None:
            # Omit tokens covered by past_key_values
            seq_len = input_ids.shape[1]
            past_length = past_key_values[0][0].shape[2]

            # Some generation methods already pass only the last input ID
            if input_ids.shape[1] > past_length:
                remove_prefix_length = past_length
            else:
                # Default to old behavior: keep only final ID
                remove_prefix_length = input_ids.shape[1] - 1

            input_ids = input_ids[:, remove_prefix_length:]

            # input_embeds have already been used and is not required anymore
            input_embeds = None
        else:
            if input_embeds is not None and kwargs.get("use_cache"):
                seq_len = input_embeds.shape[1]
            else:
                seq_len = input_ids.shape[1]

        # ensure that attention_mask and position_ids shapes are aligned with the weird Bark hack of reducing
        # sequence length on the first forward pass
        if attention_mask is not None:
            attention_mask = attention_mask[:, :seq_len]
        if position_ids is not None:
            position_ids = position_ids[:, :seq_len]

        if attention_mask is not None and position_ids is None:
            # create position_ids on the fly for batch generation
            position_ids = attention_mask.long().cumsum(-1) - 1
            position_ids = position_ids.masked_fill(attention_mask == 0, 1)
            if past_key_values:
                position_ids = position_ids[:, -input_ids.shape[1] :]
        else:
            position_ids = None

        if input_embeds is not None and kwargs.get("use_cache"):
            return {
                "input_ids": None,
                "input_embeds": input_embeds,
                "past_key_values": past_key_values,
                "use_cache": kwargs.get("use_cache"),
                "position_ids": position_ids,
                "attention_mask": attention_mask,
            }
        return {
            "input_ids": input_ids,
            "past_key_values": past_key_values,
            "use_cache": kwargs.get("use_cache"),
            "position_ids": position_ids,
            "attention_mask": attention_mask,
        }

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        past_key_values: Optional[Tuple[mindspore.Tensor]] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        position_ids: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        labels: Optional[mindspore.Tensor] = None,
        input_embeds: Optional[mindspore.Tensor] = None,
        use_cache: Optional[bool] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple[mindspore.Tensor], CausalLMOutputWithPast]:
        '''
        forwards the BarkCausalModel.

        Args:
            self: The object itself.
            input_ids (Optional[mindspore.Tensor]):
                The input tensor of shape (batch_size, sequence_length). Defaults to None.
            past_key_values (Optional[Tuple[mindspore.Tensor]]):
                The past key values tensor of shape (batch_size, num_heads, sequence_length, embed_size_per_head).
                Defaults to None.
            attention_mask (Optional[mindspore.Tensor]):
                The attention mask tensor of shape (batch_size, sequence_length). Defaults to None.
            position_ids (Optional[mindspore.Tensor]):
                The position ids tensor of shape (batch_size, sequence_length). Defaults to None.
            head_mask (Optional[mindspore.Tensor]):
                The head mask tensor of shape (num_heads, sequence_length, sequence_length). Defaults to None.
            labels (Optional[mindspore.Tensor]):
                The labels tensor of shape (batch_size, sequence_length). Defaults to None.
            input_embeds (Optional[mindspore.Tensor]):
                The input embeddings tensor of shape (batch_size, sequence_length, embed_size). Defaults to None.
            use_cache (Optional[bool]): Whether to use cache. Defaults to None.
            output_attentions (Optional[bool]): Whether to output attentions. Defaults to None.
            output_hidden_states (Optional[bool]): Whether to output hidden states. Defaults to None.
            return_dict (Optional[bool]): Whether to return a dictionary. Defaults to None.

        Returns:
            Union[Tuple[mindspore.Tensor], CausalLMOutputWithPast]:
                The output of the model. It can be a tuple containing the following elements:

                - loss (mindspore.Tensor): The loss tensor.
                - logits (mindspore.Tensor): The logits tensor.
                - past_key_values (Tuple[mindspore.Tensor]): The past key values tensor.
                - hidden_states (Tuple[mindspore.Tensor]): The hidden states tensor.
                - attentions (Tuple[mindspore.Tensor]): The attentions tensor.
            or an instance of the CausalLMOutputWithPast class.

        Raises:
            ValueError: If both input_ids and input_embeds are specified.
            ValueError: If batch_size is not defined or <= 0.
            NotImplementedError: If training is not implemented yet for Bark - ensure you do not pass labels to the model.
            '''
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )
        use_cache = use_cache if use_cache is not None else self.config.use_cache
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        # Verify if input_embeds already exists
        # then compute embeddings.
        if input_ids is not None and input_embeds is not None:
            raise ValueError("You cannot specify both input_ids and input_embeds at the same time")
        if input_embeds is not None and past_key_values is None:
            # we want to return the input_embeds in priority so that it is in line with a weird hack
            # of Bark which concatenate two bits of the input_embeds on the first forward pass of the semantic model
            pass
        elif input_ids is not None:
            input_embeds = self.input_embeds_layer(input_ids)  # token embeddings of shape (b, t, n_embd)
        elif input_embeds is not None:
            pass
        else:
            raise ValueError("You have to specify either input_ids or input_embeds")

        input_shape = input_embeds.shape[:-1]
        batch_size = input_embeds.shape[0]
        seq_length = input_shape[-1]

        if past_key_values is None:
            past_length = 0
            past_key_values = tuple([None] * len(self.layers))
        else:
            past_length = past_key_values[0][0].shape[-2]

        if position_ids is None:
            position_ids = ops.arange(past_length, seq_length + past_length, dtype=mindspore.int64)
            position_ids = position_ids.unsqueeze(0)  # shape (1, seq_length)

        position_embeds = self.position_embeds_layer(position_ids)  # position embeddings of shape (1, t, n_embd)

        # Attention mask.
        if attention_mask is not None:
            if batch_size <= 0:
                raise ValueError("batch_size has to be defined and > 0")
            attention_mask = attention_mask.view(batch_size, -1)
            # [bsz, to_seq_length] -> [bsz, 1, 1, to_seq_length]
            # from_seq_length is 1 to easily broadcast
            attention_mask = _prepare_4d_attention_mask(attention_mask, input_embeds.dtype, tgt_len=1)

        # Prepare head mask if needed
        # 1.0 in head_mask indicate we keep the head
        # attention_probs has shape bsz x num_heads x N x N
        # head_mask has shape num_layers x batch x num_heads x N x N
        head_mask = self.get_head_mask(head_mask, self.config.num_layers)

        hidden_states = self.drop(input_embeds + position_embeds)
        output_shape = input_shape + (hidden_states.shape[-1],)

        if self.gradient_checkpointing and self.training:
            if use_cache:
                logger.warning_once(
                    "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
                )
                use_cache = False

        present_key_values = () if use_cache else None
        all_self_attentions = () if output_attentions else None
        all_hidden_states = () if output_hidden_states else None

        for i, (block, past_layer_key_values) in enumerate(zip(self.layers, past_key_values)):
            if output_hidden_states:
                all_hidden_states = all_hidden_states + (hidden_states,)

            outputs = block(
                hidden_states,
                past_key_values=past_layer_key_values,
                attention_mask=attention_mask,
                head_mask=head_mask[i],
                use_cache=use_cache,
                output_attentions=output_attentions,
            )

            hidden_states = outputs[0]

            if use_cache:
                present_key_values = present_key_values + (outputs[1],)

            if output_attentions:
                all_self_attentions = all_self_attentions + (outputs[2 if use_cache else 1],)

        hidden_states = self.layernorm_final(hidden_states)

        hidden_states = hidden_states.view(output_shape)

        # Add last hidden state
        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        logits = self.lm_head(hidden_states)

        loss = None
        if labels is not None:
            raise NotImplementedError(
                "Training is not implemented yet for Bark - ensure you do not pass `labels` to the model."
            )

        if not return_dict:
            return tuple(
                v for v in [None, logits, present_key_values, all_hidden_states, all_self_attentions] if v is not None
            )

        return CausalLMOutputWithPast(
            loss=loss,
            logits=logits,
            past_key_values=present_key_values,
            hidden_states=all_hidden_states,
            attentions=all_self_attentions,
        )

    @staticmethod
    def _reorder_cache(
        past_key_values: Tuple[Tuple[mindspore.Tensor]], beam_idx: mindspore.Tensor
    ) -> Tuple[Tuple[mindspore.Tensor]]:
        """
        This function is used to re-order the `past_key_values` cache if [`~PreTrainedModel.beam_search`] or
        [`~PreTrainedModel.beam_sample`] is called. This is required to match `past_key_values` with the correct
        beam_idx at every generation step.
        """
        # Necessary for beam_search
        return tuple(
            tuple(past_state.index_select(0, beam_idx) for past_state in layer_past)
            for layer_past in past_key_values
        )

mindnlp.transformers.models.bark.modeling_bark.BarkCausalModel.__init__(config)

Initializes an instance of the BarkCausalModel class.

PARAMETER	DESCRIPTION
self	The instance of the class.
config	An object containing configuration parameters for the model. input_vocab_size (int): The size of the input vocabulary. hidden_size (int): The size of the hidden state. block_size (int): The size of the block. dropout (float): The dropout probability. num_layers (int): The number of layers. bias (bool): Whether to apply bias in BarkLayerNorm. TYPE: object

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/bark/modeling_bark.py

def __init__(self, config):
    """
    Initializes an instance of the BarkCausalModel class.

    Args:
        self: The instance of the class.
        config (object):
            An object containing configuration parameters for the model.

            - input_vocab_size (int): The size of the input vocabulary.
            - hidden_size (int): The size of the hidden state.
            - block_size (int): The size of the block.
            - dropout (float): The dropout probability.
            - num_layers (int): The number of layers.
            - bias (bool): Whether to apply bias in BarkLayerNorm.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(config)
    self.config = config

    # initialize as an autoregressive GPT-like model
    self.input_embeds_layer = nn.Embedding(config.input_vocab_size, config.hidden_size)
    self.position_embeds_layer = nn.Embedding(config.block_size, config.hidden_size)

    self.drop = nn.Dropout(p=config.dropout)

    self.layers = nn.ModuleList([BarkBlock(config, is_causal=True) for _ in range(config.num_layers)])

    self.layernorm_final = BarkLayerNorm(config.hidden_size, bias=config.bias)

    self.lm_head = nn.Linear(config.hidden_size, config.output_vocab_size, bias=False)
    self.gradient_checkpointing = False

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.bark.modeling_bark.BarkCausalModel.forward(input_ids=None, past_key_values=None, attention_mask=None, position_ids=None, head_mask=None, labels=None, input_embeds=None, use_cache=None, output_attentions=None, output_hidden_states=None, return_dict=None)

forwards the BarkCausalModel.

PARAMETER	DESCRIPTION
self	The object itself.
input_ids	The input tensor of shape (batch_size, sequence_length). Defaults to None. TYPE: Optional[Tensor] DEFAULT: None
past_key_values	The past key values tensor of shape (batch_size, num_heads, sequence_length, embed_size_per_head). Defaults to None. TYPE: Optional[Tuple[Tensor]] DEFAULT: None
attention_mask	The attention mask tensor of shape (batch_size, sequence_length). Defaults to None. TYPE: Optional[Tensor] DEFAULT: None
position_ids	The position ids tensor of shape (batch_size, sequence_length). Defaults to None. TYPE: Optional[Tensor] DEFAULT: None
head_mask	The head mask tensor of shape (num_heads, sequence_length, sequence_length). Defaults to None. TYPE: Optional[Tensor] DEFAULT: None
labels	The labels tensor of shape (batch_size, sequence_length). Defaults to None. TYPE: Optional[Tensor] DEFAULT: None
input_embeds	The input embeddings tensor of shape (batch_size, sequence_length, embed_size). Defaults to None. TYPE: Optional[Tensor] DEFAULT: None
use_cache	Whether to use cache. Defaults to None. TYPE: Optional[bool] DEFAULT: None
output_attentions	Whether to output attentions. Defaults to None. TYPE: Optional[bool] DEFAULT: None
output_hidden_states	Whether to output hidden states. Defaults to None. TYPE: Optional[bool] DEFAULT: None
return_dict	Whether to return a dictionary. Defaults to None. TYPE: Optional[bool] DEFAULT: None

RETURNS	DESCRIPTION
Union[Tuple[Tensor], CausalLMOutputWithPast]	Union[Tuple[mindspore.Tensor], CausalLMOutputWithPast]: The output of the model. It can be a tuple containing the following elements: loss (mindspore.Tensor): The loss tensor. logits (mindspore.Tensor): The logits tensor. past_key_values (Tuple[mindspore.Tensor]): The past key values tensor. hidden_states (Tuple[mindspore.Tensor]): The hidden states tensor. attentions (Tuple[mindspore.Tensor]): The attentions tensor.
Union[Tuple[Tensor], CausalLMOutputWithPast]	or an instance of the CausalLMOutputWithPast class.

RAISES	DESCRIPTION
ValueError	If both input_ids and input_embeds are specified.
ValueError	If batch_size is not defined or <= 0.
NotImplementedError	If training is not implemented yet for Bark - ensure you do not pass labels to the model.

Source code in mindnlp/transformers/models/bark/modeling_bark.py

def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    past_key_values: Optional[Tuple[mindspore.Tensor]] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    position_ids: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    labels: Optional[mindspore.Tensor] = None,
    input_embeds: Optional[mindspore.Tensor] = None,
    use_cache: Optional[bool] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple[mindspore.Tensor], CausalLMOutputWithPast]:
    '''
    forwards the BarkCausalModel.

    Args:
        self: The object itself.
        input_ids (Optional[mindspore.Tensor]):
            The input tensor of shape (batch_size, sequence_length). Defaults to None.
        past_key_values (Optional[Tuple[mindspore.Tensor]]):
            The past key values tensor of shape (batch_size, num_heads, sequence_length, embed_size_per_head).
            Defaults to None.
        attention_mask (Optional[mindspore.Tensor]):
            The attention mask tensor of shape (batch_size, sequence_length). Defaults to None.
        position_ids (Optional[mindspore.Tensor]):
            The position ids tensor of shape (batch_size, sequence_length). Defaults to None.
        head_mask (Optional[mindspore.Tensor]):
            The head mask tensor of shape (num_heads, sequence_length, sequence_length). Defaults to None.
        labels (Optional[mindspore.Tensor]):
            The labels tensor of shape (batch_size, sequence_length). Defaults to None.
        input_embeds (Optional[mindspore.Tensor]):
            The input embeddings tensor of shape (batch_size, sequence_length, embed_size). Defaults to None.
        use_cache (Optional[bool]): Whether to use cache. Defaults to None.
        output_attentions (Optional[bool]): Whether to output attentions. Defaults to None.
        output_hidden_states (Optional[bool]): Whether to output hidden states. Defaults to None.
        return_dict (Optional[bool]): Whether to return a dictionary. Defaults to None.

    Returns:
        Union[Tuple[mindspore.Tensor], CausalLMOutputWithPast]:
            The output of the model. It can be a tuple containing the following elements:

            - loss (mindspore.Tensor): The loss tensor.
            - logits (mindspore.Tensor): The logits tensor.
            - past_key_values (Tuple[mindspore.Tensor]): The past key values tensor.
            - hidden_states (Tuple[mindspore.Tensor]): The hidden states tensor.
            - attentions (Tuple[mindspore.Tensor]): The attentions tensor.
        or an instance of the CausalLMOutputWithPast class.

    Raises:
        ValueError: If both input_ids and input_embeds are specified.
        ValueError: If batch_size is not defined or <= 0.
        NotImplementedError: If training is not implemented yet for Bark - ensure you do not pass labels to the model.
        '''
    output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
    output_hidden_states = (
        output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
    )
    use_cache = use_cache if use_cache is not None else self.config.use_cache
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    # Verify if input_embeds already exists
    # then compute embeddings.
    if input_ids is not None and input_embeds is not None:
        raise ValueError("You cannot specify both input_ids and input_embeds at the same time")
    if input_embeds is not None and past_key_values is None:
        # we want to return the input_embeds in priority so that it is in line with a weird hack
        # of Bark which concatenate two bits of the input_embeds on the first forward pass of the semantic model
        pass
    elif input_ids is not None:
        input_embeds = self.input_embeds_layer(input_ids)  # token embeddings of shape (b, t, n_embd)
    elif input_embeds is not None:
        pass
    else:
        raise ValueError("You have to specify either input_ids or input_embeds")

    input_shape = input_embeds.shape[:-1]
    batch_size = input_embeds.shape[0]
    seq_length = input_shape[-1]

    if past_key_values is None:
        past_length = 0
        past_key_values = tuple([None] * len(self.layers))
    else:
        past_length = past_key_values[0][0].shape[-2]

    if position_ids is None:
        position_ids = ops.arange(past_length, seq_length + past_length, dtype=mindspore.int64)
        position_ids = position_ids.unsqueeze(0)  # shape (1, seq_length)

    position_embeds = self.position_embeds_layer(position_ids)  # position embeddings of shape (1, t, n_embd)

    # Attention mask.
    if attention_mask is not None:
        if batch_size <= 0:
            raise ValueError("batch_size has to be defined and > 0")
        attention_mask = attention_mask.view(batch_size, -1)
        # [bsz, to_seq_length] -> [bsz, 1, 1, to_seq_length]
        # from_seq_length is 1 to easily broadcast
        attention_mask = _prepare_4d_attention_mask(attention_mask, input_embeds.dtype, tgt_len=1)

    # Prepare head mask if needed
    # 1.0 in head_mask indicate we keep the head
    # attention_probs has shape bsz x num_heads x N x N
    # head_mask has shape num_layers x batch x num_heads x N x N
    head_mask = self.get_head_mask(head_mask, self.config.num_layers)

    hidden_states = self.drop(input_embeds + position_embeds)
    output_shape = input_shape + (hidden_states.shape[-1],)

    if self.gradient_checkpointing and self.training:
        if use_cache:
            logger.warning_once(
                "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
            )
            use_cache = False

    present_key_values = () if use_cache else None
    all_self_attentions = () if output_attentions else None
    all_hidden_states = () if output_hidden_states else None

    for i, (block, past_layer_key_values) in enumerate(zip(self.layers, past_key_values)):
        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        outputs = block(
            hidden_states,
            past_key_values=past_layer_key_values,
            attention_mask=attention_mask,
            head_mask=head_mask[i],
            use_cache=use_cache,
            output_attentions=output_attentions,
        )

        hidden_states = outputs[0]

        if use_cache:
            present_key_values = present_key_values + (outputs[1],)

        if output_attentions:
            all_self_attentions = all_self_attentions + (outputs[2 if use_cache else 1],)

    hidden_states = self.layernorm_final(hidden_states)

    hidden_states = hidden_states.view(output_shape)

    # Add last hidden state
    if output_hidden_states:
        all_hidden_states = all_hidden_states + (hidden_states,)

    logits = self.lm_head(hidden_states)

    loss = None
    if labels is not None:
        raise NotImplementedError(
            "Training is not implemented yet for Bark - ensure you do not pass `labels` to the model."
        )

    if not return_dict:
        return tuple(
            v for v in [None, logits, present_key_values, all_hidden_states, all_self_attentions] if v is not None
        )

    return CausalLMOutputWithPast(
        loss=loss,
        logits=logits,
        past_key_values=present_key_values,
        hidden_states=all_hidden_states,
        attentions=all_self_attentions,
    )

mindnlp.transformers.models.bark.modeling_bark.BarkCausalModel.get_input_embeddings()

This method retrieves the input embeddings from the BarkCausalModel.

PARAMETER	DESCRIPTION
self	The instance of the BarkCausalModel class. TYPE: BarkCausalModel

RETURNS	DESCRIPTION
None	This method returns the input embeddings layer of the BarkCausalModel.

Source code in mindnlp/transformers/models/bark/modeling_bark.py

def get_input_embeddings(self):
    """
    This method retrieves the input embeddings from the BarkCausalModel.

    Args:
        self (BarkCausalModel): The instance of the BarkCausalModel class.

    Returns:
        None: This method returns the input embeddings layer of the BarkCausalModel.

    Raises:
        This method does not raise any exceptions.
    """
    return self.input_embeds_layer

mindnlp.transformers.models.bark.modeling_bark.BarkCausalModel.prepare_inputs_for_generation(input_ids, past_key_values=None, **kwargs)

This method prepares inputs for generation in the BarkCausalModel class.

PARAMETER	DESCRIPTION
self	The instance of the class. TYPE: object
input_ids	The input tensor containing the tokenized input sequence. TYPE: tensor
past_key_values	The past key values for fast decoding. Defaults to None. TYPE: tuple DEFAULT: None

RETURNS	DESCRIPTION
	dict or None: A dictionary containing the prepared input values for generation,
	including the input_ids, input_embeds, past_key_values, use_cache, position_ids, and attention_mask.
	Returns None if input_embeds is not provided and use_cache is False.

RAISES	DESCRIPTION
ValueError	If the input_ids shape is incompatible with past_key_values.
ValueError	If the input_embeds shape is incompatible with use_cache.
TypeError	If the attention_mask and position_ids shapes are not compatible.
RuntimeError	If there are issues with calculating position_ids based on attention_mask.

Source code in mindnlp/transformers/models/bark/modeling_bark.py

def prepare_inputs_for_generation(self, input_ids, past_key_values=None, **kwargs):
    """
    This method prepares inputs for generation in the BarkCausalModel class.

    Args:
        self (object): The instance of the class.
        input_ids (tensor): The input tensor containing the tokenized input sequence.
        past_key_values (tuple, optional): The past key values for fast decoding. Defaults to None.

    Returns:
        dict or None: A dictionary containing the prepared input values for generation,
        including the input_ids, input_embeds, past_key_values, use_cache, position_ids, and attention_mask.
        Returns None if input_embeds is not provided and use_cache is False.

    Raises:
        ValueError: If the input_ids shape is incompatible with past_key_values.
        ValueError: If the input_embeds shape is incompatible with use_cache.
        TypeError: If the attention_mask and position_ids shapes are not compatible.
        RuntimeError: If there are issues with calculating position_ids based on attention_mask.
    """
    input_embeds = kwargs.get("input_embeds", None)

    attention_mask = kwargs.get("attention_mask", None)
    position_ids = kwargs.get("position_ids", None)

    if past_key_values is not None:
        # Omit tokens covered by past_key_values
        seq_len = input_ids.shape[1]
        past_length = past_key_values[0][0].shape[2]

        # Some generation methods already pass only the last input ID
        if input_ids.shape[1] > past_length:
            remove_prefix_length = past_length
        else:
            # Default to old behavior: keep only final ID
            remove_prefix_length = input_ids.shape[1] - 1

        input_ids = input_ids[:, remove_prefix_length:]

        # input_embeds have already been used and is not required anymore
        input_embeds = None
    else:
        if input_embeds is not None and kwargs.get("use_cache"):
            seq_len = input_embeds.shape[1]
        else:
            seq_len = input_ids.shape[1]

    # ensure that attention_mask and position_ids shapes are aligned with the weird Bark hack of reducing
    # sequence length on the first forward pass
    if attention_mask is not None:
        attention_mask = attention_mask[:, :seq_len]
    if position_ids is not None:
        position_ids = position_ids[:, :seq_len]

    if attention_mask is not None and position_ids is None:
        # create position_ids on the fly for batch generation
        position_ids = attention_mask.long().cumsum(-1) - 1
        position_ids = position_ids.masked_fill(attention_mask == 0, 1)
        if past_key_values:
            position_ids = position_ids[:, -input_ids.shape[1] :]
    else:
        position_ids = None

    if input_embeds is not None and kwargs.get("use_cache"):
        return {
            "input_ids": None,
            "input_embeds": input_embeds,
            "past_key_values": past_key_values,
            "use_cache": kwargs.get("use_cache"),
            "position_ids": position_ids,
            "attention_mask": attention_mask,
        }
    return {
        "input_ids": input_ids,
        "past_key_values": past_key_values,
        "use_cache": kwargs.get("use_cache"),
        "position_ids": position_ids,
        "attention_mask": attention_mask,
    }

mindnlp.transformers.models.bark.modeling_bark.BarkCausalModel.set_input_embeddings(new_embeddings)

Set input embeddings for the BarkCausalModel.

PARAMETER	DESCRIPTION
self	The instance of BarkCausalModel. TYPE: BarkCausalModel
new_embeddings	The new input embeddings to be set for the model. It can be of any type. TYPE: any

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/bark/modeling_bark.py

def set_input_embeddings(self, new_embeddings):
    """
    Set input embeddings for the BarkCausalModel.

    Args:
        self (BarkCausalModel): The instance of BarkCausalModel.
        new_embeddings (any): The new input embeddings to be set for the model. It can be of any type.

    Returns:
        None.

    Raises:
        None.
    """
    self.input_embeds_layer = new_embeddings

mindnlp.transformers.models.bark.processing_bark.BarkProcessor

Bases: ProcessorMixin

Constructs a Bark processor which wraps a text tokenizer and optional Bark voice presets into a single processor.

PARAMETER	DESCRIPTION
tokenizer	An instance of [PreTrainedTokenizer]. TYPE: [`PreTrainedTokenizer`]
speaker_embeddings	Optional nested speaker embeddings dictionary. The first level contains voice preset names (e.g "en_speaker_4"). The second level contains "semantic_prompt", "coarse_prompt" and "fine_prompt" embeddings. The values correspond to the path of the corresponding np.ndarray. See here for a list of voice_preset_names. TYPE: `Dict[Dict[str]]`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/bark/processing_bark.py

class BarkProcessor(ProcessorMixin):
    r"""
    Constructs a Bark processor which wraps a text tokenizer and optional Bark voice presets into a single processor.

    Args:
        tokenizer ([`PreTrainedTokenizer`]):
            An instance of [`PreTrainedTokenizer`].
        speaker_embeddings (`Dict[Dict[str]]`, *optional*):
            Optional nested speaker embeddings dictionary. The first level contains voice preset names (e.g
            `"en_speaker_4"`). The second level contains `"semantic_prompt"`, `"coarse_prompt"` and `"fine_prompt"`
            embeddings. The values correspond to the path of the corresponding `np.ndarray`. See
            [here](https://suno-ai.notion.site/8b8e8749ed514b0cbf3f699013548683?v=bc67cff786b04b50b3ceb756fd05f68c) for
            a list of `voice_preset_names`.

    """
    tokenizer_class = "AutoTokenizer"
    attributes = ["tokenizer"]

    preset_shape = {
        "semantic_prompt": 1,
        "coarse_prompt": 2,
        "fine_prompt": 2,
    }

    def __init__(self, tokenizer, speaker_embeddings=None):
        """Initialize the BarkProcessor class.

        Args:
            self (object): The instance of the class.
            tokenizer (object): The tokenizer object used for processing text data.
            speaker_embeddings (object or None, optional): The speaker embeddings associated with the text data. 
                Defaults to None.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(tokenizer)

        self.speaker_embeddings = speaker_embeddings

    @classmethod
    def from_pretrained(
        cls, pretrained_processor_name_or_path, speaker_embeddings_dict_path="speaker_embeddings_path.json", **kwargs
    ):
        r"""
        Instantiate a Bark processor associated with a pretrained model.

        Args:
            pretrained_model_name_or_path (`str` or `os.PathLike`):
                This can be either:

                - a string, the *model id* of a pretrained [`BarkProcessor`] hosted inside a model repo on
                  hf-mirror.com.
                - a path to a *directory* containing a processor saved using the [`~BarkProcessor.save_pretrained`]
                  method, e.g., `./my_model_directory/`.
            speaker_embeddings_dict_path (`str`, *optional*, defaults to `"speaker_embeddings_path.json"`):
                The name of the `.json` file containing the speaker_embeddings dictionnary located in
                `pretrained_model_name_or_path`. If `None`, no speaker_embeddings is loaded.
            **kwargs:
                Additional keyword arguments passed along to both
                [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`].
        """
        if speaker_embeddings_dict_path is not None:
            speaker_embeddings_path = cached_file(
                pretrained_processor_name_or_path,
                speaker_embeddings_dict_path,
                subfolder=kwargs.pop("subfolder", None),
                cache_dir=kwargs.pop("cache_dir", None),
                force_download=kwargs.pop("force_download", False),
                proxies=kwargs.pop("proxies", None),
                resume_download=kwargs.pop("resume_download", False),
                local_files_only=kwargs.pop("local_files_only", False),
                token=kwargs.pop("use_auth_token", None),
            )
            if speaker_embeddings_path is None:
                logger.warning(
                    f"""`{os.path.join(pretrained_processor_name_or_path,speaker_embeddings_dict_path)}` does not exists
                    , no preloaded speaker embeddings will be used - Make sure to provide a correct path to the json
                    dictionnary if wanted, otherwise set `speaker_embeddings_dict_path=None`."""
                )
                speaker_embeddings = None
            else:
                with open(speaker_embeddings_path, encoding='utf-8') as speaker_embeddings_json:
                    speaker_embeddings = json.load(speaker_embeddings_json)
        else:
            speaker_embeddings = None

        tokenizer = AutoTokenizer.from_pretrained(pretrained_processor_name_or_path, **kwargs)

        return cls(tokenizer=tokenizer, speaker_embeddings=speaker_embeddings)

    def save_pretrained(
        self,
        save_directory,
        speaker_embeddings_dict_path="speaker_embeddings_path.json",
        speaker_embeddings_directory="speaker_embeddings",
        **kwargs,
    ):
        """
        Saves the attributes of this processor (tokenizer...) in the specified directory so that it can be reloaded
        using the [`~BarkProcessor.from_pretrained`] method.

        Args:
            save_directory (`str` or `os.PathLike`):
                Directory where the tokenizer files and the speaker embeddings will be saved (directory will be created
                if it does not exist).
            speaker_embeddings_dict_path (`str`, *optional*, defaults to `"speaker_embeddings_path.json"`):
                The name of the `.json` file that will contains the speaker_embeddings nested path dictionnary, if it
                exists, and that will be located in `pretrained_model_name_or_path/speaker_embeddings_directory`.
            speaker_embeddings_directory (`str`, *optional*, defaults to `"speaker_embeddings/"`):
                The name of the folder in which the speaker_embeddings arrays will be saved.
            kwargs:
                Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method.
        """
        if self.speaker_embeddings is not None:
            os.makedirs(os.path.join(save_directory, speaker_embeddings_directory, "v2"), exist_ok=True)

            embeddings_dict = {}

            embeddings_dict["repo_or_path"] = save_directory

            for prompt_key in self.speaker_embeddings:
                if prompt_key != "repo_or_path":
                    voice_preset = self._load_voice_preset(prompt_key)

                    tmp_dict = {}
                    for key in self.speaker_embeddings[prompt_key]:
                        np.save(
                            os.path.join(
                                embeddings_dict["repo_or_path"], speaker_embeddings_directory, f"{prompt_key}_{key}"
                            ),
                            voice_preset[key],
                            allow_pickle=False,
                        )
                        tmp_dict[key] = os.path.join(speaker_embeddings_directory, f"{prompt_key}_{key}.npy")

                    embeddings_dict[prompt_key] = tmp_dict

            with open(os.path.join(save_directory, speaker_embeddings_dict_path), "w", encoding='utf-8') as fp:
                json.dump(embeddings_dict, fp)

        super().save_pretrained(save_directory, **kwargs)

    def _load_voice_preset(self, voice_preset: str = None, **kwargs):
        """
        This method '_load_voice_preset' is a member of the class 'BarkProcessor' and is responsible for loading voice presets.

        Args:
            self (object): The instance of the 'BarkProcessor' class.
            voice_preset (str): The name of the voice preset to be loaded. It is an optional parameter and defaults to None.
                It specifies the voice preset to be loaded from the speaker embeddings.

        Returns:
            dict: A dictionary containing the loaded voice preset data. The dictionary keys are 'semantic_prompt', 'coarse_prompt',
            and 'fine_prompt', and the corresponding values are NumPy arrays representing the voice preset data.

        Raises:
            ValueError: This exception is raised if the voice preset is unrecognized or if any key ('semantic_prompt', 'coarse_prompt',
            or 'fine_prompt') is missing in the speaker embeddings for the specified voice preset.
            ValueError: This exception is raised if the specified voice preset file does not exist, or if the path to the voice preset
            embeddings is incorrect.
        """
        voice_preset_paths = self.speaker_embeddings[voice_preset]

        voice_preset_dict = {}
        for key in ["semantic_prompt", "coarse_prompt", "fine_prompt"]:
            if key not in voice_preset_paths:
                raise ValueError(
                    f"Voice preset unrecognized, missing {key} as a key in self.speaker_embeddings[{voice_preset}]."
                )

            path = cached_file(
                self.speaker_embeddings.get("repo_or_path", "/"),
                voice_preset_paths[key],
                subfolder=kwargs.pop("subfolder", None),
                cache_dir=kwargs.pop("cache_dir", None),
                force_download=kwargs.pop("force_download", False),
                proxies=kwargs.pop("proxies", None),
                resume_download=kwargs.pop("resume_download", False),
                local_files_only=kwargs.pop("local_files_only", False),
                token=kwargs.pop("use_auth_token", None),
            )
            if path is None:
                raise ValueError(
                    f"""`{os.path.join(self.speaker_embeddings.get("repo_or_path", "/"),voice_preset_paths[key])}` does not exists
                    , no preloaded voice preset will be used - Make sure to provide correct paths to the {voice_preset}
                    embeddings."""
                )

            voice_preset_dict[key] = np.load(path)

        return voice_preset_dict

    def _validate_voice_preset_dict(self, voice_preset: Optional[dict] = None):
        """
        Validates the voice preset dictionary provided as input.

        Args:
            self (BarkProcessor): An instance of the BarkProcessor class.
            voice_preset (Optional[dict]):
                A dictionary representing the voice preset. This dictionary should contain the following keys:

                - 'semantic_prompt' (numpy.ndarray): A 2D ndarray representing the semantic prompt.
                - 'coarse_prompt' (numpy.ndarray): A 2D ndarray representing the coarse prompt.
                - 'fine_prompt' (numpy.ndarray): A 2D ndarray representing the fine prompt.

        Returns:
            None.

        Raises:
            ValueError:
                If any of the following conditions are not met:

                - The 'semantic_prompt', 'coarse_prompt', and 'fine_prompt' keys are missing in the voice_preset dictionary.
                - The 'semantic_prompt', 'coarse_prompt', or 'fine_prompt' values are not of type numpy.ndarray.
                - The shape of 'semantic_prompt', 'coarse_prompt', or 'fine_prompt' is not equal to the expected preset shape.

        Note:
            - The expected preset shape is determined by the preset_shape attribute of the BarkProcessor class.

        Example:
            ```python
            >>> bp = BarkProcessor()
            >>> voice_preset = {
            >>>     'semantic_prompt': np.array([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]]),
            >>>     'coarse_prompt': np.array([[0.7, 0.8, 0.9], [1.0, 1.1, 1.2]]),
            >>>     'fine_prompt': np.array([[1.3, 1.4, 1.5], [1.6, 1.7, 1.8]])
            >>> }
            >>> bp._validate_voice_preset_dict(voice_preset)
            ```
        """
        for key in ["semantic_prompt", "coarse_prompt", "fine_prompt"]:
            if key not in voice_preset:
                raise ValueError(f"Voice preset unrecognized, missing {key} as a key.")

            if not isinstance(voice_preset[key], np.ndarray):
                raise ValueError(f"{key} voice preset must be a {str(self.preset_shape[key])}D ndarray.")

            if len(voice_preset[key].shape) != self.preset_shape[key]:
                raise ValueError(f"{key} voice preset must be a {str(self.preset_shape[key])}D ndarray.")

    def __call__(
        self,
        text=None,
        voice_preset=None,
        return_tensors="ms",
        max_length=256,
        add_special_tokens=False,
        return_attention_mask=True,
        return_token_type_ids=False,
        **kwargs,
    ):
        """
        Main method to prepare for the model one or several sequences(s). This method forwards the `text` and `kwargs`
        arguments to the AutoTokenizer's [`~AutoTokenizer.__call__`] to encode the text. The method also proposes a
        voice preset which is a dictionary of arrays that conditions `Bark`'s output. `kwargs` arguments are forwarded
        to the tokenizer and to `cached_file` method if `voice_preset` is a valid filename.

        Args:
            text (`str`, `List[str]`, `List[List[str]]`):
                The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings
                (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set
                `is_split_into_words=True` (to lift the ambiguity with a batch of sequences).
            voice_preset (`str`, `Dict[np.ndarray]`):
                The voice preset, i.e the speaker embeddings. It can either be a valid voice_preset name, e.g
                `"en_speaker_1"`, or directly a dictionnary of `np.ndarray` embeddings for each submodel of `Bark`. Or
                it can be a valid file name of a local `.npz` single voice preset.
            return_tensors (`str` or [`~utils.TensorType`], *optional*):
                If set, will return tensors of a particular framework. Acceptable values are:

                - `'pt'`: Return PyTorch `torch.Tensor` objects.
                - `'np'`: Return NumPy `np.ndarray` objects.

        Returns:
            Tuple([`BatchEncoding`], [`BatchFeature`]): A tuple composed of a [`BatchEncoding`], i.e the output of the
            `tokenizer` and a [`BatchFeature`], i.e the voice preset with the right tensors type.
        """
        if voice_preset is not None and not isinstance(voice_preset, dict):
            if (
                isinstance(voice_preset, str)
                and self.speaker_embeddings is not None
                and voice_preset in self.speaker_embeddings
            ):
                voice_preset = self._load_voice_preset(voice_preset)

            else:
                if isinstance(voice_preset, str) and not voice_preset.endswith(".npz"):
                    voice_preset = voice_preset + ".npz"

                voice_preset = np.load(voice_preset)

        if voice_preset is not None:
            self._validate_voice_preset_dict(voice_preset, **kwargs)
            voice_preset = BatchFeature(data=voice_preset, tensor_type=return_tensors)

        encoded_text = self.tokenizer(
            text,
            return_tensors=return_tensors,
            padding="max_length",
            max_length=max_length,
            return_attention_mask=return_attention_mask,
            return_token_type_ids=return_token_type_ids,
            add_special_tokens=add_special_tokens,
            **kwargs,
        )

        if voice_preset is not None:
            encoded_text["history_prompt"] = voice_preset

        return encoded_text

mindnlp.transformers.models.bark.processing_bark.BarkProcessor.__call__(text=None, voice_preset=None, return_tensors='ms', max_length=256, add_special_tokens=False, return_attention_mask=True, return_token_type_ids=False, **kwargs)

Main method to prepare for the model one or several sequences(s). This method forwards the text and kwargs arguments to the AutoTokenizer's [~AutoTokenizer.__call__] to encode the text. The method also proposes a voice preset which is a dictionary of arrays that conditions Bark's output. kwargs arguments are forwarded to the tokenizer and to cached_file method if voice_preset is a valid filename.

PARAMETER	DESCRIPTION
text	The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set is_split_into_words=True (to lift the ambiguity with a batch of sequences). TYPE: `str`, `List[str]`, `List[List[str]]` DEFAULT: None
voice_preset	The voice preset, i.e the speaker embeddings. It can either be a valid voice_preset name, e.g "en_speaker_1", or directly a dictionnary of np.ndarray embeddings for each submodel of Bark. Or it can be a valid file name of a local .npz single voice preset. TYPE: `str`, `Dict[np.ndarray]` DEFAULT: None
return_tensors	If set, will return tensors of a particular framework. Acceptable values are: 'pt': Return PyTorch torch.Tensor objects. 'np': Return NumPy np.ndarray objects. TYPE: `str` or [`~utils.TensorType`], *optional* DEFAULT: 'ms'

RETURNS	DESCRIPTION
Tuple	A tuple composed of a [BatchEncoding], i.e the output of the TYPE: [`BatchEncoding`], [`BatchFeature`]
	tokenizer and a [BatchFeature], i.e the voice preset with the right tensors type.

Source code in mindnlp/transformers/models/bark/processing_bark.py

def __call__(
    self,
    text=None,
    voice_preset=None,
    return_tensors="ms",
    max_length=256,
    add_special_tokens=False,
    return_attention_mask=True,
    return_token_type_ids=False,
    **kwargs,
):
    """
    Main method to prepare for the model one or several sequences(s). This method forwards the `text` and `kwargs`
    arguments to the AutoTokenizer's [`~AutoTokenizer.__call__`] to encode the text. The method also proposes a
    voice preset which is a dictionary of arrays that conditions `Bark`'s output. `kwargs` arguments are forwarded
    to the tokenizer and to `cached_file` method if `voice_preset` is a valid filename.

    Args:
        text (`str`, `List[str]`, `List[List[str]]`):
            The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings
            (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set
            `is_split_into_words=True` (to lift the ambiguity with a batch of sequences).
        voice_preset (`str`, `Dict[np.ndarray]`):
            The voice preset, i.e the speaker embeddings. It can either be a valid voice_preset name, e.g
            `"en_speaker_1"`, or directly a dictionnary of `np.ndarray` embeddings for each submodel of `Bark`. Or
            it can be a valid file name of a local `.npz` single voice preset.
        return_tensors (`str` or [`~utils.TensorType`], *optional*):
            If set, will return tensors of a particular framework. Acceptable values are:

            - `'pt'`: Return PyTorch `torch.Tensor` objects.
            - `'np'`: Return NumPy `np.ndarray` objects.

    Returns:
        Tuple([`BatchEncoding`], [`BatchFeature`]): A tuple composed of a [`BatchEncoding`], i.e the output of the
        `tokenizer` and a [`BatchFeature`], i.e the voice preset with the right tensors type.
    """
    if voice_preset is not None and not isinstance(voice_preset, dict):
        if (
            isinstance(voice_preset, str)
            and self.speaker_embeddings is not None
            and voice_preset in self.speaker_embeddings
        ):
            voice_preset = self._load_voice_preset(voice_preset)

        else:
            if isinstance(voice_preset, str) and not voice_preset.endswith(".npz"):
                voice_preset = voice_preset + ".npz"

            voice_preset = np.load(voice_preset)

    if voice_preset is not None:
        self._validate_voice_preset_dict(voice_preset, **kwargs)
        voice_preset = BatchFeature(data=voice_preset, tensor_type=return_tensors)

    encoded_text = self.tokenizer(
        text,
        return_tensors=return_tensors,
        padding="max_length",
        max_length=max_length,
        return_attention_mask=return_attention_mask,
        return_token_type_ids=return_token_type_ids,
        add_special_tokens=add_special_tokens,
        **kwargs,
    )

    if voice_preset is not None:
        encoded_text["history_prompt"] = voice_preset

    return encoded_text

mindnlp.transformers.models.bark.processing_bark.BarkProcessor.__init__(tokenizer, speaker_embeddings=None)

Initialize the BarkProcessor class.

PARAMETER	DESCRIPTION
self	The instance of the class. TYPE: object
tokenizer	The tokenizer object used for processing text data. TYPE: object
speaker_embeddings	The speaker embeddings associated with the text data. Defaults to None. TYPE: object or None DEFAULT: None

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/bark/processing_bark.py

def __init__(self, tokenizer, speaker_embeddings=None):
    """Initialize the BarkProcessor class.

    Args:
        self (object): The instance of the class.
        tokenizer (object): The tokenizer object used for processing text data.
        speaker_embeddings (object or None, optional): The speaker embeddings associated with the text data. 
            Defaults to None.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(tokenizer)

    self.speaker_embeddings = speaker_embeddings

mindnlp.transformers.models.bark.processing_bark.BarkProcessor.from_pretrained(pretrained_processor_name_or_path, speaker_embeddings_dict_path='speaker_embeddings_path.json', **kwargs) classmethod

Instantiate a Bark processor associated with a pretrained model.

PARAMETER	DESCRIPTION
pretrained_model_name_or_path	This can be either: a string, the model id of a pretrained [BarkProcessor] hosted inside a model repo on hf-mirror.com. a path to a directory containing a processor saved using the [~BarkProcessor.save_pretrained] method, e.g., ./my_model_directory/. TYPE: `str` or `os.PathLike`
speaker_embeddings_dict_path	The name of the .json file containing the speaker_embeddings dictionnary located in pretrained_model_name_or_path. If None, no speaker_embeddings is loaded. TYPE: `str`, *optional*, defaults to `"speaker_embeddings_path.json"` DEFAULT: 'speaker_embeddings_path.json'
**kwargs	Additional keyword arguments passed along to both [~tokenization_utils_base.PreTrainedTokenizer.from_pretrained]. DEFAULT: {}

Source code in mindnlp/transformers/models/bark/processing_bark.py

@classmethod
def from_pretrained(
    cls, pretrained_processor_name_or_path, speaker_embeddings_dict_path="speaker_embeddings_path.json", **kwargs
):
    r"""
    Instantiate a Bark processor associated with a pretrained model.

    Args:
        pretrained_model_name_or_path (`str` or `os.PathLike`):
            This can be either:

            - a string, the *model id* of a pretrained [`BarkProcessor`] hosted inside a model repo on
              hf-mirror.com.
            - a path to a *directory* containing a processor saved using the [`~BarkProcessor.save_pretrained`]
              method, e.g., `./my_model_directory/`.
        speaker_embeddings_dict_path (`str`, *optional*, defaults to `"speaker_embeddings_path.json"`):
            The name of the `.json` file containing the speaker_embeddings dictionnary located in
            `pretrained_model_name_or_path`. If `None`, no speaker_embeddings is loaded.
        **kwargs:
            Additional keyword arguments passed along to both
            [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`].
    """
    if speaker_embeddings_dict_path is not None:
        speaker_embeddings_path = cached_file(
            pretrained_processor_name_or_path,
            speaker_embeddings_dict_path,
            subfolder=kwargs.pop("subfolder", None),
            cache_dir=kwargs.pop("cache_dir", None),
            force_download=kwargs.pop("force_download", False),
            proxies=kwargs.pop("proxies", None),
            resume_download=kwargs.pop("resume_download", False),
            local_files_only=kwargs.pop("local_files_only", False),
            token=kwargs.pop("use_auth_token", None),
        )
        if speaker_embeddings_path is None:
            logger.warning(
                f"""`{os.path.join(pretrained_processor_name_or_path,speaker_embeddings_dict_path)}` does not exists
                , no preloaded speaker embeddings will be used - Make sure to provide a correct path to the json
                dictionnary if wanted, otherwise set `speaker_embeddings_dict_path=None`."""
            )
            speaker_embeddings = None
        else:
            with open(speaker_embeddings_path, encoding='utf-8') as speaker_embeddings_json:
                speaker_embeddings = json.load(speaker_embeddings_json)
    else:
        speaker_embeddings = None

    tokenizer = AutoTokenizer.from_pretrained(pretrained_processor_name_or_path, **kwargs)

    return cls(tokenizer=tokenizer, speaker_embeddings=speaker_embeddings)

mindnlp.transformers.models.bark.processing_bark.BarkProcessor.save_pretrained(save_directory, speaker_embeddings_dict_path='speaker_embeddings_path.json', speaker_embeddings_directory='speaker_embeddings', **kwargs)

Saves the attributes of this processor (tokenizer...) in the specified directory so that it can be reloaded using the [~BarkProcessor.from_pretrained] method.

PARAMETER	DESCRIPTION
save_directory	Directory where the tokenizer files and the speaker embeddings will be saved (directory will be created if it does not exist). TYPE: `str` or `os.PathLike`
speaker_embeddings_dict_path	The name of the .json file that will contains the speaker_embeddings nested path dictionnary, if it exists, and that will be located in pretrained_model_name_or_path/speaker_embeddings_directory. TYPE: `str`, *optional*, defaults to `"speaker_embeddings_path.json"` DEFAULT: 'speaker_embeddings_path.json'
speaker_embeddings_directory	The name of the folder in which the speaker_embeddings arrays will be saved. TYPE: `str`, *optional*, defaults to `"speaker_embeddings/"` DEFAULT: 'speaker_embeddings'
kwargs	Additional key word arguments passed along to the [~utils.PushToHubMixin.push_to_hub] method. DEFAULT: {}

Source code in mindnlp/transformers/models/bark/processing_bark.py

def save_pretrained(
    self,
    save_directory,
    speaker_embeddings_dict_path="speaker_embeddings_path.json",
    speaker_embeddings_directory="speaker_embeddings",
    **kwargs,
):
    """
    Saves the attributes of this processor (tokenizer...) in the specified directory so that it can be reloaded
    using the [`~BarkProcessor.from_pretrained`] method.

    Args:
        save_directory (`str` or `os.PathLike`):
            Directory where the tokenizer files and the speaker embeddings will be saved (directory will be created
            if it does not exist).
        speaker_embeddings_dict_path (`str`, *optional*, defaults to `"speaker_embeddings_path.json"`):
            The name of the `.json` file that will contains the speaker_embeddings nested path dictionnary, if it
            exists, and that will be located in `pretrained_model_name_or_path/speaker_embeddings_directory`.
        speaker_embeddings_directory (`str`, *optional*, defaults to `"speaker_embeddings/"`):
            The name of the folder in which the speaker_embeddings arrays will be saved.
        kwargs:
            Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method.
    """
    if self.speaker_embeddings is not None:
        os.makedirs(os.path.join(save_directory, speaker_embeddings_directory, "v2"), exist_ok=True)

        embeddings_dict = {}

        embeddings_dict["repo_or_path"] = save_directory

        for prompt_key in self.speaker_embeddings:
            if prompt_key != "repo_or_path":
                voice_preset = self._load_voice_preset(prompt_key)

                tmp_dict = {}
                for key in self.speaker_embeddings[prompt_key]:
                    np.save(
                        os.path.join(
                            embeddings_dict["repo_or_path"], speaker_embeddings_directory, f"{prompt_key}_{key}"
                        ),
                        voice_preset[key],
                        allow_pickle=False,
                    )
                    tmp_dict[key] = os.path.join(speaker_embeddings_directory, f"{prompt_key}_{key}.npy")

                embeddings_dict[prompt_key] = tmp_dict

        with open(os.path.join(save_directory, speaker_embeddings_dict_path), "w", encoding='utf-8') as fp:
            json.dump(embeddings_dict, fp)

    super().save_pretrained(save_directory, **kwargs)