blip_2

mindnlp.transformers.models.blip_2.configuration_blip_2.Blip2Config

Bases: PretrainedConfig

[Blip2Config] is the configuration class to store the configuration of a [Blip2ForConditionalGeneration]. It is used to instantiate a BLIP-2 model according to the specified arguments, defining the vision model, Q-Former model and language model configs. Instantiating a configuration with the defaults will yield a similar configuration to that of the BLIP-2 Salesforce/blip2-opt-2.7b architecture.

Configuration objects inherit from [PretrainedConfig] and can be used to control the model outputs. Read the documentation from [PretrainedConfig] for more information.

PARAMETER	DESCRIPTION
vision_config	Dictionary of configuration options used to initialize [Blip2VisionConfig]. TYPE: `dict`, *optional* DEFAULT: None
qformer_config	Dictionary of configuration options used to initialize [Blip2QFormerConfig]. TYPE: `dict`, *optional* DEFAULT: None
text_config	Dictionary of configuration options used to initialize any [PretrainedConfig]. TYPE: `dict`, *optional* DEFAULT: None
num_query_tokens	The number of query tokens passed through the Transformer. TYPE: `int`, *optional*, defaults to 32 DEFAULT: 32
kwargs	Dictionary of keyword arguments. TYPE: *optional* DEFAULT: {}

Example

>>> from transformers import (
...     Blip2VisionConfig,
...     Blip2QFormerConfig,
...     OPTConfig,
...     Blip2Config,
...     Blip2ForConditionalGeneration,
... )
...
>>> # Initializing a Blip2Config with Salesforce/blip2-opt-2.7b style configuration
>>> configuration = Blip2Config()
...
>>> # Initializing a Blip2ForConditionalGeneration (with random weights) from the Salesforce/blip2-opt-2.7b style configuration
>>> model = Blip2ForConditionalGeneration(configuration)
...
>>> # Accessing the model configuration
>>> configuration = model.config
...
>>> # We can also initialize a Blip2Config from a Blip2VisionConfig, Blip2QFormerConfig and any PretrainedConfig
...
>>> # Initializing BLIP-2 vision, BLIP-2 Q-Former and language model configurations
>>> vision_config = Blip2VisionConfig()
>>> qformer_config = Blip2QFormerConfig()
>>> text_config = OPTConfig()
...
>>> config = Blip2Config.from_text_vision_configs(vision_config, qformer_config, text_config)

Source code in mindnlp/transformers/models/blip_2/configuration_blip_2.py

class Blip2Config(PretrainedConfig):
    r"""
    [`Blip2Config`] is the configuration class to store the configuration of a [`Blip2ForConditionalGeneration`]. It is
    used to instantiate a BLIP-2 model according to the specified arguments, defining the vision model, Q-Former model
    and language model configs. Instantiating a configuration with the defaults will yield a similar configuration to
    that of the BLIP-2 [Salesforce/blip2-opt-2.7b](https://huggingface.co/Salesforce/blip2-opt-2.7b) architecture.

    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
    documentation from [`PretrainedConfig`] for more information.

    Args:
        vision_config (`dict`, *optional*):
            Dictionary of configuration options used to initialize [`Blip2VisionConfig`].
        qformer_config (`dict`, *optional*):
            Dictionary of configuration options used to initialize [`Blip2QFormerConfig`].
        text_config (`dict`, *optional*):
            Dictionary of configuration options used to initialize any [`PretrainedConfig`].
        num_query_tokens (`int`, *optional*, defaults to 32):
            The number of query tokens passed through the Transformer.
        kwargs (*optional*):
            Dictionary of keyword arguments.

    Example:
        ```python
        >>> from transformers import (
        ...     Blip2VisionConfig,
        ...     Blip2QFormerConfig,
        ...     OPTConfig,
        ...     Blip2Config,
        ...     Blip2ForConditionalGeneration,
        ... )
        ...
        >>> # Initializing a Blip2Config with Salesforce/blip2-opt-2.7b style configuration
        >>> configuration = Blip2Config()
        ...
        >>> # Initializing a Blip2ForConditionalGeneration (with random weights) from the Salesforce/blip2-opt-2.7b style configuration
        >>> model = Blip2ForConditionalGeneration(configuration)
        ...
        >>> # Accessing the model configuration
        >>> configuration = model.config
        ...
        >>> # We can also initialize a Blip2Config from a Blip2VisionConfig, Blip2QFormerConfig and any PretrainedConfig
        ...
        >>> # Initializing BLIP-2 vision, BLIP-2 Q-Former and language model configurations
        >>> vision_config = Blip2VisionConfig()
        >>> qformer_config = Blip2QFormerConfig()
        >>> text_config = OPTConfig()
        ...
        >>> config = Blip2Config.from_text_vision_configs(vision_config, qformer_config, text_config)
        ```
    """
    model_type = "blip-2"

    def __init__(self, vision_config=None, qformer_config=None, text_config=None, num_query_tokens=32, **kwargs):
        """
        This method initializes an instance of the Blip2Config class.

        Args:
            self: The instance of the class.
            vision_config (dict, optional): Configuration for vision. Defaults to None.
            qformer_config (dict, optional): Configuration for qformer. Defaults to None.
            text_config (dict, optional): Configuration for text. Defaults to None.
            num_query_tokens (int, optional): The number of query tokens. Defaults to 32.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(**kwargs)

        if vision_config is None:
            vision_config = {}
            logger.info("vision_config is None. initializing the Blip2VisionConfig with default values.")

        if qformer_config is None:
            qformer_config = {}
            logger.info("qformer_config is None. Initializing the Blip2QFormerConfig with default values.")

        if text_config is None:
            text_config = {}
            logger.info("text_config is None. Initializing the text config with default values (`OPTConfig`).")

        self.vision_config = Blip2VisionConfig(**vision_config)
        self.qformer_config = Blip2QFormerConfig(**qformer_config)
        text_model_type = text_config["model_type"] if "model_type" in text_config else "opt"
        self.text_config = CONFIG_MAPPING[text_model_type](**text_config)

        self.tie_word_embeddings = self.text_config.tie_word_embeddings
        self.is_encoder_decoder = self.text_config.is_encoder_decoder

        self.num_query_tokens = num_query_tokens
        self.qformer_config.encoder_hidden_size = self.vision_config.hidden_size
        self.use_decoder_only_language_model = self.text_config.model_type in MODEL_FOR_CAUSAL_LM_MAPPING_NAMES
        self.initializer_factor = 1.0
        self.initializer_range = 0.02

    @classmethod
    def from_vision_qformer_text_configs(
        cls,
        vision_config: Blip2VisionConfig,
        qformer_config: Blip2QFormerConfig,
        text_config: PretrainedConfig,
        **kwargs,
    ):
        r"""
        Instantiate a [`Blip2Config`] (or a derived class) from a BLIP-2 vision model, Q-Former and language model
        configurations.

        Returns:
            [`Blip2Config`]: An instance of a configuration object
        """
        return cls(
            vision_config=vision_config.to_dict(),
            qformer_config=qformer_config.to_dict(),
            text_config=text_config.to_dict(),
            **kwargs,
        )

mindnlp.transformers.models.blip_2.configuration_blip_2.Blip2Config.__init__(vision_config=None, qformer_config=None, text_config=None, num_query_tokens=32, **kwargs)

This method initializes an instance of the Blip2Config class.

PARAMETER	DESCRIPTION
self	The instance of the class.
vision_config	Configuration for vision. Defaults to None. TYPE: dict DEFAULT: None
qformer_config	Configuration for qformer. Defaults to None. TYPE: dict DEFAULT: None
text_config	Configuration for text. Defaults to None. TYPE: dict DEFAULT: None
num_query_tokens	The number of query tokens. Defaults to 32. TYPE: int DEFAULT: 32

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/blip_2/configuration_blip_2.py

def __init__(self, vision_config=None, qformer_config=None, text_config=None, num_query_tokens=32, **kwargs):
    """
    This method initializes an instance of the Blip2Config class.

    Args:
        self: The instance of the class.
        vision_config (dict, optional): Configuration for vision. Defaults to None.
        qformer_config (dict, optional): Configuration for qformer. Defaults to None.
        text_config (dict, optional): Configuration for text. Defaults to None.
        num_query_tokens (int, optional): The number of query tokens. Defaults to 32.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(**kwargs)

    if vision_config is None:
        vision_config = {}
        logger.info("vision_config is None. initializing the Blip2VisionConfig with default values.")

    if qformer_config is None:
        qformer_config = {}
        logger.info("qformer_config is None. Initializing the Blip2QFormerConfig with default values.")

    if text_config is None:
        text_config = {}
        logger.info("text_config is None. Initializing the text config with default values (`OPTConfig`).")

    self.vision_config = Blip2VisionConfig(**vision_config)
    self.qformer_config = Blip2QFormerConfig(**qformer_config)
    text_model_type = text_config["model_type"] if "model_type" in text_config else "opt"
    self.text_config = CONFIG_MAPPING[text_model_type](**text_config)

    self.tie_word_embeddings = self.text_config.tie_word_embeddings
    self.is_encoder_decoder = self.text_config.is_encoder_decoder

    self.num_query_tokens = num_query_tokens
    self.qformer_config.encoder_hidden_size = self.vision_config.hidden_size
    self.use_decoder_only_language_model = self.text_config.model_type in MODEL_FOR_CAUSAL_LM_MAPPING_NAMES
    self.initializer_factor = 1.0
    self.initializer_range = 0.02

mindnlp.transformers.models.blip_2.configuration_blip_2.Blip2Config.from_vision_qformer_text_configs(vision_config, qformer_config, text_config, **kwargs) classmethod

Instantiate a [Blip2Config] (or a derived class) from a BLIP-2 vision model, Q-Former and language model configurations.

RETURNS	DESCRIPTION
	[Blip2Config]: An instance of a configuration object

Source code in mindnlp/transformers/models/blip_2/configuration_blip_2.py

@classmethod
def from_vision_qformer_text_configs(
    cls,
    vision_config: Blip2VisionConfig,
    qformer_config: Blip2QFormerConfig,
    text_config: PretrainedConfig,
    **kwargs,
):
    r"""
    Instantiate a [`Blip2Config`] (or a derived class) from a BLIP-2 vision model, Q-Former and language model
    configurations.

    Returns:
        [`Blip2Config`]: An instance of a configuration object
    """
    return cls(
        vision_config=vision_config.to_dict(),
        qformer_config=qformer_config.to_dict(),
        text_config=text_config.to_dict(),
        **kwargs,
    )

mindnlp.transformers.models.blip_2.configuration_blip_2.Blip2QFormerConfig

Bases: PretrainedConfig

This is the configuration class to store the configuration of a [Blip2QFormerModel]. It is used to instantiate a BLIP-2 Querying Transformer (Q-Former) model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the BLIP-2 Salesforce/blip2-opt-2.7b architecture. Configuration objects inherit from [PretrainedConfig] and can be used to control the model outputs. Read the documentation from [PretrainedConfig] for more information.

Note that [Blip2QFormerModel] is very similar to [BertLMHeadModel] with interleaved cross-attention.

PARAMETER	DESCRIPTION
vocab_size	Vocabulary size of the Q-Former model. Defines the number of different tokens that can be represented by the inputs_ids passed when calling the model. TYPE: `int`, *optional*, defaults to 30522 DEFAULT: 30522
hidden_size	Dimensionality of the encoder layers and the pooler layer. TYPE: `int`, *optional*, defaults to 768 DEFAULT: 768
num_hidden_layers	Number of hidden layers in the Transformer encoder. TYPE: `int`, *optional*, defaults to 12 DEFAULT: 12
num_attention_heads	Number of attention heads for each attention layer in the Transformer encoder. TYPE: `int`, *optional*, defaults to 12 DEFAULT: 12
intermediate_size	Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer encoder. TYPE: `int`, *optional*, defaults to 3072 DEFAULT: 3072
hidden_act	The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu", "relu", "silu" and "gelu_new" are supported. TYPE: `str` or `Callable`, *optional*, defaults to `"gelu"` DEFAULT: 'gelu'
hidden_dropout_prob	The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
attention_probs_dropout_prob	The dropout ratio for the attention probabilities. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
max_position_embeddings	The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048). TYPE: `int`, *optional*, defaults to 512 DEFAULT: 512
initializer_range	The standard deviation of the truncated_normal_initializer for initializing all weight matrices. TYPE: `float`, *optional*, defaults to 0.02 DEFAULT: 0.02
layer_norm_eps	The epsilon used by the layer normalization layers. TYPE: `float`, *optional*, defaults to 1e-12 DEFAULT: 1e-12
position_embedding_type	Type of position embedding. Choose one of "absolute", "relative_key", "relative_key_query". For positional embeddings use "absolute". For more information on "relative_key", please refer to Self-Attention with Relative Position Representations (Shaw et al.). For more information on "relative_key_query", please refer to Method 4 in Improve Transformer Models with Better Relative Position Embeddings (Huang et al.). TYPE: `str`, *optional*, defaults to `"absolute"` DEFAULT: 'absolute'
cross_attention_frequency	The frequency of adding cross-attention to the Transformer layers. TYPE: `int`, *optional*, defaults to 2 DEFAULT: 2
encoder_hidden_size	The hidden size of the hidden states for cross-attention. TYPE: `int`, *optional*, defaults to 1408 DEFAULT: 1408

Example

>>> from transformers import Blip2QFormerConfig, Blip2QFormerModel
...
>>> # Initializing a BLIP-2 Salesforce/blip2-opt-2.7b style configuration
>>> configuration = Blip2QFormerConfig()
...
>>> # Initializing a model (with random weights) from the Salesforce/blip2-opt-2.7b style configuration
>>> model = Blip2QFormerModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config

Source code in mindnlp/transformers/models/blip_2/configuration_blip_2.py

class Blip2QFormerConfig(PretrainedConfig):
    r"""
    This is the configuration class to store the configuration of a [`Blip2QFormerModel`]. It is used to instantiate a
    BLIP-2 Querying Transformer (Q-Former) model according to the specified arguments, defining the model architecture.
    Instantiating a configuration with the defaults will yield a similar configuration to that of the BLIP-2
    [Salesforce/blip2-opt-2.7b](https://huggingface.co/Salesforce/blip2-opt-2.7b) architecture. Configuration objects
    inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from
    [`PretrainedConfig`] for more information.

    Note that [`Blip2QFormerModel`] is very similar to [`BertLMHeadModel`] with interleaved cross-attention.

    Args:
        vocab_size (`int`, *optional*, defaults to 30522):
            Vocabulary size of the Q-Former model. Defines the number of different tokens that can be represented by
            the `inputs_ids` passed when calling the model.
        hidden_size (`int`, *optional*, defaults to 768):
            Dimensionality of the encoder layers and the pooler layer.
        num_hidden_layers (`int`, *optional*, defaults to 12):
            Number of hidden layers in the Transformer encoder.
        num_attention_heads (`int`, *optional*, defaults to 12):
            Number of attention heads for each attention layer in the Transformer encoder.
        intermediate_size (`int`, *optional*, defaults to 3072):
            Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer encoder.
        hidden_act (`str` or `Callable`, *optional*, defaults to `"gelu"`):
            The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`,
            `"relu"`, `"silu"` and `"gelu_new"` are supported.
        hidden_dropout_prob (`float`, *optional*, defaults to 0.1):
            The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
        attention_probs_dropout_prob (`float`, *optional*, defaults to 0.1):
            The dropout ratio for the attention probabilities.
        max_position_embeddings (`int`, *optional*, defaults to 512):
            The maximum sequence length that this model might ever be used with. Typically set this to something large
            just in case (e.g., 512 or 1024 or 2048).
        initializer_range (`float`, *optional*, defaults to 0.02):
            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
        layer_norm_eps (`float`, *optional*, defaults to 1e-12):
            The epsilon used by the layer normalization layers.
        position_embedding_type (`str`, *optional*, defaults to `"absolute"`):
            Type of position embedding. Choose one of `"absolute"`, `"relative_key"`, `"relative_key_query"`. For
            positional embeddings use `"absolute"`. For more information on `"relative_key"`, please refer to
            [Self-Attention with Relative Position Representations (Shaw et al.)](https://arxiv.org/abs/1803.02155).
            For more information on `"relative_key_query"`, please refer to *Method 4* in [Improve Transformer Models
            with Better Relative Position Embeddings (Huang et al.)](https://arxiv.org/abs/2009.13658).
        cross_attention_frequency (`int`, *optional*, defaults to 2):
            The frequency of adding cross-attention to the Transformer layers.
        encoder_hidden_size (`int`, *optional*, defaults to 1408):
            The hidden size of the hidden states for cross-attention.

    Example:
        ```python
        >>> from transformers import Blip2QFormerConfig, Blip2QFormerModel
        ...
        >>> # Initializing a BLIP-2 Salesforce/blip2-opt-2.7b style configuration
        >>> configuration = Blip2QFormerConfig()
        ...
        >>> # Initializing a model (with random weights) from the Salesforce/blip2-opt-2.7b style configuration
        >>> model = Blip2QFormerModel(configuration)
        >>> # Accessing the model configuration
        >>> configuration = model.config
        ```
    """
    model_type = "blip_2_qformer"

    def __init__(
        self,
        vocab_size=30522,
        hidden_size=768,
        num_hidden_layers=12,
        num_attention_heads=12,
        intermediate_size=3072,
        hidden_act="gelu",
        hidden_dropout_prob=0.1,
        attention_probs_dropout_prob=0.1,
        max_position_embeddings=512,
        initializer_range=0.02,
        layer_norm_eps=1e-12,
        pad_token_id=0,
        position_embedding_type="absolute",
        cross_attention_frequency=2,
        encoder_hidden_size=1408,
        **kwargs,
    ):
        """
        Initialize a Blip2QFormerConfig object with the specified parameters.

        Args:
            self: The object itself.
            vocab_size (int): The size of the vocabulary.
            hidden_size (int): The size of the hidden layers.
            num_hidden_layers (int): The number of hidden layers in the model.
            num_attention_heads (int): The number of attention heads in the model.
            intermediate_size (int): The size of the intermediate layer in the model.
            hidden_act (str): The activation function for the hidden layers.
            hidden_dropout_prob (float): The dropout probability for the hidden layers.
            attention_probs_dropout_prob (float): The dropout probability for attention probabilities.
            max_position_embeddings (int): The maximum position embeddings allowed.
            initializer_range (float): The range for weight initializations.
            layer_norm_eps (float): The epsilon value for layer normalization.
            pad_token_id (int): The token ID for padding.
            position_embedding_type (str): The type of position embedding used (e.g., absolute).
            cross_attention_frequency (int): The frequency of cross-attention layers in the model.
            encoder_hidden_size (int): The size of the encoder hidden layers.

        Returns:
            None

        Raises:
            None
        """
        super().__init__(pad_token_id=pad_token_id, **kwargs)

        self.vocab_size = vocab_size
        self.hidden_size = hidden_size
        self.num_hidden_layers = num_hidden_layers
        self.num_attention_heads = num_attention_heads
        self.hidden_act = hidden_act
        self.intermediate_size = intermediate_size
        self.hidden_dropout_prob = hidden_dropout_prob
        self.attention_probs_dropout_prob = attention_probs_dropout_prob
        self.max_position_embeddings = max_position_embeddings
        self.initializer_range = initializer_range
        self.layer_norm_eps = layer_norm_eps
        self.position_embedding_type = position_embedding_type
        self.cross_attention_frequency = cross_attention_frequency
        self.encoder_hidden_size = encoder_hidden_size

    @classmethod
    def from_pretrained(cls, pretrained_model_name_or_path: Union[str, os.PathLike], **kwargs) -> "PretrainedConfig":
        """
        This method creates a 'Blip2QFormerConfig' instance from a pretrained model.

        Args:
            cls (type): The class type of 'Blip2QFormerConfig'.
            pretrained_model_name_or_path (Union[str, os.PathLike]): The name or path of the pretrained model.
                It accepts a string or os.PathLike object.

        Returns:
            PretrainedConfig: Returns a 'PretrainedConfig' object representing the configuration of the pretrained model.

        Raises:
            ValueError: If the 'config_dict' does not contain the required keys or values.
            TypeError: If the input parameters are of incorrect types.
            Warning: If the instantiated model type differs from the provided model type, a warning is issued.
        """
        config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)

        # get the qformer config dict if we are loading from Blip2Config
        if config_dict.get("model_type") == "blip-2":
            config_dict = config_dict["qformer_config"]

        if "model_type" in config_dict and hasattr(cls, "model_type") and config_dict["model_type"] != cls.model_type:
            logger.warning(
                f"You are using a model of type {config_dict['model_type']} to instantiate a model of type "
                f"{cls.model_type}. This is not supported for all configurations of models and can yield errors."
            )

        return cls.from_dict(config_dict, **kwargs)

mindnlp.transformers.models.blip_2.configuration_blip_2.Blip2QFormerConfig.__init__(vocab_size=30522, hidden_size=768, num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072, hidden_act='gelu', hidden_dropout_prob=0.1, attention_probs_dropout_prob=0.1, max_position_embeddings=512, initializer_range=0.02, layer_norm_eps=1e-12, pad_token_id=0, position_embedding_type='absolute', cross_attention_frequency=2, encoder_hidden_size=1408, **kwargs)

Initialize a Blip2QFormerConfig object with the specified parameters.

PARAMETER	DESCRIPTION
self	The object itself.
vocab_size	The size of the vocabulary. TYPE: int DEFAULT: 30522
hidden_size	The size of the hidden layers. TYPE: int DEFAULT: 768
num_hidden_layers	The number of hidden layers in the model. TYPE: int DEFAULT: 12
num_attention_heads	The number of attention heads in the model. TYPE: int DEFAULT: 12
intermediate_size	The size of the intermediate layer in the model. TYPE: int DEFAULT: 3072
hidden_act	The activation function for the hidden layers. TYPE: str DEFAULT: 'gelu'
hidden_dropout_prob	The dropout probability for the hidden layers. TYPE: float DEFAULT: 0.1
attention_probs_dropout_prob	The dropout probability for attention probabilities. TYPE: float DEFAULT: 0.1
max_position_embeddings	The maximum position embeddings allowed. TYPE: int DEFAULT: 512
initializer_range	The range for weight initializations. TYPE: float DEFAULT: 0.02
layer_norm_eps	The epsilon value for layer normalization. TYPE: float DEFAULT: 1e-12
pad_token_id	The token ID for padding. TYPE: int DEFAULT: 0
position_embedding_type	The type of position embedding used (e.g., absolute). TYPE: str DEFAULT: 'absolute'
cross_attention_frequency	The frequency of cross-attention layers in the model. TYPE: int DEFAULT: 2
encoder_hidden_size	The size of the encoder hidden layers. TYPE: int DEFAULT: 1408

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/blip_2/configuration_blip_2.py

def __init__(
    self,
    vocab_size=30522,
    hidden_size=768,
    num_hidden_layers=12,
    num_attention_heads=12,
    intermediate_size=3072,
    hidden_act="gelu",
    hidden_dropout_prob=0.1,
    attention_probs_dropout_prob=0.1,
    max_position_embeddings=512,
    initializer_range=0.02,
    layer_norm_eps=1e-12,
    pad_token_id=0,
    position_embedding_type="absolute",
    cross_attention_frequency=2,
    encoder_hidden_size=1408,
    **kwargs,
):
    """
    Initialize a Blip2QFormerConfig object with the specified parameters.

    Args:
        self: The object itself.
        vocab_size (int): The size of the vocabulary.
        hidden_size (int): The size of the hidden layers.
        num_hidden_layers (int): The number of hidden layers in the model.
        num_attention_heads (int): The number of attention heads in the model.
        intermediate_size (int): The size of the intermediate layer in the model.
        hidden_act (str): The activation function for the hidden layers.
        hidden_dropout_prob (float): The dropout probability for the hidden layers.
        attention_probs_dropout_prob (float): The dropout probability for attention probabilities.
        max_position_embeddings (int): The maximum position embeddings allowed.
        initializer_range (float): The range for weight initializations.
        layer_norm_eps (float): The epsilon value for layer normalization.
        pad_token_id (int): The token ID for padding.
        position_embedding_type (str): The type of position embedding used (e.g., absolute).
        cross_attention_frequency (int): The frequency of cross-attention layers in the model.
        encoder_hidden_size (int): The size of the encoder hidden layers.

    Returns:
        None

    Raises:
        None
    """
    super().__init__(pad_token_id=pad_token_id, **kwargs)

    self.vocab_size = vocab_size
    self.hidden_size = hidden_size
    self.num_hidden_layers = num_hidden_layers
    self.num_attention_heads = num_attention_heads
    self.hidden_act = hidden_act
    self.intermediate_size = intermediate_size
    self.hidden_dropout_prob = hidden_dropout_prob
    self.attention_probs_dropout_prob = attention_probs_dropout_prob
    self.max_position_embeddings = max_position_embeddings
    self.initializer_range = initializer_range
    self.layer_norm_eps = layer_norm_eps
    self.position_embedding_type = position_embedding_type
    self.cross_attention_frequency = cross_attention_frequency
    self.encoder_hidden_size = encoder_hidden_size

mindnlp.transformers.models.blip_2.configuration_blip_2.Blip2QFormerConfig.from_pretrained(pretrained_model_name_or_path, **kwargs) classmethod

This method creates a 'Blip2QFormerConfig' instance from a pretrained model.

PARAMETER	DESCRIPTION
cls	The class type of 'Blip2QFormerConfig'. TYPE: type
pretrained_model_name_or_path	The name or path of the pretrained model. It accepts a string or os.PathLike object. TYPE: Union[str, PathLike]

RETURNS	DESCRIPTION
PretrainedConfig	Returns a 'PretrainedConfig' object representing the configuration of the pretrained model. TYPE: PretrainedConfig

RAISES	DESCRIPTION
ValueError	If the 'config_dict' does not contain the required keys or values.
TypeError	If the input parameters are of incorrect types.
Warning	If the instantiated model type differs from the provided model type, a warning is issued.

Source code in mindnlp/transformers/models/blip_2/configuration_blip_2.py

@classmethod
def from_pretrained(cls, pretrained_model_name_or_path: Union[str, os.PathLike], **kwargs) -> "PretrainedConfig":
    """
    This method creates a 'Blip2QFormerConfig' instance from a pretrained model.

    Args:
        cls (type): The class type of 'Blip2QFormerConfig'.
        pretrained_model_name_or_path (Union[str, os.PathLike]): The name or path of the pretrained model.
            It accepts a string or os.PathLike object.

    Returns:
        PretrainedConfig: Returns a 'PretrainedConfig' object representing the configuration of the pretrained model.

    Raises:
        ValueError: If the 'config_dict' does not contain the required keys or values.
        TypeError: If the input parameters are of incorrect types.
        Warning: If the instantiated model type differs from the provided model type, a warning is issued.
    """
    config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)

    # get the qformer config dict if we are loading from Blip2Config
    if config_dict.get("model_type") == "blip-2":
        config_dict = config_dict["qformer_config"]

    if "model_type" in config_dict and hasattr(cls, "model_type") and config_dict["model_type"] != cls.model_type:
        logger.warning(
            f"You are using a model of type {config_dict['model_type']} to instantiate a model of type "
            f"{cls.model_type}. This is not supported for all configurations of models and can yield errors."
        )

    return cls.from_dict(config_dict, **kwargs)

mindnlp.transformers.models.blip_2.configuration_blip_2.Blip2VisionConfig

Bases: PretrainedConfig

This is the configuration class to store the configuration of a [Blip2VisionModel]. It is used to instantiate a BLIP-2 vision encoder according to the specified arguments, defining the model architecture. Instantiating a configuration defaults will yield a similar configuration to that of the BLIP-2 Salesforce/blip2-opt-2.7b architecture.

Configuration objects inherit from [PretrainedConfig] and can be used to control the model outputs. Read the documentation from [PretrainedConfig] for more information.

PARAMETER	DESCRIPTION
hidden_size	Dimensionality of the encoder layers and the pooler layer. TYPE: `int`, *optional*, defaults to 1408 DEFAULT: 1408
intermediate_size	Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. TYPE: `int`, *optional*, defaults to 6144 DEFAULT: 6144
num_hidden_layers	Number of hidden layers in the Transformer encoder. TYPE: `int`, *optional*, defaults to 39 DEFAULT: 39
num_attention_heads	Number of attention heads for each attention layer in the Transformer encoder. TYPE: `int`, *optional*, defaults to 16 DEFAULT: 16
image_size	The size (resolution) of each image. TYPE: `int`, *optional*, defaults to 224 DEFAULT: 224
patch_size	The size (resolution) of each patch. TYPE: `int`, *optional*, defaults to 14 DEFAULT: 14
hidden_act	The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu", "relu", "selu" and "gelu_new" `"gelu" are supported. layer_norm_eps (float, optional, defaults to 1e-5): The epsilon used by the layer normalization layers. TYPE: `str` or `function`, *optional*, defaults to `"gelu"` DEFAULT: 'gelu'
attention_dropout	The dropout ratio for the attention probabilities. TYPE: `float`, *optional*, defaults to 0.0 DEFAULT: 0.0
initializer_range	The standard deviation of the truncated_normal_initializer for initializing all weight matrices. TYPE: `float`, *optional*, defaults to 0.02 DEFAULT: 1e-10
qkv_bias	Whether to add a bias to the queries and values in the self-attention layers. TYPE: `bool`, *optional*, defaults to `True` DEFAULT: True

Example

>>> from transformers import Blip2VisionConfig, Blip2VisionModel
...
>>> # Initializing a Blip2VisionConfig with Salesforce/blip2-opt-2.7b style configuration
>>> configuration = Blip2VisionConfig()
...
>>> # Initializing a Blip2VisionModel (with random weights) from the Salesforce/blip2-opt-2.7b style configuration
>>> model = Blip2VisionModel(configuration)
...
>>> # Accessing the model configuration
>>> configuration = model.config

Source code in mindnlp/transformers/models/blip_2/configuration_blip_2.py

class Blip2VisionConfig(PretrainedConfig):
    r"""
    This is the configuration class to store the configuration of a [`Blip2VisionModel`]. It is used to instantiate a
    BLIP-2 vision encoder according to the specified arguments, defining the model architecture. Instantiating a
    configuration defaults will yield a similar configuration to that of the BLIP-2
    [Salesforce/blip2-opt-2.7b](https://huggingface.co/Salesforce/blip2-opt-2.7b) architecture.

    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
    documentation from [`PretrainedConfig`] for more information.

    Args:
        hidden_size (`int`, *optional*, defaults to 1408):
            Dimensionality of the encoder layers and the pooler layer.
        intermediate_size (`int`, *optional*, defaults to 6144):
            Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder.
        num_hidden_layers (`int`, *optional*, defaults to 39):
            Number of hidden layers in the Transformer encoder.
        num_attention_heads (`int`, *optional*, defaults to 16):
            Number of attention heads for each attention layer in the Transformer encoder.
        image_size (`int`, *optional*, defaults to 224):
            The size (resolution) of each image.
        patch_size (`int`, *optional*, defaults to 14):
            The size (resolution) of each patch.
        hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`):
            The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`,
            `"relu"`, `"selu"` and `"gelu_new"` ``"gelu"` are supported. layer_norm_eps (`float`, *optional*, defaults
            to 1e-5): The epsilon used by the layer normalization layers.
        attention_dropout (`float`, *optional*, defaults to 0.0):
            The dropout ratio for the attention probabilities.
        initializer_range (`float`, *optional*, defaults to 0.02):
            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
        qkv_bias (`bool`, *optional*, defaults to `True`):
            Whether to add a bias to the queries and values in the self-attention layers.

    Example:
        ```python
        >>> from transformers import Blip2VisionConfig, Blip2VisionModel
        ...
        >>> # Initializing a Blip2VisionConfig with Salesforce/blip2-opt-2.7b style configuration
        >>> configuration = Blip2VisionConfig()
        ...
        >>> # Initializing a Blip2VisionModel (with random weights) from the Salesforce/blip2-opt-2.7b style configuration
        >>> model = Blip2VisionModel(configuration)
        ...
        >>> # Accessing the model configuration
        >>> configuration = model.config
        ```
    """
    model_type = "blip_2_vision_model"

    def __init__(
        self,
        hidden_size=1408,
        intermediate_size=6144,
        num_hidden_layers=39,
        num_attention_heads=16,
        image_size=224,
        patch_size=14,
        hidden_act="gelu",
        layer_norm_eps=1e-6,
        attention_dropout=0.0,
        initializer_range=1e-10,
        qkv_bias=True,
        **kwargs,
    ):
        """
        Initializes an instance of the Blip2VisionConfig class.

        Args:
            self: The instance of the class.
            hidden_size (int, optional): The size of the hidden layers. Defaults to 1408.
            intermediate_size (int, optional): The size of the intermediate layers. Defaults to 6144.
            num_hidden_layers (int, optional): The number of hidden layers. Defaults to 39.
            num_attention_heads (int, optional): The number of attention heads. Defaults to 16.
            image_size (int, optional): The size of the image. Defaults to 224.
            patch_size (int, optional): The size of the patches. Defaults to 14.
            hidden_act (str, optional): The activation function for the hidden layers. Defaults to 'gelu'.
            layer_norm_eps (float, optional): The epsilon value for layer normalization. Defaults to 1e-06.
            attention_dropout (float, optional): The dropout rate for attention layers. Defaults to 0.0.
            initializer_range (float, optional): The range for weight initialization. Defaults to 1e-10.
            qkv_bias (bool, optional): Whether to include bias in the query, key, value layers. Defaults to True.

        Returns:
            None

        Raises:
            None
        """
        super().__init__(**kwargs)

        self.hidden_size = hidden_size
        self.intermediate_size = intermediate_size
        self.num_hidden_layers = num_hidden_layers
        self.num_attention_heads = num_attention_heads
        self.patch_size = patch_size
        self.image_size = image_size
        self.initializer_range = initializer_range
        self.attention_dropout = attention_dropout
        self.layer_norm_eps = layer_norm_eps
        self.hidden_act = hidden_act
        self.qkv_bias = qkv_bias

    @classmethod
    def from_pretrained(cls, pretrained_model_name_or_path: Union[str, os.PathLike], **kwargs) -> "PretrainedConfig":
        """
        Method to create a Blip2VisionConfig instance from a pretrained model.

        Args:
            cls (type): The class object itself.
            pretrained_model_name_or_path (Union[str, os.PathLike]):
                A string representing the name or path of the pretrained model to load.

        Returns:
            PretrainedConfig:
                An instance of the Blip2VisionConfig class initialized with the configuration obtained from the pretrained model.

        Raises:
            KeyError: If the 'model_type' key is not found in the configuration dictionary.
            AttributeError: If the 'model_type' key is present in the configuration dictionary
                but the class does not have a 'model_type' attribute.
            Warning: If the model type in the configuration dictionary does not match the class's model type,
                a warning is logged to alert the user.
        """
        config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)

        # get the vision config dict if we are loading from Blip2Config
        if config_dict.get("model_type") == "blip-2":
            config_dict = config_dict["vision_config"]

        if "model_type" in config_dict and hasattr(cls, "model_type") and config_dict["model_type"] != cls.model_type:
            logger.warning(
                f"You are using a model of type {config_dict['model_type']} to instantiate a model of type "
                f"{cls.model_type}. This is not supported for all configurations of models and can yield errors."
            )

        return cls.from_dict(config_dict, **kwargs)

mindnlp.transformers.models.blip_2.configuration_blip_2.Blip2VisionConfig.__init__(hidden_size=1408, intermediate_size=6144, num_hidden_layers=39, num_attention_heads=16, image_size=224, patch_size=14, hidden_act='gelu', layer_norm_eps=1e-06, attention_dropout=0.0, initializer_range=1e-10, qkv_bias=True, **kwargs)

Initializes an instance of the Blip2VisionConfig class.

PARAMETER	DESCRIPTION
self	The instance of the class.
hidden_size	The size of the hidden layers. Defaults to 1408. TYPE: int DEFAULT: 1408
intermediate_size	The size of the intermediate layers. Defaults to 6144. TYPE: int DEFAULT: 6144
num_hidden_layers	The number of hidden layers. Defaults to 39. TYPE: int DEFAULT: 39
num_attention_heads	The number of attention heads. Defaults to 16. TYPE: int DEFAULT: 16
image_size	The size of the image. Defaults to 224. TYPE: int DEFAULT: 224
patch_size	The size of the patches. Defaults to 14. TYPE: int DEFAULT: 14
hidden_act	The activation function for the hidden layers. Defaults to 'gelu'. TYPE: str DEFAULT: 'gelu'
layer_norm_eps	The epsilon value for layer normalization. Defaults to 1e-06. TYPE: float DEFAULT: 1e-06
attention_dropout	The dropout rate for attention layers. Defaults to 0.0. TYPE: float DEFAULT: 0.0
initializer_range	The range for weight initialization. Defaults to 1e-10. TYPE: float DEFAULT: 1e-10
qkv_bias	Whether to include bias in the query, key, value layers. Defaults to True. TYPE: bool DEFAULT: True

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/blip_2/configuration_blip_2.py

def __init__(
    self,
    hidden_size=1408,
    intermediate_size=6144,
    num_hidden_layers=39,
    num_attention_heads=16,
    image_size=224,
    patch_size=14,
    hidden_act="gelu",
    layer_norm_eps=1e-6,
    attention_dropout=0.0,
    initializer_range=1e-10,
    qkv_bias=True,
    **kwargs,
):
    """
    Initializes an instance of the Blip2VisionConfig class.

    Args:
        self: The instance of the class.
        hidden_size (int, optional): The size of the hidden layers. Defaults to 1408.
        intermediate_size (int, optional): The size of the intermediate layers. Defaults to 6144.
        num_hidden_layers (int, optional): The number of hidden layers. Defaults to 39.
        num_attention_heads (int, optional): The number of attention heads. Defaults to 16.
        image_size (int, optional): The size of the image. Defaults to 224.
        patch_size (int, optional): The size of the patches. Defaults to 14.
        hidden_act (str, optional): The activation function for the hidden layers. Defaults to 'gelu'.
        layer_norm_eps (float, optional): The epsilon value for layer normalization. Defaults to 1e-06.
        attention_dropout (float, optional): The dropout rate for attention layers. Defaults to 0.0.
        initializer_range (float, optional): The range for weight initialization. Defaults to 1e-10.
        qkv_bias (bool, optional): Whether to include bias in the query, key, value layers. Defaults to True.

    Returns:
        None

    Raises:
        None
    """
    super().__init__(**kwargs)

    self.hidden_size = hidden_size
    self.intermediate_size = intermediate_size
    self.num_hidden_layers = num_hidden_layers
    self.num_attention_heads = num_attention_heads
    self.patch_size = patch_size
    self.image_size = image_size
    self.initializer_range = initializer_range
    self.attention_dropout = attention_dropout
    self.layer_norm_eps = layer_norm_eps
    self.hidden_act = hidden_act
    self.qkv_bias = qkv_bias

mindnlp.transformers.models.blip_2.configuration_blip_2.Blip2VisionConfig.from_pretrained(pretrained_model_name_or_path, **kwargs) classmethod

Method to create a Blip2VisionConfig instance from a pretrained model.

PARAMETER	DESCRIPTION
cls	The class object itself. TYPE: type
pretrained_model_name_or_path	A string representing the name or path of the pretrained model to load. TYPE: Union[str, PathLike]

RETURNS	DESCRIPTION
PretrainedConfig	An instance of the Blip2VisionConfig class initialized with the configuration obtained from the pretrained model. TYPE: PretrainedConfig

RAISES	DESCRIPTION
KeyError	If the 'model_type' key is not found in the configuration dictionary.
AttributeError	If the 'model_type' key is present in the configuration dictionary but the class does not have a 'model_type' attribute.
Warning	If the model type in the configuration dictionary does not match the class's model type, a warning is logged to alert the user.

Source code in mindnlp/transformers/models/blip_2/configuration_blip_2.py

@classmethod
def from_pretrained(cls, pretrained_model_name_or_path: Union[str, os.PathLike], **kwargs) -> "PretrainedConfig":
    """
    Method to create a Blip2VisionConfig instance from a pretrained model.

    Args:
        cls (type): The class object itself.
        pretrained_model_name_or_path (Union[str, os.PathLike]):
            A string representing the name or path of the pretrained model to load.

    Returns:
        PretrainedConfig:
            An instance of the Blip2VisionConfig class initialized with the configuration obtained from the pretrained model.

    Raises:
        KeyError: If the 'model_type' key is not found in the configuration dictionary.
        AttributeError: If the 'model_type' key is present in the configuration dictionary
            but the class does not have a 'model_type' attribute.
        Warning: If the model type in the configuration dictionary does not match the class's model type,
            a warning is logged to alert the user.
    """
    config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)

    # get the vision config dict if we are loading from Blip2Config
    if config_dict.get("model_type") == "blip-2":
        config_dict = config_dict["vision_config"]

    if "model_type" in config_dict and hasattr(cls, "model_type") and config_dict["model_type"] != cls.model_type:
        logger.warning(
            f"You are using a model of type {config_dict['model_type']} to instantiate a model of type "
            f"{cls.model_type}. This is not supported for all configurations of models and can yield errors."
        )

    return cls.from_dict(config_dict, **kwargs)

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2Model

Bases: Blip2PreTrainedModel

A class representing the Blip2Model for multimodal language understanding tasks.

Blip2Model is a multimodal transformer model that combines vision and text inputs to generate high-level representations and perform tasks such as image captioning, visual question answering, and multimodal language generation.

This class extends the Blip2PreTrainedModel, which is the base class for all models in the Blip2 project.

ATTRIBUTE	DESCRIPTION
vision_model	The vision model that processes the image inputs. TYPE: Blip2VisionModel
query_tokens	Query tokens used in the QFormer model. TYPE: Parameter
qformer	The QFormer model that processes the query tokens and image embeddings. TYPE: Blip2QFormerModel
language_projection	Projection layer that maps the QFormer output to the input size of the language model. TYPE: Linear
language_model	The language model used for text processing. TYPE: Union[AutoModelForCausalLM, AutoModelForSeq2SeqLM]
_tied_weights_keys	List of tied weights keys for the language model. TYPE: List[str]

METHOD	DESCRIPTION
__init__	Initializes the Blip2Model with the given configuration.
get_input_embeddings	Returns the input embeddings of the language model.
set_input_embeddings	Sets the input embeddings of the language model.
set_output_embeddings	Sets the output embeddings of the language model.
get_output_embeddings	Returns the output embeddings of the language model.
get_encoder	Returns the encoder of the language model.
get_decoder	Returns the decoder of the language model.
_tie_weights	Ties the weights of the encoder and decoder in the language model.
get_text_features	Retrieves the text features from the language model.
get_image_features	Retrieves the image features from the vision model.
get_qformer_features	Retrieves the query transformer (QFormer) features from the vision model.
forward	Constructs the Blip2Model with the given inputs and returns the model outputs.

Example

>>> import torch
>>> from PIL import Image
>>> import requests
>>> from transformers import Blip2Processor, Blip2Model
...
>>> processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
>>> model = Blip2Model.from_pretrained("Salesforce/blip2-opt-2.7b")
...
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(images=image, return_tensors="pt")
>>> outputs = model.get_image_features(**inputs)

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

class Blip2Model(Blip2PreTrainedModel):

    """
    A class representing the Blip2Model for multimodal language understanding tasks.

    Blip2Model is a multimodal transformer model that combines vision and text inputs to generate high-level
    representations and perform tasks such as image captioning, visual question answering, and multimodal
    language generation.

    This class extends the Blip2PreTrainedModel, which is the base class for all models in the Blip2 project.

    Attributes:
        vision_model (Blip2VisionModel): The vision model that processes the image inputs.
        query_tokens (Parameter): Query tokens used in the QFormer model.
        qformer (Blip2QFormerModel): The QFormer model that processes the query tokens and image embeddings.
        language_projection (nn.Linear): Projection layer that maps the QFormer output to the input size of the language model.
        language_model (Union[AutoModelForCausalLM, AutoModelForSeq2SeqLM]): The language model used for text processing.
        _tied_weights_keys (List[str]): List of tied weights keys for the language model.

    Methods:
        __init__: Initializes the Blip2Model with the given configuration.
        get_input_embeddings: Returns the input embeddings of the language model.
        set_input_embeddings: Sets the input embeddings of the language model.
        set_output_embeddings: Sets the output embeddings of the language model.
        get_output_embeddings: Returns the output embeddings of the language model.
        get_encoder: Returns the encoder of the language model.
        get_decoder: Returns the decoder of the language model.
        _tie_weights: Ties the weights of the encoder and decoder in the language model.
        get_text_features: Retrieves the text features from the language model.
        get_image_features: Retrieves the image features from the vision model.
        get_qformer_features: Retrieves the query transformer (QFormer) features from the vision model.
        forward: Constructs the Blip2Model with the given inputs and returns the model outputs.

    Example:
        ```python
        >>> import torch
        >>> from PIL import Image
        >>> import requests
        >>> from transformers import Blip2Processor, Blip2Model
        ...
        >>> processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
        >>> model = Blip2Model.from_pretrained("Salesforce/blip2-opt-2.7b")
        ...
        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
        >>> image = Image.open(requests.get(url, stream=True).raw)
        >>> inputs = processor(images=image, return_tensors="pt")
        >>> outputs = model.get_image_features(**inputs)
        ```
    """
    config_class = Blip2Config
    main_input_name = "pixel_values"

    def __init__(self, config: Blip2Config):
        """
        Initialize the Blip2Model with the specified configuration.

        Args:
            self: The instance of the Blip2Model class.
            config (Blip2Config): An instance of Blip2Config containing the configuration parameters for the model.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(config)

        self.vision_model = Blip2VisionModel(config.vision_config)

        self.query_tokens = Parameter(ops.zeros(1, config.num_query_tokens, config.qformer_config.hidden_size))
        self.qformer = Blip2QFormerModel(config.qformer_config)

        self.language_projection = nn.Linear(config.qformer_config.hidden_size, config.text_config.hidden_size)
        if config.use_decoder_only_language_model:
            language_model = AutoModelForCausalLM.from_config(config.text_config)
        else:
            language_model = AutoModelForSeq2SeqLM.from_config(config.text_config)

        # Update _tied_weights_keys using the base model used.
        if language_model._tied_weights_keys is not None:
            self._tied_weights_keys = [f"language_model.{k}" for k in language_model._tied_weights_keys]

        self.language_model = language_model

        # Initialize weights and apply final processing
        self.post_init()

    def get_input_embeddings(self):
        """
        Get the input embeddings from the Blip2Model.

        Args:
            self (Blip2Model): An instance of the Blip2Model class.

        Returns:
            None.

        Raises:
            None.
        """
        return self.language_model.get_input_embeddings()

    def set_input_embeddings(self, value):
        """
        Set the input embeddings for the Blip2Model.

        Args:
            self (Blip2Model): The instance of the Blip2Model class.
            value: The input embeddings to be set for the language model.
                It should be of type torch.Tensor or any compatible type.

        Returns:
            None.

        Raises:
            TypeError: If the value provided is not of the expected type.
            ValueError: If the value provided is invalid or cannot be used for setting input embeddings.
        """
        self.language_model.set_input_embeddings(value)

    def set_output_embeddings(self, new_embeddings):
        """
        This method sets the output embeddings for the Blip2Model.

        Args:
            self (Blip2Model): The instance of the Blip2Model class.
            new_embeddings (object): The new output embeddings to be set for the language model.

        Returns:
            None.

        Raises:
            None.
        """
        self.language_model.set_output_embeddings(new_embeddings)

    def get_output_embeddings(self) -> nn.Module:
        """
        Returns the output embeddings of the Blip2Model.

        Args:
            self: Blip2Model - The instance of the Blip2Model class.

        Returns:
            nn.Module: The output embeddings of the Blip2Model language model.

        Raises:
            None.
        """
        return self.language_model.get_output_embeddings()

    def get_encoder(self):
        """
        Method to retrieve the encoder from the language model within the Blip2Model class.

        Args:
            self (Blip2Model):
                The instance of the Blip2Model class.

                - This parameter refers to the current instance of the Blip2Model class.

        Returns:
            None:
                This method returns None as it retrieves the encoder from the language model.

        Raises:
            None.
        """
        return self.language_model.get_encoder()

    def get_decoder(self):
        """
        This method returns the decoder from the language model.

        Args:
            self: The instance of the Blip2Model class.

        Returns:
            None:
                This method returns the decoder obtained from the language model.

        Raises:
            None
        """
        return self.language_model.get_decoder()

    def _tie_weights(self):
        """
        Method to tie weights in the Blip2Model class.

        Args:
            self: The instance of the Blip2Model class.

        Returns:
            None: This method does not return any value.

        Raises:
            None.
        """
        if not self.config.use_decoder_only_language_model:
            self.language_model.encoder.embed_tokens = self.language_model.shared
            self.language_model.decoder.embed_tokens = self.language_model.shared

    def get_text_features(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        decoder_input_ids: Optional[mindspore.Tensor] = None,
        decoder_attention_mask: Optional[mindspore.Tensor] = None,
        labels: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ):
        r"""

        Returns:
            text_outputs (`CausalLMOutputWithPast`, or `tuple(mindspore.Tensor)` if `return_dict=False`):
                The language model outputs. If `return_dict=True`, the output is a [`CausalLMOutputWithPast`] that
                contains the language model logits, the past key values and the hidden states if
                `output_hidden_states=True`.

        Example:
            ```python
            >>> import torch
            >>> from transformers import AutoTokenizer, Blip2Model
            ...
            >>> model = Blip2Model.from_pretrained("Salesforce/blip2-opt-2.7b")
            ...
            >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/blip2-opt-2.7b")
            >>> inputs = tokenizer(["a photo of a cat"], padding=True, return_tensors="pt")
            >>> text_features = model.get_text_features(**inputs)
            ```
        """
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        if self.config.use_decoder_only_language_model:
            text_outputs = self.language_model(
                input_ids=input_ids,
                attention_mask=attention_mask,
                output_attentions=output_attentions,
                output_hidden_states=output_hidden_states,
                return_dict=return_dict,
            )
        else:
            inputs_embeds = self.language_model.get_input_embeddings()(input_ids)

            text_outputs = self.language_model(
                inputs_embeds=inputs_embeds,
                attention_mask=attention_mask,
                decoder_input_ids=decoder_input_ids,
                decoder_attention_mask=decoder_attention_mask,
                output_attentions=output_attentions,
                output_hidden_states=output_hidden_states,
                return_dict=return_dict,
                labels=labels,
            )

        return text_outputs

    def get_image_features(
        self,
        pixel_values: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ):
        r"""

        Returns:
            vision_outputs (`BaseModelOutputWithPooling` or tuple of `mindspore.Tensor`):
                The vision model outputs. If `return_dict=True`, the output is a [`BaseModelOutputWithPooling`] that
                contains the image features, the pooled image features and the hidden states if
                `output_hidden_states=True`.

        Example:
            ```python
            >>> import torch
            >>> from PIL import Image
            >>> import requests
            >>> from transformers import AutoProcessor, Blip2Model
            ...
            >>> model = Blip2Model.from_pretrained("Salesforce/blip2-opt-2.7b")
            ...
            >>> processor = AutoProcessor.from_pretrained("Salesforce/blip2-opt-2.7b")
            >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
            >>> image = Image.open(requests.get(url, stream=True).raw)
            >>> inputs = processor(images=image, return_tensors="pt")
            >>> image_outputs = model.get_image_features(**inputs)
            ```
        """
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        vision_outputs = self.vision_model(
            pixel_values=pixel_values,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        return vision_outputs

    def get_qformer_features(
        self,
        pixel_values: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ):
        r"""

        Returns:
            vision_outputs (`BaseModelOutputWithPooling` or tuple of `mindspore.Tensor`):
                The vision model outputs. If `return_dict=True`, the output is a [`BaseModelOutputWithPooling`] that
                contains the image features, the pooled image features and the hidden states if
                `output_hidden_states=True`.

        Example:
            ```python
            >>> import torch
            >>> from PIL import Image
            >>> import requests
            >>> from transformers import Blip2Processor, Blip2Model
            ...
            >>> processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
            >>> model = Blip2Model.from_pretrained("Salesforce/blip2-opt-2.7b")
            ...
            >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
            >>> image = Image.open(requests.get(url, stream=True).raw)
            >>> inputs = processor(images=image, return_tensors="pt")
            >>> qformer_outputs = model.get_qformer_features(**inputs)
            ```
        """
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        vision_outputs = self.vision_model(
            pixel_values=pixel_values,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        image_embeds = vision_outputs[0]

        # step 2: forward the query tokens through the QFormer, using the image embeddings for cross-attention
        image_attention_mask = ops.ones(*image_embeds.shape[:-1], dtype=mindspore.int64)

        query_tokens = ops.broadcast_to(self.query_tokens, (image_embeds.shape[0], -1, -1))
        query_outputs = self.qformer(
            query_embeds=query_tokens,
            encoder_hidden_states=image_embeds,
            encoder_attention_mask=image_attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        return query_outputs

    def forward(
        self,
        pixel_values: mindspore.Tensor,
        input_ids: mindspore.Tensor,
        attention_mask: Optional[mindspore.Tensor] = None,
        decoder_input_ids: Optional[mindspore.Tensor] = None,
        decoder_attention_mask: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        labels: Optional[mindspore.Tensor] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, Blip2ForConditionalGenerationModelOutput]:
        r"""

        Returns:
            `Union[Tuple, Blip2ForConditionalGenerationModelOutput]`

        Example:
            ```python
            >>> from PIL import Image
            >>> import requests
            >>> from transformers import Blip2Processor, Blip2Model
            >>> import torch
            ...
            ...
            >>> processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
            >>> model = Blip2Model.from_pretrained("Salesforce/blip2-opt-2.7b", torch_dtype=torch.float16)
            ...
            >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
            >>> image = Image.open(requests.get(url, stream=True).raw)
            ...
            >>> prompt = "Question: how many cats are there? Answer:"
            >>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(torch.float16)
            ...
            >>> outputs = model(**inputs)
            ```
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        # step 1: forward the images through the vision encoder,
        # to get image embeddings of shape (batch_size, seq_len, hidden_size)
        vision_outputs = self.vision_model(
            pixel_values=pixel_values,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )
        image_embeds = vision_outputs[0]

        # step 2: forward the query tokens through the QFormer, using the image embeddings for cross-attention
        image_attention_mask = ops.ones(*image_embeds.shape[:-1], dtype=mindspore.int64)

        query_tokens = ops.broadcast_to(self.query_tokens, (image_embeds.shape[0], -1, -1))
        query_outputs = self.qformer(
            query_embeds=query_tokens,
            encoder_hidden_states=image_embeds,
            encoder_attention_mask=image_attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )
        query_output = query_outputs[0]

        # step 3: use the language model, conditioned on the query outputs and the prompt
        language_model_inputs = self.language_projection(query_output)
        language_model_attention_mask = ops.ones(
            *language_model_inputs.shape[:-1], dtype=mindspore.int64
        )
        inputs_embeds = self.language_model.get_input_embeddings()(input_ids)
        inputs_embeds = ops.cat([language_model_inputs, inputs_embeds], dim=1)

        if attention_mask is None:
            attention_mask = ops.ones_like(input_ids)
        attention_mask = ops.cat([language_model_attention_mask, attention_mask], dim=1)

        if self.config.use_decoder_only_language_model:
            outputs = self.language_model(
                inputs_embeds=inputs_embeds,
                attention_mask=attention_mask,
                output_attentions=output_attentions,
                output_hidden_states=output_hidden_states,
                return_dict=return_dict,
            )
            logits = outputs.logits if return_dict else outputs[0]
            loss = None
            # we compute the loss here since we need to take into account the sequence length of the query embeds
            if labels is not None:
                logits = logits[:, -labels.shape[1] :, :]
                # Shift so that tokens < n predict n
                shift_logits = logits[..., :-1, :]
                shift_labels = labels[..., 1:]

                # Flatten the tokens
                loss = F.cross_entropy(shift_logits.view(-1, self.config.text_config.vocab_size), shift_labels.view(-1))
        else:
            outputs = self.language_model(
                inputs_embeds=inputs_embeds,
                attention_mask=attention_mask,
                decoder_input_ids=decoder_input_ids,
                decoder_attention_mask=decoder_attention_mask,
                output_attentions=output_attentions,
                output_hidden_states=output_hidden_states,
                return_dict=return_dict,
                labels=labels,
            )
            loss = outputs.loss if return_dict else outputs[0]
            logits = outputs.logits if return_dict else outputs[1]

        if not return_dict:
            output = (logits, vision_outputs, query_outputs, outputs)
            return ((loss,) + output) if loss is not None else output

        return Blip2ForConditionalGenerationModelOutput(
            loss=loss,
            logits=logits,
            vision_outputs=vision_outputs,
            qformer_outputs=query_outputs,
            language_model_outputs=outputs,
        )

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2Model.__init__(config)

Initialize the Blip2Model with the specified configuration.

PARAMETER	DESCRIPTION
self	The instance of the Blip2Model class.
config	An instance of Blip2Config containing the configuration parameters for the model. TYPE: Blip2Config

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def __init__(self, config: Blip2Config):
    """
    Initialize the Blip2Model with the specified configuration.

    Args:
        self: The instance of the Blip2Model class.
        config (Blip2Config): An instance of Blip2Config containing the configuration parameters for the model.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(config)

    self.vision_model = Blip2VisionModel(config.vision_config)

    self.query_tokens = Parameter(ops.zeros(1, config.num_query_tokens, config.qformer_config.hidden_size))
    self.qformer = Blip2QFormerModel(config.qformer_config)

    self.language_projection = nn.Linear(config.qformer_config.hidden_size, config.text_config.hidden_size)
    if config.use_decoder_only_language_model:
        language_model = AutoModelForCausalLM.from_config(config.text_config)
    else:
        language_model = AutoModelForSeq2SeqLM.from_config(config.text_config)

    # Update _tied_weights_keys using the base model used.
    if language_model._tied_weights_keys is not None:
        self._tied_weights_keys = [f"language_model.{k}" for k in language_model._tied_weights_keys]

    self.language_model = language_model

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2Model.forward(pixel_values, input_ids, attention_mask=None, decoder_input_ids=None, decoder_attention_mask=None, output_attentions=None, output_hidden_states=None, labels=None, return_dict=None)

RETURNS	DESCRIPTION
Union[Tuple, Blip2ForConditionalGenerationModelOutput]	Union[Tuple, Blip2ForConditionalGenerationModelOutput]

Example

>>> from PIL import Image
>>> import requests
>>> from transformers import Blip2Processor, Blip2Model
>>> import torch
...
...
>>> processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
>>> model = Blip2Model.from_pretrained("Salesforce/blip2-opt-2.7b", torch_dtype=torch.float16)
...
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
...
>>> prompt = "Question: how many cats are there? Answer:"
>>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(torch.float16)
...
>>> outputs = model(**inputs)

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def forward(
    self,
    pixel_values: mindspore.Tensor,
    input_ids: mindspore.Tensor,
    attention_mask: Optional[mindspore.Tensor] = None,
    decoder_input_ids: Optional[mindspore.Tensor] = None,
    decoder_attention_mask: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    labels: Optional[mindspore.Tensor] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple, Blip2ForConditionalGenerationModelOutput]:
    r"""

    Returns:
        `Union[Tuple, Blip2ForConditionalGenerationModelOutput]`

    Example:
        ```python
        >>> from PIL import Image
        >>> import requests
        >>> from transformers import Blip2Processor, Blip2Model
        >>> import torch
        ...
        ...
        >>> processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
        >>> model = Blip2Model.from_pretrained("Salesforce/blip2-opt-2.7b", torch_dtype=torch.float16)
        ...
        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
        >>> image = Image.open(requests.get(url, stream=True).raw)
        ...
        >>> prompt = "Question: how many cats are there? Answer:"
        >>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(torch.float16)
        ...
        >>> outputs = model(**inputs)
        ```
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    # step 1: forward the images through the vision encoder,
    # to get image embeddings of shape (batch_size, seq_len, hidden_size)
    vision_outputs = self.vision_model(
        pixel_values=pixel_values,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )
    image_embeds = vision_outputs[0]

    # step 2: forward the query tokens through the QFormer, using the image embeddings for cross-attention
    image_attention_mask = ops.ones(*image_embeds.shape[:-1], dtype=mindspore.int64)

    query_tokens = ops.broadcast_to(self.query_tokens, (image_embeds.shape[0], -1, -1))
    query_outputs = self.qformer(
        query_embeds=query_tokens,
        encoder_hidden_states=image_embeds,
        encoder_attention_mask=image_attention_mask,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )
    query_output = query_outputs[0]

    # step 3: use the language model, conditioned on the query outputs and the prompt
    language_model_inputs = self.language_projection(query_output)
    language_model_attention_mask = ops.ones(
        *language_model_inputs.shape[:-1], dtype=mindspore.int64
    )
    inputs_embeds = self.language_model.get_input_embeddings()(input_ids)
    inputs_embeds = ops.cat([language_model_inputs, inputs_embeds], dim=1)

    if attention_mask is None:
        attention_mask = ops.ones_like(input_ids)
    attention_mask = ops.cat([language_model_attention_mask, attention_mask], dim=1)

    if self.config.use_decoder_only_language_model:
        outputs = self.language_model(
            inputs_embeds=inputs_embeds,
            attention_mask=attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )
        logits = outputs.logits if return_dict else outputs[0]
        loss = None
        # we compute the loss here since we need to take into account the sequence length of the query embeds
        if labels is not None:
            logits = logits[:, -labels.shape[1] :, :]
            # Shift so that tokens < n predict n
            shift_logits = logits[..., :-1, :]
            shift_labels = labels[..., 1:]

            # Flatten the tokens
            loss = F.cross_entropy(shift_logits.view(-1, self.config.text_config.vocab_size), shift_labels.view(-1))
    else:
        outputs = self.language_model(
            inputs_embeds=inputs_embeds,
            attention_mask=attention_mask,
            decoder_input_ids=decoder_input_ids,
            decoder_attention_mask=decoder_attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
            labels=labels,
        )
        loss = outputs.loss if return_dict else outputs[0]
        logits = outputs.logits if return_dict else outputs[1]

    if not return_dict:
        output = (logits, vision_outputs, query_outputs, outputs)
        return ((loss,) + output) if loss is not None else output

    return Blip2ForConditionalGenerationModelOutput(
        loss=loss,
        logits=logits,
        vision_outputs=vision_outputs,
        qformer_outputs=query_outputs,
        language_model_outputs=outputs,
    )

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2Model.get_decoder()

This method returns the decoder from the language model.

PARAMETER	DESCRIPTION
self	The instance of the Blip2Model class.

RETURNS	DESCRIPTION
None	This method returns the decoder obtained from the language model.

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def get_decoder(self):
    """
    This method returns the decoder from the language model.

    Args:
        self: The instance of the Blip2Model class.

    Returns:
        None:
            This method returns the decoder obtained from the language model.

    Raises:
        None
    """
    return self.language_model.get_decoder()

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2Model.get_encoder()

Method to retrieve the encoder from the language model within the Blip2Model class.

PARAMETER	DESCRIPTION
self	The instance of the Blip2Model class. This parameter refers to the current instance of the Blip2Model class. TYPE: Blip2Model

RETURNS	DESCRIPTION
None	This method returns None as it retrieves the encoder from the language model.

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def get_encoder(self):
    """
    Method to retrieve the encoder from the language model within the Blip2Model class.

    Args:
        self (Blip2Model):
            The instance of the Blip2Model class.

            - This parameter refers to the current instance of the Blip2Model class.

    Returns:
        None:
            This method returns None as it retrieves the encoder from the language model.

    Raises:
        None.
    """
    return self.language_model.get_encoder()

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2Model.get_image_features(pixel_values=None, output_attentions=None, output_hidden_states=None, return_dict=None)

RETURNS	DESCRIPTION
vision_outputs	The vision model outputs. If return_dict=True, the output is a [BaseModelOutputWithPooling] that contains the image features, the pooled image features and the hidden states if output_hidden_states=True. TYPE: `BaseModelOutputWithPooling` or tuple of `mindspore.Tensor`

Example

>>> import torch
>>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, Blip2Model
...
>>> model = Blip2Model.from_pretrained("Salesforce/blip2-opt-2.7b")
...
>>> processor = AutoProcessor.from_pretrained("Salesforce/blip2-opt-2.7b")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(images=image, return_tensors="pt")
>>> image_outputs = model.get_image_features(**inputs)

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def get_image_features(
    self,
    pixel_values: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
):
    r"""

    Returns:
        vision_outputs (`BaseModelOutputWithPooling` or tuple of `mindspore.Tensor`):
            The vision model outputs. If `return_dict=True`, the output is a [`BaseModelOutputWithPooling`] that
            contains the image features, the pooled image features and the hidden states if
            `output_hidden_states=True`.

    Example:
        ```python
        >>> import torch
        >>> from PIL import Image
        >>> import requests
        >>> from transformers import AutoProcessor, Blip2Model
        ...
        >>> model = Blip2Model.from_pretrained("Salesforce/blip2-opt-2.7b")
        ...
        >>> processor = AutoProcessor.from_pretrained("Salesforce/blip2-opt-2.7b")
        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
        >>> image = Image.open(requests.get(url, stream=True).raw)
        >>> inputs = processor(images=image, return_tensors="pt")
        >>> image_outputs = model.get_image_features(**inputs)
        ```
    """
    output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
    output_hidden_states = (
        output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
    )
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    vision_outputs = self.vision_model(
        pixel_values=pixel_values,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    return vision_outputs

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2Model.get_input_embeddings()

Get the input embeddings from the Blip2Model.

PARAMETER	DESCRIPTION
self	An instance of the Blip2Model class. TYPE: Blip2Model

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def get_input_embeddings(self):
    """
    Get the input embeddings from the Blip2Model.

    Args:
        self (Blip2Model): An instance of the Blip2Model class.

    Returns:
        None.

    Raises:
        None.
    """
    return self.language_model.get_input_embeddings()

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2Model.get_output_embeddings()

Returns the output embeddings of the Blip2Model.

PARAMETER	DESCRIPTION
self	Blip2Model - The instance of the Blip2Model class.

RETURNS	DESCRIPTION
Module	nn.Module: The output embeddings of the Blip2Model language model.

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def get_output_embeddings(self) -> nn.Module:
    """
    Returns the output embeddings of the Blip2Model.

    Args:
        self: Blip2Model - The instance of the Blip2Model class.

    Returns:
        nn.Module: The output embeddings of the Blip2Model language model.

    Raises:
        None.
    """
    return self.language_model.get_output_embeddings()

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2Model.get_qformer_features(pixel_values=None, output_attentions=None, output_hidden_states=None, return_dict=None)

RETURNS	DESCRIPTION
vision_outputs	The vision model outputs. If return_dict=True, the output is a [BaseModelOutputWithPooling] that contains the image features, the pooled image features and the hidden states if output_hidden_states=True. TYPE: `BaseModelOutputWithPooling` or tuple of `mindspore.Tensor`

Example

>>> import torch
>>> from PIL import Image
>>> import requests
>>> from transformers import Blip2Processor, Blip2Model
...
>>> processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
>>> model = Blip2Model.from_pretrained("Salesforce/blip2-opt-2.7b")
...
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(images=image, return_tensors="pt")
>>> qformer_outputs = model.get_qformer_features(**inputs)

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def get_qformer_features(
    self,
    pixel_values: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
):
    r"""

    Returns:
        vision_outputs (`BaseModelOutputWithPooling` or tuple of `mindspore.Tensor`):
            The vision model outputs. If `return_dict=True`, the output is a [`BaseModelOutputWithPooling`] that
            contains the image features, the pooled image features and the hidden states if
            `output_hidden_states=True`.

    Example:
        ```python
        >>> import torch
        >>> from PIL import Image
        >>> import requests
        >>> from transformers import Blip2Processor, Blip2Model
        ...
        >>> processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
        >>> model = Blip2Model.from_pretrained("Salesforce/blip2-opt-2.7b")
        ...
        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
        >>> image = Image.open(requests.get(url, stream=True).raw)
        >>> inputs = processor(images=image, return_tensors="pt")
        >>> qformer_outputs = model.get_qformer_features(**inputs)
        ```
    """
    output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
    output_hidden_states = (
        output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
    )
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    vision_outputs = self.vision_model(
        pixel_values=pixel_values,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    image_embeds = vision_outputs[0]

    # step 2: forward the query tokens through the QFormer, using the image embeddings for cross-attention
    image_attention_mask = ops.ones(*image_embeds.shape[:-1], dtype=mindspore.int64)

    query_tokens = ops.broadcast_to(self.query_tokens, (image_embeds.shape[0], -1, -1))
    query_outputs = self.qformer(
        query_embeds=query_tokens,
        encoder_hidden_states=image_embeds,
        encoder_attention_mask=image_attention_mask,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    return query_outputs

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2Model.get_text_features(input_ids=None, attention_mask=None, decoder_input_ids=None, decoder_attention_mask=None, labels=None, output_attentions=None, output_hidden_states=None, return_dict=None)

RETURNS	DESCRIPTION
text_outputs	The language model outputs. If return_dict=True, the output is a [CausalLMOutputWithPast] that contains the language model logits, the past key values and the hidden states if output_hidden_states=True. TYPE: `CausalLMOutputWithPast`, or `tuple(mindspore.Tensor)` if `return_dict=False`

Example

>>> import torch
>>> from transformers import AutoTokenizer, Blip2Model
...
>>> model = Blip2Model.from_pretrained("Salesforce/blip2-opt-2.7b")
...
>>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/blip2-opt-2.7b")
>>> inputs = tokenizer(["a photo of a cat"], padding=True, return_tensors="pt")
>>> text_features = model.get_text_features(**inputs)

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def get_text_features(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    decoder_input_ids: Optional[mindspore.Tensor] = None,
    decoder_attention_mask: Optional[mindspore.Tensor] = None,
    labels: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
):
    r"""

    Returns:
        text_outputs (`CausalLMOutputWithPast`, or `tuple(mindspore.Tensor)` if `return_dict=False`):
            The language model outputs. If `return_dict=True`, the output is a [`CausalLMOutputWithPast`] that
            contains the language model logits, the past key values and the hidden states if
            `output_hidden_states=True`.

    Example:
        ```python
        >>> import torch
        >>> from transformers import AutoTokenizer, Blip2Model
        ...
        >>> model = Blip2Model.from_pretrained("Salesforce/blip2-opt-2.7b")
        ...
        >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/blip2-opt-2.7b")
        >>> inputs = tokenizer(["a photo of a cat"], padding=True, return_tensors="pt")
        >>> text_features = model.get_text_features(**inputs)
        ```
    """
    output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
    output_hidden_states = (
        output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
    )
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    if self.config.use_decoder_only_language_model:
        text_outputs = self.language_model(
            input_ids=input_ids,
            attention_mask=attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )
    else:
        inputs_embeds = self.language_model.get_input_embeddings()(input_ids)

        text_outputs = self.language_model(
            inputs_embeds=inputs_embeds,
            attention_mask=attention_mask,
            decoder_input_ids=decoder_input_ids,
            decoder_attention_mask=decoder_attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
            labels=labels,
        )

    return text_outputs

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2Model.set_input_embeddings(value)

Set the input embeddings for the Blip2Model.

PARAMETER	DESCRIPTION
self	The instance of the Blip2Model class. TYPE: Blip2Model
value	The input embeddings to be set for the language model. It should be of type torch.Tensor or any compatible type.

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
TypeError	If the value provided is not of the expected type.
ValueError	If the value provided is invalid or cannot be used for setting input embeddings.

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def set_input_embeddings(self, value):
    """
    Set the input embeddings for the Blip2Model.

    Args:
        self (Blip2Model): The instance of the Blip2Model class.
        value: The input embeddings to be set for the language model.
            It should be of type torch.Tensor or any compatible type.

    Returns:
        None.

    Raises:
        TypeError: If the value provided is not of the expected type.
        ValueError: If the value provided is invalid or cannot be used for setting input embeddings.
    """
    self.language_model.set_input_embeddings(value)

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2Model.set_output_embeddings(new_embeddings)

This method sets the output embeddings for the Blip2Model.

PARAMETER	DESCRIPTION
self	The instance of the Blip2Model class. TYPE: Blip2Model
new_embeddings	The new output embeddings to be set for the language model. TYPE: object

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def set_output_embeddings(self, new_embeddings):
    """
    This method sets the output embeddings for the Blip2Model.

    Args:
        self (Blip2Model): The instance of the Blip2Model class.
        new_embeddings (object): The new output embeddings to be set for the language model.

    Returns:
        None.

    Raises:
        None.
    """
    self.language_model.set_output_embeddings(new_embeddings)

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2QFormerModel

Bases: Blip2PreTrainedModel

Querying Transformer (Q-Former), used in BLIP-2.

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

class Blip2QFormerModel(Blip2PreTrainedModel):
    """
    Querying Transformer (Q-Former), used in BLIP-2.
    """
    def __init__(self, config: Blip2QFormerConfig):
        """
        Initializes a new instance of the Blip2QFormerModel class.

        Args:
            self: The instance of the class.
            config (Blip2QFormerConfig): The configuration object containing the model settings.
                It should be an instance of Blip2QFormerConfig class.

        Returns:
            None.

        Raises:
            None.

        Description:
            This method initializes the Blip2QFormerModel instance with the provided configuration.
            It sets the instance variables as follows:

            - self.config: The provided config object is stored as an instance variable.
            - self.layernorm: A LayerNorm module is created with the hidden size specified in the config.
            The epsilon value for LayerNorm is set to the value specified in the config.
            - self.dropout: A Dropout module is created with the dropout probability specified in the config.
            - self.encoder: An instance of Blip2QFormerEncoder class is created with the provided config.

        Note:
            After the initialization, self.post_init() method is called to perform any additional
            post-initialization steps.
        """
        super().__init__(config)
        self.config = config

        self.layernorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
        self.dropout = nn.Dropout(p=config.hidden_dropout_prob)

        self.encoder = Blip2QFormerEncoder(config)

        self.post_init()

    def get_input_embeddings(self):
        """
        This method retrieves the input embeddings for the Blip2QFormerModel.

        Args:
            self: The reference to the current instance of the Blip2QFormerModel class.

        Returns:
            word_embeddings: The method returns the word embeddings for the input.

        Raises:
            None.
        """
        return self.embeddings.word_embeddings

    def set_input_embeddings(self, value):
        """
        Set the input embeddings for the Blip2QFormerModel.

        Args:
            self (Blip2QFormerModel): The instance of the Blip2QFormerModel class.
            value: The input embeddings to be set. It can be of any valid type.

        Returns:
            None.

        Raises:
            None.
        """
        self.embeddings.word_embeddings = value

    def _prune_heads(self, heads_to_prune):
        """
        Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base
        class PreTrainedModel
        """
        for layer, heads in heads_to_prune.items():
            self.encoder.layer[layer].attention.prune_heads(heads)

    def get_extended_attention_mask(
        self,
        attention_mask: mindspore.Tensor,
        input_shape: Tuple[int],
        has_query: bool = False,
    ) -> mindspore.Tensor:
        """
        Makes broadcastable attention and causal masks so that future and masked tokens are ignored.

        Arguments:
            attention_mask (`mindspore.Tensor`):
                Mask with ones indicating tokens to attend to, zeros for tokens to ignore.
            input_shape (`Tuple[int]`):
                The shape of the input to the model.

        Returns:
            `mindspore.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`.
        """
        # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length]
        # ourselves in which case we just need to make it broadcastable to all heads.
        if attention_mask.ndim == 3:
            extended_attention_mask = attention_mask[:, None, :, :]
        elif attention_mask.ndim == 2:
            # Provided a padding mask of dimensions [batch_size, seq_length]
            # - the model is an encoder, so make the mask broadcastable to [batch_size, num_heads, seq_length, seq_length]
            extended_attention_mask = attention_mask[:, None, None, :]
        else:
            raise ValueError(
                "Wrong shape for input_ids (shape {}) or attention_mask (shape {})".format(
                    input_shape, attention_mask.shape
                )
            )

        # Since attention_mask is 1.0 for positions we want to attend and 0.0 for
        # masked positions, this operation will create a tensor which is 0.0 for
        # positions we want to attend and -10000.0 for masked positions.
        # Since we are adding it to the raw scores before the softmax, this is
        # effectively the same as removing these entirely.
        extended_attention_mask = extended_attention_mask.to(dtype=self.dtype)  # fp16 compatibility
        extended_attention_mask = (1.0 - extended_attention_mask) * -10000.0
        return extended_attention_mask

    def forward(
        self,
        query_embeds: mindspore.Tensor,
        attention_mask: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        encoder_hidden_states: Optional[mindspore.Tensor] = None,
        encoder_attention_mask: Optional[mindspore.Tensor] = None,
        past_key_values: Optional[Tuple[Tuple[mindspore.Tensor]]] = None,
        use_cache: Optional[bool] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple[mindspore.Tensor], BaseModelOutputWithPoolingAndCrossAttentions]:
        r"""
        Args:
            encoder_hidden_states  (`mindspore.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, `optional`):
                Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if
                the model is configured as a decoder.
            encoder_attention_mask (`mindspore.Tensor` of shape `(batch_size, sequence_length)`, `optional`):
                Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in
                the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`:

                - 1 for tokens that are **not masked**,
                - 0 for tokens that are **masked**.
            past_key_values (`tuple(tuple(mindspore.Tensor))` of length `config.n_layers` with each tuple having 4 tensors of:
                shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and
                value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are
                used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key
                value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape
                `(batch_size, sequence_length)`.
            use_cache (`bool`, `optional`):
                If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see
                `past_key_values`).
        """
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        # past_key_values_length
        past_key_values_length = (
            past_key_values[0][0].shape[2] - self.config.query_length if past_key_values is not None else 0
        )

        query_length = query_embeds.shape[1] if query_embeds is not None else 0

        embedding_output = self.layernorm(query_embeds)
        embedding_output = self.dropout(embedding_output)

        input_shape = embedding_output.shape[:-1]
        batch_size, seq_length = input_shape

        if attention_mask is None:
            attention_mask = ops.ones(batch_size, seq_length + past_key_values_length)

        # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length]
        # ourselves in which case we just need to make it broadcastable to all heads.
        extended_attention_mask = self.get_extended_attention_mask(attention_mask, input_shape)

        # If a 2D or 3D attention mask is provided for the cross-attention
        # we need to make broadcastable to [batch_size, num_heads, seq_length, seq_length]
        if encoder_hidden_states is not None:
            if isinstance(encoder_hidden_states, list):
                encoder_batch_size, encoder_sequence_length, _ = encoder_hidden_states[0].shape
            else:
                encoder_batch_size, encoder_sequence_length, _ = encoder_hidden_states.shape
            encoder_hidden_shape = (encoder_batch_size, encoder_sequence_length)

            if isinstance(encoder_attention_mask, list):
                encoder_extended_attention_mask = [self.invert_attention_mask(mask) for mask in encoder_attention_mask]
            elif encoder_attention_mask is None:
                encoder_attention_mask = ops.ones(*encoder_hidden_shape)
                encoder_extended_attention_mask = self.invert_attention_mask(encoder_attention_mask)
            else:
                encoder_extended_attention_mask = self.invert_attention_mask(encoder_attention_mask)
        else:
            encoder_extended_attention_mask = None

        # Prepare head mask if needed
        # 1.0 in head_mask indicate we keep the head
        # attention_probs has shape bsz x n_heads x N x N
        # input head_mask has shape [num_heads] or [num_hidden_layers x num_heads]
        # and head_mask is converted to shape [num_hidden_layers x batch x num_heads x seq_length x seq_length]
        head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)

        encoder_outputs = self.encoder(
            embedding_output,
            attention_mask=extended_attention_mask,
            head_mask=head_mask,
            encoder_hidden_states=encoder_hidden_states,
            encoder_attention_mask=encoder_extended_attention_mask,
            past_key_values=past_key_values,
            use_cache=use_cache,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
            query_length=query_length,
        )
        sequence_output = encoder_outputs[0]
        pooled_output = sequence_output[:, 0, :]

        if not return_dict:
            return (sequence_output, pooled_output) + encoder_outputs[1:]

        return BaseModelOutputWithPoolingAndCrossAttentions(
            last_hidden_state=sequence_output,
            pooler_output=pooled_output,
            past_key_values=encoder_outputs.past_key_values,
            hidden_states=encoder_outputs.hidden_states,
            attentions=encoder_outputs.attentions,
            cross_attentions=encoder_outputs.cross_attentions,
        )

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2QFormerModel.__init__(config)

Initializes a new instance of the Blip2QFormerModel class.

PARAMETER	DESCRIPTION
self	The instance of the class.
config	The configuration object containing the model settings. It should be an instance of Blip2QFormerConfig class. TYPE: Blip2QFormerConfig

RETURNS	DESCRIPTION
	None.

Description

This method initializes the Blip2QFormerModel instance with the provided configuration. It sets the instance variables as follows:

• self.config: The provided config object is stored as an instance variable.
• self.layernorm: A LayerNorm module is created with the hidden size specified in the config. The epsilon value for LayerNorm is set to the value specified in the config.
• self.dropout: A Dropout module is created with the dropout probability specified in the config.
• self.encoder: An instance of Blip2QFormerEncoder class is created with the provided config.

Note

After the initialization, self.post_init() method is called to perform any additional post-initialization steps.

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def __init__(self, config: Blip2QFormerConfig):
    """
    Initializes a new instance of the Blip2QFormerModel class.

    Args:
        self: The instance of the class.
        config (Blip2QFormerConfig): The configuration object containing the model settings.
            It should be an instance of Blip2QFormerConfig class.

    Returns:
        None.

    Raises:
        None.

    Description:
        This method initializes the Blip2QFormerModel instance with the provided configuration.
        It sets the instance variables as follows:

        - self.config: The provided config object is stored as an instance variable.
        - self.layernorm: A LayerNorm module is created with the hidden size specified in the config.
        The epsilon value for LayerNorm is set to the value specified in the config.
        - self.dropout: A Dropout module is created with the dropout probability specified in the config.
        - self.encoder: An instance of Blip2QFormerEncoder class is created with the provided config.

    Note:
        After the initialization, self.post_init() method is called to perform any additional
        post-initialization steps.
    """
    super().__init__(config)
    self.config = config

    self.layernorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
    self.dropout = nn.Dropout(p=config.hidden_dropout_prob)

    self.encoder = Blip2QFormerEncoder(config)

    self.post_init()

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2QFormerModel.forward(query_embeds, attention_mask=None, head_mask=None, encoder_hidden_states=None, encoder_attention_mask=None, past_key_values=None, use_cache=None, output_attentions=None, output_hidden_states=None, return_dict=None)

PARAMETER	DESCRIPTION
encoder_hidden_states	Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. TYPE: (`mindspore.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, `optional` DEFAULT: None
encoder_attention_mask	Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in [0, 1]: 1 for tokens that are not masked, 0 for tokens that are masked. TYPE: `mindspore.Tensor` of shape `(batch_size, sequence_length)`, `optional` DEFAULT: None
past_key_values	shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head)): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that don't have their past key value states given to this model) of shape (batch_size, 1) instead of all decoder_input_ids of shape (batch_size, sequence_length). TYPE: `tuple(tuple(mindspore.Tensor))` of length `config.n_layers` with each tuple having 4 tensors of DEFAULT: None
use_cache	If set to True, past_key_values key value states are returned and can be used to speed up decoding (see past_key_values). TYPE: `bool`, `optional` DEFAULT: None

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def forward(
    self,
    query_embeds: mindspore.Tensor,
    attention_mask: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    encoder_hidden_states: Optional[mindspore.Tensor] = None,
    encoder_attention_mask: Optional[mindspore.Tensor] = None,
    past_key_values: Optional[Tuple[Tuple[mindspore.Tensor]]] = None,
    use_cache: Optional[bool] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple[mindspore.Tensor], BaseModelOutputWithPoolingAndCrossAttentions]:
    r"""
    Args:
        encoder_hidden_states  (`mindspore.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, `optional`):
            Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if
            the model is configured as a decoder.
        encoder_attention_mask (`mindspore.Tensor` of shape `(batch_size, sequence_length)`, `optional`):
            Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in
            the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`:

            - 1 for tokens that are **not masked**,
            - 0 for tokens that are **masked**.
        past_key_values (`tuple(tuple(mindspore.Tensor))` of length `config.n_layers` with each tuple having 4 tensors of:
            shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and
            value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are
            used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key
            value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape
            `(batch_size, sequence_length)`.
        use_cache (`bool`, `optional`):
            If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see
            `past_key_values`).
    """
    output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
    output_hidden_states = (
        output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
    )
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    # past_key_values_length
    past_key_values_length = (
        past_key_values[0][0].shape[2] - self.config.query_length if past_key_values is not None else 0
    )

    query_length = query_embeds.shape[1] if query_embeds is not None else 0

    embedding_output = self.layernorm(query_embeds)
    embedding_output = self.dropout(embedding_output)

    input_shape = embedding_output.shape[:-1]
    batch_size, seq_length = input_shape

    if attention_mask is None:
        attention_mask = ops.ones(batch_size, seq_length + past_key_values_length)

    # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length]
    # ourselves in which case we just need to make it broadcastable to all heads.
    extended_attention_mask = self.get_extended_attention_mask(attention_mask, input_shape)

    # If a 2D or 3D attention mask is provided for the cross-attention
    # we need to make broadcastable to [batch_size, num_heads, seq_length, seq_length]
    if encoder_hidden_states is not None:
        if isinstance(encoder_hidden_states, list):
            encoder_batch_size, encoder_sequence_length, _ = encoder_hidden_states[0].shape
        else:
            encoder_batch_size, encoder_sequence_length, _ = encoder_hidden_states.shape
        encoder_hidden_shape = (encoder_batch_size, encoder_sequence_length)

        if isinstance(encoder_attention_mask, list):
            encoder_extended_attention_mask = [self.invert_attention_mask(mask) for mask in encoder_attention_mask]
        elif encoder_attention_mask is None:
            encoder_attention_mask = ops.ones(*encoder_hidden_shape)
            encoder_extended_attention_mask = self.invert_attention_mask(encoder_attention_mask)
        else:
            encoder_extended_attention_mask = self.invert_attention_mask(encoder_attention_mask)
    else:
        encoder_extended_attention_mask = None

    # Prepare head mask if needed
    # 1.0 in head_mask indicate we keep the head
    # attention_probs has shape bsz x n_heads x N x N
    # input head_mask has shape [num_heads] or [num_hidden_layers x num_heads]
    # and head_mask is converted to shape [num_hidden_layers x batch x num_heads x seq_length x seq_length]
    head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)

    encoder_outputs = self.encoder(
        embedding_output,
        attention_mask=extended_attention_mask,
        head_mask=head_mask,
        encoder_hidden_states=encoder_hidden_states,
        encoder_attention_mask=encoder_extended_attention_mask,
        past_key_values=past_key_values,
        use_cache=use_cache,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
        query_length=query_length,
    )
    sequence_output = encoder_outputs[0]
    pooled_output = sequence_output[:, 0, :]

    if not return_dict:
        return (sequence_output, pooled_output) + encoder_outputs[1:]

    return BaseModelOutputWithPoolingAndCrossAttentions(
        last_hidden_state=sequence_output,
        pooler_output=pooled_output,
        past_key_values=encoder_outputs.past_key_values,
        hidden_states=encoder_outputs.hidden_states,
        attentions=encoder_outputs.attentions,
        cross_attentions=encoder_outputs.cross_attentions,
    )

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2QFormerModel.get_extended_attention_mask(attention_mask, input_shape, has_query=False)

Makes broadcastable attention and causal masks so that future and masked tokens are ignored.

PARAMETER	DESCRIPTION
attention_mask	Mask with ones indicating tokens to attend to, zeros for tokens to ignore. TYPE: `mindspore.Tensor`
input_shape	The shape of the input to the model. TYPE: `Tuple[int]`

RETURNS	DESCRIPTION
Tensor	mindspore.Tensor The extended attention mask, with a the same dtype as attention_mask.dtype.

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def get_extended_attention_mask(
    self,
    attention_mask: mindspore.Tensor,
    input_shape: Tuple[int],
    has_query: bool = False,
) -> mindspore.Tensor:
    """
    Makes broadcastable attention and causal masks so that future and masked tokens are ignored.

    Arguments:
        attention_mask (`mindspore.Tensor`):
            Mask with ones indicating tokens to attend to, zeros for tokens to ignore.
        input_shape (`Tuple[int]`):
            The shape of the input to the model.

    Returns:
        `mindspore.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`.
    """
    # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length]
    # ourselves in which case we just need to make it broadcastable to all heads.
    if attention_mask.ndim == 3:
        extended_attention_mask = attention_mask[:, None, :, :]
    elif attention_mask.ndim == 2:
        # Provided a padding mask of dimensions [batch_size, seq_length]
        # - the model is an encoder, so make the mask broadcastable to [batch_size, num_heads, seq_length, seq_length]
        extended_attention_mask = attention_mask[:, None, None, :]
    else:
        raise ValueError(
            "Wrong shape for input_ids (shape {}) or attention_mask (shape {})".format(
                input_shape, attention_mask.shape
            )
        )

    # Since attention_mask is 1.0 for positions we want to attend and 0.0 for
    # masked positions, this operation will create a tensor which is 0.0 for
    # positions we want to attend and -10000.0 for masked positions.
    # Since we are adding it to the raw scores before the softmax, this is
    # effectively the same as removing these entirely.
    extended_attention_mask = extended_attention_mask.to(dtype=self.dtype)  # fp16 compatibility
    extended_attention_mask = (1.0 - extended_attention_mask) * -10000.0
    return extended_attention_mask

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2QFormerModel.get_input_embeddings()

This method retrieves the input embeddings for the Blip2QFormerModel.

PARAMETER	DESCRIPTION
self	The reference to the current instance of the Blip2QFormerModel class.

RETURNS	DESCRIPTION
word_embeddings	The method returns the word embeddings for the input.

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def get_input_embeddings(self):
    """
    This method retrieves the input embeddings for the Blip2QFormerModel.

    Args:
        self: The reference to the current instance of the Blip2QFormerModel class.

    Returns:
        word_embeddings: The method returns the word embeddings for the input.

    Raises:
        None.
    """
    return self.embeddings.word_embeddings

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2QFormerModel.set_input_embeddings(value)

Set the input embeddings for the Blip2QFormerModel.

PARAMETER	DESCRIPTION
self	The instance of the Blip2QFormerModel class. TYPE: Blip2QFormerModel
value	The input embeddings to be set. It can be of any valid type.

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def set_input_embeddings(self, value):
    """
    Set the input embeddings for the Blip2QFormerModel.

    Args:
        self (Blip2QFormerModel): The instance of the Blip2QFormerModel class.
        value: The input embeddings to be set. It can be of any valid type.

    Returns:
        None.

    Raises:
        None.
    """
    self.embeddings.word_embeddings = value

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2PreTrainedModel

Bases: PreTrainedModel

An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models.

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

class Blip2PreTrainedModel(PreTrainedModel):
    """
    An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained
    models.
    """
    config_class = Blip2Config
    base_model_prefix = "blip"
    supports_gradient_checkpointing = True
    _no_split_modules = ["Blip2Attention", "T5Block", "OPTDecoderLayer"]
    _skip_keys_device_placement = "past_key_values"
    _keep_in_fp32_modules = ["wo"]

    def _init_weights(self, cell):
        """Initialize the weights"""
        factor = self.config.initializer_range
        if isinstance(cell, (nn.Conv2d, nn.Linear, nn.Embedding)):
            cell.weight.set_data(initializer(Normal(factor), cell.weight.shape, cell.weight.dtype))
            if hasattr(cell, "bias") and cell.bias is not None:
                cell.bias.set_data(initializer('zeros', cell.bias.shape, cell.bias.dtype))

        if isinstance(cell, Blip2VisionEmbeddings):
            if hasattr(self.config, "vision_config"):
                factor = self.config.vision_config.initializer_range

            cell.position_embedding.set_data(initializer(TruncatedNormal(factor), cell.position_embedding.shape, cell.position_embedding.dtype))
            cell.class_embedding.set_data(initializer(TruncatedNormal(factor), cell.class_embedding.shape, cell.class_embedding.dtype))

        elif isinstance(cell, nn.LayerNorm):
            cell.weight.set_data(initializer('ones', cell.weight.shape, cell.weight.dtype))
            cell.bias.set_data(initializer('zeros', cell.bias.shape, cell.bias.dtype))

        elif isinstance(cell, nn.Linear) and cell.bias is not None:
            cell.bias.set_data(initializer('zeros', cell.bias.shape, cell.bias.dtype))

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2ForConditionalGeneration

Bases: Blip2PreTrainedModel

The Blip2ForConditionalGeneration class is a model for image captioning and visual question answering. It is a conditional generator that takes as input an image and generates captions or answers to questions based on the image.

The class inherits from the Blip2PreTrainedModel class.

Example

>>> from transformers import Blip2ForConditionalGeneration
...
>>>model = Blip2ForConditionalGeneration()

METHOD	DESCRIPTION
`__init__`	Initializes the Blip2ForConditionalGeneration model with the given configuration.
`get_input_embeddings`	Returns the input embeddings of the language model.
`set_input_embeddings`	Sets the input embeddings of the language model to the given value.
`set_output_embeddings`	Sets the output embeddings of the language model to the given new embeddings.
`get_output_embeddings`	Returns the output embeddings of the language model.
`get_encoder`	Returns the encoder of the language model.
`get_decoder`	Returns the decoder of the language model.
`_tie_weights`	Ties the weights of the encoder and decoder if the model is not using a decoder-only language model.
`forward`	Constructs the Blip2ForConditionalGeneration model with the given inputs and returns the output.
`generate`	Generates captions or answers based on the given image and optionally the input sequence and attention mask.

Please refer to the docstrings of each method for more detailed information.

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

class Blip2ForConditionalGeneration(Blip2PreTrainedModel):

    """
    The `Blip2ForConditionalGeneration` class is a model for image captioning and visual question answering. It is a conditional generator that takes as input an image and generates captions or answers to
    questions based on the image.

    The class inherits from the `Blip2PreTrainedModel` class.

    Example:
        ```python
        >>> from transformers import Blip2ForConditionalGeneration
        ...
        >>>model = Blip2ForConditionalGeneration()
        ```

    Methods:
        `__init__`: Initializes the Blip2ForConditionalGeneration model with the given configuration.
        `get_input_embeddings`: Returns the input embeddings of the language model.
        `set_input_embeddings`: Sets the input embeddings of the language model to the given value.
        `set_output_embeddings`: Sets the output embeddings of the language model to the given new embeddings.
        `get_output_embeddings`: Returns the output embeddings of the language model.
        `get_encoder`: Returns the encoder of the language model.
        `get_decoder`: Returns the decoder of the language model.
        `_tie_weights`: Ties the weights of the encoder and decoder if the model is not using a decoder-only language model.
        `forward`: Constructs the Blip2ForConditionalGeneration model with the given inputs and returns the output.
        `generate`: Generates captions or answers based on the given image and optionally the input sequence and attention mask.

    Please refer to the docstrings of each method for more detailed information.
    """
    config_class = Blip2Config
    main_input_name = "pixel_values"

    def __init__(self, config: Blip2Config):
        """
        This method initializes an instance of the Blip2ForConditionalGeneration class.

        Args:
            self: The instance of the Blip2ForConditionalGeneration class.
            config (Blip2Config): An object containing the configuration settings for the Blip2 model.
                It is used to initialize the various components of the model, such as vision model, query tokens, qformer,
                language projection, and language model. The config parameter is required and must be of type Blip2Config.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(config)

        self.vision_model = Blip2VisionModel(config.vision_config)

        self.query_tokens = Parameter(ops.zeros(1, config.num_query_tokens, config.qformer_config.hidden_size))
        self.qformer = Blip2QFormerModel(config.qformer_config)

        self.language_projection = nn.Linear(config.qformer_config.hidden_size, config.text_config.hidden_size)
        if config.use_decoder_only_language_model:
            language_model = AutoModelForCausalLM.from_config(config.text_config)
        else:
            language_model = AutoModelForSeq2SeqLM.from_config(config.text_config)

        # Update _tied_weights_keys using the base model used.
        if language_model._tied_weights_keys is not None:
            self._tied_weights_keys = [f"language_model.{k}" for k in language_model._tied_weights_keys]

        self.language_model = language_model

        # Initialize weights and apply final processing
        self.post_init()

    def get_input_embeddings(self):
        """
        Retrieves the input embeddings from the Blip2 language model.

        Args:
            self: An instance of the Blip2ForConditionalGeneration class.

        Returns:
            None.

        Raises:
            None.

        This method retrieves the input embeddings from the underlying language model used in Blip2ForConditionalGeneration.
        It returns None since it directly calls the 'get_input_embeddings' method of the language model and does not modify
        or process the embeddings further.

        Please note that this method takes only one parameter, which is the instance of the Blip2ForConditionalGeneration class itself (self).
        """
        return self.language_model.get_input_embeddings()

    def set_input_embeddings(self, value):
        """
        Set the input embeddings for the Blip2ForConditionalGeneration model.

        Args:
            self (Blip2ForConditionalGeneration): The instance of the Blip2ForConditionalGeneration class.
            value (torch.Tensor): The input embeddings to be set for the language model.

        Returns:
            None.

        Raises:
            None.

        This method sets the input embeddings for the underlying language model of Blip2ForConditionalGeneration.
        The input embeddings are used to represent the input tokens during the model's forward pass.
        The 'value' parameter should be a tensor of shape (vocab_size, embedding_dim), where 'vocab_size'
        represents the size of the vocabulary and 'embedding_dim' represents the dimensionality of the embedding space.
        The method updates the input embeddings of the language model with the provided 'value'.

        Example:
            ```python
            >>> model = Blip2ForConditionalGeneration()
            >>> embeddings = torch.randn(10000, 300)
            >>> model.set_input_embeddings(embeddings)
            ```
        """
        self.language_model.set_input_embeddings(value)

    def set_output_embeddings(self, new_embeddings):
        """
        Sets the output embeddings for the Blip2ForConditionalGeneration class.

        Args:
            self (Blip2ForConditionalGeneration): The instance of the Blip2ForConditionalGeneration class.
            new_embeddings: The new embeddings to be set for the language model.

        Returns:
            None.

        Raises:
            None.
        """
        self.language_model.set_output_embeddings(new_embeddings)

    def get_output_embeddings(self) -> nn.Module:
        """
        Get the output embeddings of the Blip2ForConditionalGeneration class.

        Args:
            self (Blip2ForConditionalGeneration): The instance of the Blip2ForConditionalGeneration class.

        Returns:
            nn.Module: The output embeddings of the language model.

        Raises:
            None.

        This method returns the output embeddings of the language model used in the Blip2ForConditionalGeneration class.
        The output embeddings are retrieved by calling the 'get_output_embeddings()' method of the language model.

        Note:
            The output embeddings are typically used to map the hidden states of the language model to the vocabulary
            space. They can be further processed or used in downstream tasks.
        """
        return self.language_model.get_output_embeddings()

    def get_encoder(self):
        """
        Method to retrieve the encoder from the language model for Blip2ForConditionalGeneration.

        Args:
            self:
                The instance of the Blip2ForConditionalGeneration class.

                - Type: Blip2ForConditionalGeneration
                - Purpose: To access the methods and attributes of the Blip2ForConditionalGeneration class.

        Returns:
            None:
                The method returns None as it retrieves the encoder from the language model
                and does not return any specific value.

                - Type: None
                - Purpose: Indicate that the method successfully retrieved the encoder from the language model.

        Raises:
            No specific exceptions are raised by this method.
        """
        return self.language_model.get_encoder()

    def get_decoder(self):
        """
        This method returns the decoder from the language model associated with the Blip2ForConditionalGeneration instance.

        Args:
            self (Blip2ForConditionalGeneration): The instance of the Blip2ForConditionalGeneration class.
                It is used to access the language model and retrieve the decoder.

        Returns:
            None:
                This method does not return any value directly.
                It retrieves the decoder from the language model associated with Blip2ForConditionalGeneration.

        Raises:
            None
        """
        return self.language_model.get_decoder()

    def _tie_weights(self):
        """
        Ties the weights between encoder and decoder in the Blip2ForConditionalGeneration model.

        Args:
            self: The instance of the Blip2ForConditionalGeneration class.

        Returns:
            None.

        Raises:
            None.
        """
        if not self.config.use_decoder_only_language_model:
            self.language_model.encoder.embed_tokens = self.language_model.shared
            self.language_model.decoder.embed_tokens = self.language_model.shared

    def forward(
        self,
        pixel_values: mindspore.Tensor,
        input_ids: mindspore.Tensor,
        attention_mask: Optional[mindspore.Tensor] = None,
        decoder_input_ids: Optional[mindspore.Tensor] = None,
        decoder_attention_mask: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        labels: Optional[mindspore.Tensor] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, Blip2ForConditionalGenerationModelOutput]:
        r"""

        Returns:
            Union[Tuple, Blip2ForConditionalGenerationModelOutput]

        Example:
            Prepare processor, model and image input

            ```python
            >>> from PIL import Image
            >>> import requests
            >>> from transformers import Blip2Processor, Blip2ForConditionalGeneration
            >>> import torch
            ...
            ...
            >>> processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
            >>> model = Blip2ForConditionalGeneration.from_pretrained(
            ...     "Salesforce/blip2-opt-2.7b", load_in_8bit=True, torch_dtype=torch.float16
            ... )  # doctest: +IGNORE_RESULT
            ...
            >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
            >>> image = Image.open(requests.get(url, stream=True).raw)
            ```

            Image captioning (without providing a text prompt):

            ```python
            >>> inputs = processor(images=image, return_tensors="pt").to(torch.float16)
            ...
            >>> generated_ids = model.generate(**inputs)
            >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
            >>> print(generated_text)
            two cats laying on a couch
            ```

            Visual question answering (prompt = question):

            ```python
            >>> prompt = "Question: how many cats are there? Answer:"
            >>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(dtype=torch.float16)
            ...
            >>> generated_ids = model.generate(**inputs)
            >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
            >>> print(generated_text)
            two
            ```

            Note that int8 inference is also supported through [bitsandbytes](https://github.com/TimDettmers/bitsandbytes).
            This greatly reduces the amount of memory used by the model while maintaining the same performance.

            ```python
            >>> model = Blip2ForConditionalGeneration.from_pretrained(
            ...     "Salesforce/blip2-opt-2.7b", load_in_8bit=True, torch_dtype=torch.bfloat16
            ... )  # doctest: +IGNORE_RESULT
            ...
            >>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(dtype=torch.bfloat16)
            ...
            >>> generated_ids = model.generate(**inputs)
            >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
            >>> print(generated_text)
            two
            ```
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        # step 1: forward the images through the vision encoder,
        # to get image embeddings of shape (batch_size, seq_len, hidden_size)
        vision_outputs = self.vision_model(
            pixel_values=pixel_values,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )
        image_embeds = vision_outputs[0]
        # step 2: forward the query tokens through the QFormer, using the image embeddings for cross-attention
        image_attention_mask = ops.ones(*image_embeds.shape[:-1], dtype=mindspore.int64)

        query_tokens = ops.broadcast_to(self.query_tokens, (image_embeds.shape[0], -1, -1))
        query_outputs = self.qformer(
            query_embeds=query_tokens,
            encoder_hidden_states=image_embeds,
            encoder_attention_mask=image_attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )
        query_output = query_outputs[0]

        # step 3: use the language model, conditioned on the query outputs and the prompt
        language_model_inputs = self.language_projection(query_output)
        language_model_attention_mask = ops.ones(
            *language_model_inputs.shape[:-1], dtype=mindspore.int64
        )
        inputs_embeds = self.language_model.get_input_embeddings()(input_ids)
        inputs_embeds = ops.cat([language_model_inputs, inputs_embeds], dim=1)

        if attention_mask is None:
            attention_mask = ops.ones_like(input_ids)
        attention_mask = ops.cat([language_model_attention_mask.astype(mindspore.bool_), attention_mask.astype(mindspore.bool_)], dim=1)

        if self.config.use_decoder_only_language_model:
            outputs = self.language_model(
                inputs_embeds=inputs_embeds,
                attention_mask=attention_mask,
                output_attentions=output_attentions,
                output_hidden_states=output_hidden_states,
                return_dict=return_dict,
            )
            logits = outputs.logits if return_dict else outputs[0]
            loss = None
            # we compute the loss here since we need to take into account the sequence length of the query embeds
            if labels is not None:
                logits = logits[:, -labels.shape[1] :, :]
                # Shift so that tokens < n predict n
                shift_logits = logits[..., :-1, :]
                shift_labels = labels[..., 1:]

                # Flatten the tokens
                loss = F.cross_entropy(shift_logits.view(-1, self.config.text_config.vocab_size), shift_labels.view(-1))
        else:
            outputs = self.language_model(
                inputs_embeds=inputs_embeds,
                attention_mask=attention_mask,
                decoder_input_ids=decoder_input_ids,
                decoder_attention_mask=decoder_attention_mask,
                output_attentions=output_attentions,
                output_hidden_states=output_hidden_states,
                return_dict=return_dict,
                labels=labels,
            )
            loss = outputs.loss if return_dict else outputs[0]
            logits = outputs.logits if return_dict else outputs[1]

        if not return_dict:
            output = (logits, vision_outputs, query_outputs, outputs)
            return ((loss,) + output) if loss is not None else output

        return Blip2ForConditionalGenerationModelOutput(
            loss=loss,
            logits=logits,
            vision_outputs=vision_outputs,
            qformer_outputs=query_outputs,
            language_model_outputs=outputs,
        )

    def generate(
        self,
        pixel_values: mindspore.Tensor,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        **generate_kwargs,
    ) -> mindspore.Tensor:
        """
        Overrides `generate` function to be able to use the model as a conditional generator.

        Args:
            pixel_values (`mindspore.Tensor` of shape (batch_size, num_channels, height, width)):
                Input images to be processed.
            input_ids (`mindspore.Tensor` of shape (batch_size, sequence_length), *optional*):
                The sequence used as a prompt for the generation.
            attention_mask (`mindspore.Tensor` of shape (batch_size, sequence_length), *optional*):
                Mask to avoid performing attention on padding token indices

        Returns:
            captions (list): A list of strings of length batch_size * num_captions.
        """
        batch_size = pixel_values.shape[0]
        image_embeds = self.vision_model(pixel_values, return_dict=True).last_hidden_state

        image_attention_mask = ops.ones(*image_embeds.shape[:-1], dtype=mindspore.int64)

        query_tokens = ops.broadcast_to(self.query_tokens, (image_embeds.shape[0], -1, -1))
        query_outputs = self.qformer(
            query_embeds=query_tokens,
            encoder_hidden_states=image_embeds,
            encoder_attention_mask=image_attention_mask,
            return_dict=True,
        )
        query_output = query_outputs.last_hidden_state

        language_model_inputs = self.language_projection(query_output)
        language_attention_mask = ops.ones(
            *language_model_inputs.shape[:-1], dtype=mindspore.int64
        )
        if input_ids is None:
            input_ids = (
                mindspore.Tensor([[self.config.text_config.bos_token_id]])
                .repeat(batch_size, 1)
            )
        if attention_mask is None:
            attention_mask = ops.ones_like(input_ids)
        attention_mask = ops.cat([language_attention_mask, attention_mask], dim=1)

        # concatenate query embeddings with prompt embeddings
        inputs_embeds = self.get_input_embeddings()(input_ids)
        inputs_embeds = ops.cat([language_model_inputs, inputs_embeds], dim=1)

        # add image_embeds length to max_length, so that the final max_length in counted only on token embeds
        # -1 is to account for the prepended BOS after `generate.`
        # TODO (joao, raushan): refactor `generate` to avoid these operations with VLMs
        if not self.language_model.config.is_encoder_decoder:
            generate_kwargs["max_length"] = generate_kwargs.get("max_length", 20) + language_model_inputs.shape[1] - 1
            generate_kwargs["min_length"] = generate_kwargs.get("min_length", 0) + language_model_inputs.shape[1]

        outputs = self.language_model.generate(
            inputs_embeds=inputs_embeds,
            attention_mask=attention_mask,
            **generate_kwargs,
        )

        return outputs

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2ForConditionalGeneration.__init__(config)

This method initializes an instance of the Blip2ForConditionalGeneration class.

PARAMETER	DESCRIPTION
self	The instance of the Blip2ForConditionalGeneration class.
config	An object containing the configuration settings for the Blip2 model. It is used to initialize the various components of the model, such as vision model, query tokens, qformer, language projection, and language model. The config parameter is required and must be of type Blip2Config. TYPE: Blip2Config

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def __init__(self, config: Blip2Config):
    """
    This method initializes an instance of the Blip2ForConditionalGeneration class.

    Args:
        self: The instance of the Blip2ForConditionalGeneration class.
        config (Blip2Config): An object containing the configuration settings for the Blip2 model.
            It is used to initialize the various components of the model, such as vision model, query tokens, qformer,
            language projection, and language model. The config parameter is required and must be of type Blip2Config.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(config)

    self.vision_model = Blip2VisionModel(config.vision_config)

    self.query_tokens = Parameter(ops.zeros(1, config.num_query_tokens, config.qformer_config.hidden_size))
    self.qformer = Blip2QFormerModel(config.qformer_config)

    self.language_projection = nn.Linear(config.qformer_config.hidden_size, config.text_config.hidden_size)
    if config.use_decoder_only_language_model:
        language_model = AutoModelForCausalLM.from_config(config.text_config)
    else:
        language_model = AutoModelForSeq2SeqLM.from_config(config.text_config)

    # Update _tied_weights_keys using the base model used.
    if language_model._tied_weights_keys is not None:
        self._tied_weights_keys = [f"language_model.{k}" for k in language_model._tied_weights_keys]

    self.language_model = language_model

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2ForConditionalGeneration.forward(pixel_values, input_ids, attention_mask=None, decoder_input_ids=None, decoder_attention_mask=None, output_attentions=None, output_hidden_states=None, labels=None, return_dict=None)

RETURNS	DESCRIPTION
Union[Tuple, Blip2ForConditionalGenerationModelOutput]	Union[Tuple, Blip2ForConditionalGenerationModelOutput]

Example

Prepare processor, model and image input

>>> from PIL import Image
>>> import requests
>>> from transformers import Blip2Processor, Blip2ForConditionalGeneration
>>> import torch
...
...
>>> processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
>>> model = Blip2ForConditionalGeneration.from_pretrained(
...     "Salesforce/blip2-opt-2.7b", load_in_8bit=True, torch_dtype=torch.float16
... )  # doctest: +IGNORE_RESULT
...
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)

Image captioning (without providing a text prompt):

>>> inputs = processor(images=image, return_tensors="pt").to(torch.float16)
...
>>> generated_ids = model.generate(**inputs)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
>>> print(generated_text)
two cats laying on a couch

Visual question answering (prompt = question):

>>> prompt = "Question: how many cats are there? Answer:"
>>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(dtype=torch.float16)
...
>>> generated_ids = model.generate(**inputs)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
>>> print(generated_text)
two

Note that int8 inference is also supported through bitsandbytes. This greatly reduces the amount of memory used by the model while maintaining the same performance.

>>> model = Blip2ForConditionalGeneration.from_pretrained(
...     "Salesforce/blip2-opt-2.7b", load_in_8bit=True, torch_dtype=torch.bfloat16
... )  # doctest: +IGNORE_RESULT
...
>>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(dtype=torch.bfloat16)
...
>>> generated_ids = model.generate(**inputs)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
>>> print(generated_text)
two

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def forward(
    self,
    pixel_values: mindspore.Tensor,
    input_ids: mindspore.Tensor,
    attention_mask: Optional[mindspore.Tensor] = None,
    decoder_input_ids: Optional[mindspore.Tensor] = None,
    decoder_attention_mask: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    labels: Optional[mindspore.Tensor] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple, Blip2ForConditionalGenerationModelOutput]:
    r"""

    Returns:
        Union[Tuple, Blip2ForConditionalGenerationModelOutput]

    Example:
        Prepare processor, model and image input

        ```python
        >>> from PIL import Image
        >>> import requests
        >>> from transformers import Blip2Processor, Blip2ForConditionalGeneration
        >>> import torch
        ...
        ...
        >>> processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
        >>> model = Blip2ForConditionalGeneration.from_pretrained(
        ...     "Salesforce/blip2-opt-2.7b", load_in_8bit=True, torch_dtype=torch.float16
        ... )  # doctest: +IGNORE_RESULT
        ...
        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
        >>> image = Image.open(requests.get(url, stream=True).raw)
        ```

        Image captioning (without providing a text prompt):

        ```python
        >>> inputs = processor(images=image, return_tensors="pt").to(torch.float16)
        ...
        >>> generated_ids = model.generate(**inputs)
        >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
        >>> print(generated_text)
        two cats laying on a couch
        ```

        Visual question answering (prompt = question):

        ```python
        >>> prompt = "Question: how many cats are there? Answer:"
        >>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(dtype=torch.float16)
        ...
        >>> generated_ids = model.generate(**inputs)
        >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
        >>> print(generated_text)
        two
        ```

        Note that int8 inference is also supported through [bitsandbytes](https://github.com/TimDettmers/bitsandbytes).
        This greatly reduces the amount of memory used by the model while maintaining the same performance.

        ```python
        >>> model = Blip2ForConditionalGeneration.from_pretrained(
        ...     "Salesforce/blip2-opt-2.7b", load_in_8bit=True, torch_dtype=torch.bfloat16
        ... )  # doctest: +IGNORE_RESULT
        ...
        >>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(dtype=torch.bfloat16)
        ...
        >>> generated_ids = model.generate(**inputs)
        >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
        >>> print(generated_text)
        two
        ```
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    # step 1: forward the images through the vision encoder,
    # to get image embeddings of shape (batch_size, seq_len, hidden_size)
    vision_outputs = self.vision_model(
        pixel_values=pixel_values,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )
    image_embeds = vision_outputs[0]
    # step 2: forward the query tokens through the QFormer, using the image embeddings for cross-attention
    image_attention_mask = ops.ones(*image_embeds.shape[:-1], dtype=mindspore.int64)

    query_tokens = ops.broadcast_to(self.query_tokens, (image_embeds.shape[0], -1, -1))
    query_outputs = self.qformer(
        query_embeds=query_tokens,
        encoder_hidden_states=image_embeds,
        encoder_attention_mask=image_attention_mask,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )
    query_output = query_outputs[0]

    # step 3: use the language model, conditioned on the query outputs and the prompt
    language_model_inputs = self.language_projection(query_output)
    language_model_attention_mask = ops.ones(
        *language_model_inputs.shape[:-1], dtype=mindspore.int64
    )
    inputs_embeds = self.language_model.get_input_embeddings()(input_ids)
    inputs_embeds = ops.cat([language_model_inputs, inputs_embeds], dim=1)

    if attention_mask is None:
        attention_mask = ops.ones_like(input_ids)
    attention_mask = ops.cat([language_model_attention_mask.astype(mindspore.bool_), attention_mask.astype(mindspore.bool_)], dim=1)

    if self.config.use_decoder_only_language_model:
        outputs = self.language_model(
            inputs_embeds=inputs_embeds,
            attention_mask=attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )
        logits = outputs.logits if return_dict else outputs[0]
        loss = None
        # we compute the loss here since we need to take into account the sequence length of the query embeds
        if labels is not None:
            logits = logits[:, -labels.shape[1] :, :]
            # Shift so that tokens < n predict n
            shift_logits = logits[..., :-1, :]
            shift_labels = labels[..., 1:]

            # Flatten the tokens
            loss = F.cross_entropy(shift_logits.view(-1, self.config.text_config.vocab_size), shift_labels.view(-1))
    else:
        outputs = self.language_model(
            inputs_embeds=inputs_embeds,
            attention_mask=attention_mask,
            decoder_input_ids=decoder_input_ids,
            decoder_attention_mask=decoder_attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
            labels=labels,
        )
        loss = outputs.loss if return_dict else outputs[0]
        logits = outputs.logits if return_dict else outputs[1]

    if not return_dict:
        output = (logits, vision_outputs, query_outputs, outputs)
        return ((loss,) + output) if loss is not None else output

    return Blip2ForConditionalGenerationModelOutput(
        loss=loss,
        logits=logits,
        vision_outputs=vision_outputs,
        qformer_outputs=query_outputs,
        language_model_outputs=outputs,
    )

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2ForConditionalGeneration.generate(pixel_values, input_ids=None, attention_mask=None, **generate_kwargs)

Overrides generate function to be able to use the model as a conditional generator.

PARAMETER	DESCRIPTION
pixel_values	Input images to be processed. TYPE: `mindspore.Tensor` of shape (batch_size, num_channels, height, width
input_ids	The sequence used as a prompt for the generation. TYPE: `mindspore.Tensor` of shape (batch_size, sequence_length), *optional* DEFAULT: None
attention_mask	Mask to avoid performing attention on padding token indices TYPE: `mindspore.Tensor` of shape (batch_size, sequence_length), *optional* DEFAULT: None

RETURNS	DESCRIPTION
captions	A list of strings of length batch_size * num_captions. TYPE: list

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def generate(
    self,
    pixel_values: mindspore.Tensor,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    **generate_kwargs,
) -> mindspore.Tensor:
    """
    Overrides `generate` function to be able to use the model as a conditional generator.

    Args:
        pixel_values (`mindspore.Tensor` of shape (batch_size, num_channels, height, width)):
            Input images to be processed.
        input_ids (`mindspore.Tensor` of shape (batch_size, sequence_length), *optional*):
            The sequence used as a prompt for the generation.
        attention_mask (`mindspore.Tensor` of shape (batch_size, sequence_length), *optional*):
            Mask to avoid performing attention on padding token indices

    Returns:
        captions (list): A list of strings of length batch_size * num_captions.
    """
    batch_size = pixel_values.shape[0]
    image_embeds = self.vision_model(pixel_values, return_dict=True).last_hidden_state

    image_attention_mask = ops.ones(*image_embeds.shape[:-1], dtype=mindspore.int64)

    query_tokens = ops.broadcast_to(self.query_tokens, (image_embeds.shape[0], -1, -1))
    query_outputs = self.qformer(
        query_embeds=query_tokens,
        encoder_hidden_states=image_embeds,
        encoder_attention_mask=image_attention_mask,
        return_dict=True,
    )
    query_output = query_outputs.last_hidden_state

    language_model_inputs = self.language_projection(query_output)
    language_attention_mask = ops.ones(
        *language_model_inputs.shape[:-1], dtype=mindspore.int64
    )
    if input_ids is None:
        input_ids = (
            mindspore.Tensor([[self.config.text_config.bos_token_id]])
            .repeat(batch_size, 1)
        )
    if attention_mask is None:
        attention_mask = ops.ones_like(input_ids)
    attention_mask = ops.cat([language_attention_mask, attention_mask], dim=1)

    # concatenate query embeddings with prompt embeddings
    inputs_embeds = self.get_input_embeddings()(input_ids)
    inputs_embeds = ops.cat([language_model_inputs, inputs_embeds], dim=1)

    # add image_embeds length to max_length, so that the final max_length in counted only on token embeds
    # -1 is to account for the prepended BOS after `generate.`
    # TODO (joao, raushan): refactor `generate` to avoid these operations with VLMs
    if not self.language_model.config.is_encoder_decoder:
        generate_kwargs["max_length"] = generate_kwargs.get("max_length", 20) + language_model_inputs.shape[1] - 1
        generate_kwargs["min_length"] = generate_kwargs.get("min_length", 0) + language_model_inputs.shape[1]

    outputs = self.language_model.generate(
        inputs_embeds=inputs_embeds,
        attention_mask=attention_mask,
        **generate_kwargs,
    )

    return outputs

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2ForConditionalGeneration.get_decoder()

This method returns the decoder from the language model associated with the Blip2ForConditionalGeneration instance.

PARAMETER	DESCRIPTION
self	The instance of the Blip2ForConditionalGeneration class. It is used to access the language model and retrieve the decoder. TYPE: Blip2ForConditionalGeneration

RETURNS	DESCRIPTION
None	This method does not return any value directly. It retrieves the decoder from the language model associated with Blip2ForConditionalGeneration.

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def get_decoder(self):
    """
    This method returns the decoder from the language model associated with the Blip2ForConditionalGeneration instance.

    Args:
        self (Blip2ForConditionalGeneration): The instance of the Blip2ForConditionalGeneration class.
            It is used to access the language model and retrieve the decoder.

    Returns:
        None:
            This method does not return any value directly.
            It retrieves the decoder from the language model associated with Blip2ForConditionalGeneration.

    Raises:
        None
    """
    return self.language_model.get_decoder()

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2ForConditionalGeneration.get_encoder()

Method to retrieve the encoder from the language model for Blip2ForConditionalGeneration.

PARAMETER	DESCRIPTION
self	The instance of the Blip2ForConditionalGeneration class. Type: Blip2ForConditionalGeneration Purpose: To access the methods and attributes of the Blip2ForConditionalGeneration class.

RETURNS	DESCRIPTION
None	The method returns None as it retrieves the encoder from the language model and does not return any specific value. Type: None Purpose: Indicate that the method successfully retrieved the encoder from the language model.

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def get_encoder(self):
    """
    Method to retrieve the encoder from the language model for Blip2ForConditionalGeneration.

    Args:
        self:
            The instance of the Blip2ForConditionalGeneration class.

            - Type: Blip2ForConditionalGeneration
            - Purpose: To access the methods and attributes of the Blip2ForConditionalGeneration class.

    Returns:
        None:
            The method returns None as it retrieves the encoder from the language model
            and does not return any specific value.

            - Type: None
            - Purpose: Indicate that the method successfully retrieved the encoder from the language model.

    Raises:
        No specific exceptions are raised by this method.
    """
    return self.language_model.get_encoder()

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2ForConditionalGeneration.get_input_embeddings()

Retrieves the input embeddings from the Blip2 language model.

PARAMETER	DESCRIPTION
self	An instance of the Blip2ForConditionalGeneration class.

RETURNS	DESCRIPTION
	None.

This method retrieves the input embeddings from the underlying language model used in Blip2ForConditionalGeneration. It returns None since it directly calls the 'get_input_embeddings' method of the language model and does not modify or process the embeddings further.

Please note that this method takes only one parameter, which is the instance of the Blip2ForConditionalGeneration class itself (self).

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def get_input_embeddings(self):
    """
    Retrieves the input embeddings from the Blip2 language model.

    Args:
        self: An instance of the Blip2ForConditionalGeneration class.

    Returns:
        None.

    Raises:
        None.

    This method retrieves the input embeddings from the underlying language model used in Blip2ForConditionalGeneration.
    It returns None since it directly calls the 'get_input_embeddings' method of the language model and does not modify
    or process the embeddings further.

    Please note that this method takes only one parameter, which is the instance of the Blip2ForConditionalGeneration class itself (self).
    """
    return self.language_model.get_input_embeddings()

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2ForConditionalGeneration.get_output_embeddings()

Get the output embeddings of the Blip2ForConditionalGeneration class.

PARAMETER	DESCRIPTION
self	The instance of the Blip2ForConditionalGeneration class. TYPE: Blip2ForConditionalGeneration

RETURNS	DESCRIPTION
Module	nn.Module: The output embeddings of the language model.

This method returns the output embeddings of the language model used in the Blip2ForConditionalGeneration class. The output embeddings are retrieved by calling the 'get_output_embeddings()' method of the language model.

Note

The output embeddings are typically used to map the hidden states of the language model to the vocabulary space. They can be further processed or used in downstream tasks.

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def get_output_embeddings(self) -> nn.Module:
    """
    Get the output embeddings of the Blip2ForConditionalGeneration class.

    Args:
        self (Blip2ForConditionalGeneration): The instance of the Blip2ForConditionalGeneration class.

    Returns:
        nn.Module: The output embeddings of the language model.

    Raises:
        None.

    This method returns the output embeddings of the language model used in the Blip2ForConditionalGeneration class.
    The output embeddings are retrieved by calling the 'get_output_embeddings()' method of the language model.

    Note:
        The output embeddings are typically used to map the hidden states of the language model to the vocabulary
        space. They can be further processed or used in downstream tasks.
    """
    return self.language_model.get_output_embeddings()

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2ForConditionalGeneration.set_input_embeddings(value)

Set the input embeddings for the Blip2ForConditionalGeneration model.

PARAMETER	DESCRIPTION
self	The instance of the Blip2ForConditionalGeneration class. TYPE: Blip2ForConditionalGeneration
value	The input embeddings to be set for the language model. TYPE: Tensor

RETURNS	DESCRIPTION
	None.

This method sets the input embeddings for the underlying language model of Blip2ForConditionalGeneration. The input embeddings are used to represent the input tokens during the model's forward pass. The 'value' parameter should be a tensor of shape (vocab_size, embedding_dim), where 'vocab_size' represents the size of the vocabulary and 'embedding_dim' represents the dimensionality of the embedding space. The method updates the input embeddings of the language model with the provided 'value'.

Example

>>> model = Blip2ForConditionalGeneration()
>>> embeddings = torch.randn(10000, 300)
>>> model.set_input_embeddings(embeddings)

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def set_input_embeddings(self, value):
    """
    Set the input embeddings for the Blip2ForConditionalGeneration model.

    Args:
        self (Blip2ForConditionalGeneration): The instance of the Blip2ForConditionalGeneration class.
        value (torch.Tensor): The input embeddings to be set for the language model.

    Returns:
        None.

    Raises:
        None.

    This method sets the input embeddings for the underlying language model of Blip2ForConditionalGeneration.
    The input embeddings are used to represent the input tokens during the model's forward pass.
    The 'value' parameter should be a tensor of shape (vocab_size, embedding_dim), where 'vocab_size'
    represents the size of the vocabulary and 'embedding_dim' represents the dimensionality of the embedding space.
    The method updates the input embeddings of the language model with the provided 'value'.

    Example:
        ```python
        >>> model = Blip2ForConditionalGeneration()
        >>> embeddings = torch.randn(10000, 300)
        >>> model.set_input_embeddings(embeddings)
        ```
    """
    self.language_model.set_input_embeddings(value)

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2ForConditionalGeneration.set_output_embeddings(new_embeddings)

Sets the output embeddings for the Blip2ForConditionalGeneration class.

PARAMETER	DESCRIPTION
self	The instance of the Blip2ForConditionalGeneration class. TYPE: Blip2ForConditionalGeneration
new_embeddings	The new embeddings to be set for the language model.

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def set_output_embeddings(self, new_embeddings):
    """
    Sets the output embeddings for the Blip2ForConditionalGeneration class.

    Args:
        self (Blip2ForConditionalGeneration): The instance of the Blip2ForConditionalGeneration class.
        new_embeddings: The new embeddings to be set for the language model.

    Returns:
        None.

    Raises:
        None.
    """
    self.language_model.set_output_embeddings(new_embeddings)

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2VisionModel

Bases: Blip2PreTrainedModel

This class represents a Blip2VisionModel which is designed for vision tasks using Blip2 models. The Blip2VisionModel class inherits from the Blip2PreTrainedModel class and provides functionality for processing pixel values, forwarding embeddings, and generating output for vision-related tasks.

ATTRIBUTE	DESCRIPTION
config	An instance of Blip2VisionConfig containing configuration settings for the model.
embeddings	An instance of Blip2VisionEmbeddings for creating embeddings from pixel values.
encoder	An instance of Blip2Encoder for encoding input embeddings.
post_layernorm	A LayerNorm module for applying layer normalization to the output.

METHOD	DESCRIPTION
__init__	Initializes the Blip2VisionModel with the provided configuration.
forward	Constructs the model by processing pixel values, generating embeddings, and producing output for vision tasks.
get_input_embeddings	Retrieves the embeddings module used by the model for processing input pixel values.

The Blip2VisionModel class provides a comprehensive solution for vision tasks by leveraging the Blip2 architecture and incorporating advanced features such as layer normalization and configurable output options.

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

class Blip2VisionModel(Blip2PreTrainedModel):

    """
    This class represents a Blip2VisionModel which is designed for vision tasks using Blip2 models.
    The Blip2VisionModel class inherits from the Blip2PreTrainedModel class and provides functionality for processing pixel values,
    forwarding embeddings, and generating output for vision-related tasks.

    Attributes:
        config: An instance of Blip2VisionConfig containing configuration settings for the model.
        embeddings: An instance of Blip2VisionEmbeddings for creating embeddings from pixel values.
        encoder: An instance of Blip2Encoder for encoding input embeddings.
        post_layernorm: A LayerNorm module for applying layer normalization to the output.

    Methods:
        __init__: Initializes the Blip2VisionModel with the provided configuration.
        forward:
            Constructs the model by processing pixel values, generating embeddings, and producing output for vision tasks.
        get_input_embeddings: Retrieves the embeddings module used by the model for processing input pixel values.

    The Blip2VisionModel class provides a comprehensive solution for vision tasks by leveraging the Blip2 architecture
    and incorporating advanced features such as layer normalization and configurable output options.
    """
    main_input_name = "pixel_values"
    config_class = Blip2VisionConfig

    def __init__(self, config: Blip2VisionConfig):
        """
        Initializes a new instance of the Blip2VisionModel class.

        Args:
            self: The object instance.
            config (Blip2VisionConfig): The configuration object containing various settings for the model.

        Returns:
            None

        Raises:
            None
        """
        super().__init__(config)
        self.config = config
        embed_dim = config.hidden_size

        self.embeddings = Blip2VisionEmbeddings(config)
        self.encoder = Blip2Encoder(config)
        self.post_layernorm = nn.LayerNorm(embed_dim, eps=config.layer_norm_eps)

        self.post_init()

    def forward(
        self,
        pixel_values: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, BaseModelOutputWithPooling]:
        r"""

        Returns:
            `Union[Tuple, BaseModelOutputWithPooling]`
        """
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        if pixel_values is None:
            raise ValueError("You have to specify pixel_values")

        hidden_states = self.embeddings(pixel_values)

        encoder_outputs = self.encoder(
            inputs_embeds=hidden_states,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        last_hidden_state = encoder_outputs[0]
        last_hidden_state = self.post_layernorm(last_hidden_state)

        pooled_output = last_hidden_state[:, 0, :]
        pooled_output = self.post_layernorm(pooled_output)

        if not return_dict:
            return (last_hidden_state, pooled_output) + encoder_outputs[1:]

        return BaseModelOutputWithPooling(
            last_hidden_state=last_hidden_state,
            pooler_output=pooled_output,
            hidden_states=encoder_outputs.hidden_states,
            attentions=encoder_outputs.attentions,
        )

    def get_input_embeddings(self):
        """
        This method retrieves the input embeddings from the Blip2VisionModel.

        Args:
            self (Blip2VisionModel): The instance of the Blip2VisionModel class.

        Returns:
            embeddings: This method returns the input embeddings.

        Raises:
            None.
        """
        return self.embeddings

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2VisionModel.__init__(config)

Initializes a new instance of the Blip2VisionModel class.

PARAMETER	DESCRIPTION
self	The object instance.
config	The configuration object containing various settings for the model. TYPE: Blip2VisionConfig

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def __init__(self, config: Blip2VisionConfig):
    """
    Initializes a new instance of the Blip2VisionModel class.

    Args:
        self: The object instance.
        config (Blip2VisionConfig): The configuration object containing various settings for the model.

    Returns:
        None

    Raises:
        None
    """
    super().__init__(config)
    self.config = config
    embed_dim = config.hidden_size

    self.embeddings = Blip2VisionEmbeddings(config)
    self.encoder = Blip2Encoder(config)
    self.post_layernorm = nn.LayerNorm(embed_dim, eps=config.layer_norm_eps)

    self.post_init()

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2VisionModel.forward(pixel_values=None, output_attentions=None, output_hidden_states=None, return_dict=None)

RETURNS	DESCRIPTION
Union[Tuple, BaseModelOutputWithPooling]	Union[Tuple, BaseModelOutputWithPooling]

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def forward(
    self,
    pixel_values: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple, BaseModelOutputWithPooling]:
    r"""

    Returns:
        `Union[Tuple, BaseModelOutputWithPooling]`
    """
    output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
    output_hidden_states = (
        output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
    )
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    if pixel_values is None:
        raise ValueError("You have to specify pixel_values")

    hidden_states = self.embeddings(pixel_values)

    encoder_outputs = self.encoder(
        inputs_embeds=hidden_states,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    last_hidden_state = encoder_outputs[0]
    last_hidden_state = self.post_layernorm(last_hidden_state)

    pooled_output = last_hidden_state[:, 0, :]
    pooled_output = self.post_layernorm(pooled_output)

    if not return_dict:
        return (last_hidden_state, pooled_output) + encoder_outputs[1:]

    return BaseModelOutputWithPooling(
        last_hidden_state=last_hidden_state,
        pooler_output=pooled_output,
        hidden_states=encoder_outputs.hidden_states,
        attentions=encoder_outputs.attentions,
    )

mindnlp.transformers.models.blip_2.modeling_blip_2.Blip2VisionModel.get_input_embeddings()

This method retrieves the input embeddings from the Blip2VisionModel.

PARAMETER	DESCRIPTION
self	The instance of the Blip2VisionModel class. TYPE: Blip2VisionModel

RETURNS	DESCRIPTION
embeddings	This method returns the input embeddings.

Source code in mindnlp/transformers/models/blip_2/modeling_blip_2.py

def get_input_embeddings(self):
    """
    This method retrieves the input embeddings from the Blip2VisionModel.

    Args:
        self (Blip2VisionModel): The instance of the Blip2VisionModel class.

    Returns:
        embeddings: This method returns the input embeddings.

    Raises:
        None.
    """
    return self.embeddings

mindnlp.transformers.models.blip_2.processing_blip_2.Blip2Processor

Bases: ProcessorMixin

Constructs a BLIP-2 processor which wraps a BLIP image processor and an OPT/T5 tokenizer into a single processor.

[BlipProcessor] offers all the functionalities of [BlipImageProcessor] and [AutoTokenizer]. See the docstring of [~BlipProcessor.__call__] and [~BlipProcessor.decode] for more information.

PARAMETER	DESCRIPTION
image_processor	An instance of [BlipImageProcessor]. The image processor is a required input. TYPE: `BlipImageProcessor`
tokenizer	An instance of ['PreTrainedTokenizer`]. The tokenizer is a required input. TYPE: `AutoTokenizer`

Source code in mindnlp/transformers/models/blip_2/processing_blip_2.py

class Blip2Processor(ProcessorMixin):
    r"""
    Constructs a BLIP-2 processor which wraps a BLIP image processor and an OPT/T5 tokenizer into a single processor.

    [`BlipProcessor`] offers all the functionalities of [`BlipImageProcessor`] and [`AutoTokenizer`]. See the docstring
    of [`~BlipProcessor.__call__`] and [`~BlipProcessor.decode`] for more information.

    Args:
        image_processor (`BlipImageProcessor`):
            An instance of [`BlipImageProcessor`]. The image processor is a required input.
        tokenizer (`AutoTokenizer`):
            An instance of ['PreTrainedTokenizer`]. The tokenizer is a required input.
    """
    attributes = ["image_processor", "tokenizer"]
    image_processor_class = "BlipImageProcessor"
    tokenizer_class = "AutoTokenizer"

    # Copied from transformers.models.blip.processing_blip.BlipProcessor.__init__
    def __init__(self, image_processor, tokenizer):
        """
        Initializes a new instance of the Blip2Processor class.

        Args:
            self: The instance of the class.
            image_processor: An object representing the image processor to be used.
                It should have the necessary methods and attributes required for image processing.
            tokenizer: An object representing the tokenizer to be used.
                t should have the necessary methods and attributes required for tokenization.
                The 'return_token_type_ids' attribute of the tokenizer will be set to False.

        Returns:
            None.

        Raises:
            None.
        """
        tokenizer.return_token_type_ids = False
        super().__init__(image_processor, tokenizer)
        self.current_processor = self.image_processor

    # Copied from transformers.models.blip.processing_blip.BlipProcessor.__call__
    def __call__(
        self,
        images: ImageInput = None,
        text: Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]] = None,
        add_special_tokens: bool = True,
        padding: Union[bool, str, PaddingStrategy] = False,
        truncation: Union[bool, str, TruncationStrategy] = None,
        max_length: Optional[int] = None,
        stride: int = 0,
        pad_to_multiple_of: Optional[int] = None,
        return_attention_mask: Optional[bool] = None,
        return_overflowing_tokens: bool = False,
        return_special_tokens_mask: bool = False,
        return_offsets_mapping: bool = False,
        return_token_type_ids: bool = False,
        return_length: bool = False,
        verbose: bool = True,
        return_tensors: Optional[Union[str, TensorType]] = None,
        **kwargs,
    ) -> BatchEncoding:
        """
        This method uses [`BlipImageProcessor.__call__`] method to prepare image(s) for the model, and
        [`BertTokenizerFast.__call__`] to prepare text for the model.

        Please refer to the docstring of the above two methods for more information.
        """
        if images is None and text is None:
            raise ValueError("You have to specify either images or text.")

        # Get only text
        if images is None:
            self.current_processor = self.tokenizer
            text_encoding = self.tokenizer(
                text=text,
                add_special_tokens=add_special_tokens,
                padding=padding,
                truncation=truncation,
                max_length=max_length,
                stride=stride,
                pad_to_multiple_of=pad_to_multiple_of,
                return_attention_mask=return_attention_mask,
                return_overflowing_tokens=return_overflowing_tokens,
                return_special_tokens_mask=return_special_tokens_mask,
                return_offsets_mapping=return_offsets_mapping,
                return_token_type_ids=return_token_type_ids,
                return_length=return_length,
                verbose=verbose,
                return_tensors=return_tensors,
                **kwargs,
            )
            return text_encoding

        # add pixel_values
        encoding_image_processor = self.image_processor(images, return_tensors=return_tensors)

        if text is not None:
            text_encoding = self.tokenizer(
                text=text,
                add_special_tokens=add_special_tokens,
                padding=padding,
                truncation=truncation,
                max_length=max_length,
                stride=stride,
                pad_to_multiple_of=pad_to_multiple_of,
                return_attention_mask=return_attention_mask,
                return_overflowing_tokens=return_overflowing_tokens,
                return_special_tokens_mask=return_special_tokens_mask,
                return_offsets_mapping=return_offsets_mapping,
                return_token_type_ids=return_token_type_ids,
                return_length=return_length,
                verbose=verbose,
                return_tensors=return_tensors,
                **kwargs,
            )
        else:
            text_encoding = None

        if text_encoding is not None:
            encoding_image_processor.update(text_encoding)

        return encoding_image_processor

    # Copied from transformers.models.blip.processing_blip.BlipProcessor.batch_decode with BertTokenizerFast->PreTrainedTokenizer
    def batch_decode(self, *args, **kwargs):
        """
        This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please
        refer to the docstring of this method for more information.
        """
        return self.tokenizer.batch_decode(*args, **kwargs)

    # Copied from transformers.models.blip.processing_blip.BlipProcessor.decode with BertTokenizerFast->PreTrainedTokenizer
    def decode(self, *args, **kwargs):
        """
        This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to
        the docstring of this method for more information.
        """
        return self.tokenizer.decode(*args, **kwargs)

    @property
    # Copied from transformers.models.blip.processing_blip.BlipProcessor.model_input_names
    def model_input_names(self):
        """
        Retrieves the names of the model inputs for the 'Blip2Processor' class.

        Args:
            self: An instance of the 'Blip2Processor' class.

        Returns:
            A list of strings containing the names of the model inputs.

        Raises:
            None.
        """
        tokenizer_input_names = self.tokenizer.model_input_names
        image_processor_input_names = self.image_processor.model_input_names
        return list(dict.fromkeys(tokenizer_input_names + image_processor_input_names))

mindnlp.transformers.models.blip_2.processing_blip_2.Blip2Processor.model_input_names property

Retrieves the names of the model inputs for the 'Blip2Processor' class.

PARAMETER	DESCRIPTION
self	An instance of the 'Blip2Processor' class.

RETURNS	DESCRIPTION
	A list of strings containing the names of the model inputs.

mindnlp.transformers.models.blip_2.processing_blip_2.Blip2Processor.__call__(images=None, text=None, add_special_tokens=True, padding=False, truncation=None, max_length=None, stride=0, pad_to_multiple_of=None, return_attention_mask=None, return_overflowing_tokens=False, return_special_tokens_mask=False, return_offsets_mapping=False, return_token_type_ids=False, return_length=False, verbose=True, return_tensors=None, **kwargs)

This method uses [BlipImageProcessor.__call__] method to prepare image(s) for the model, and [BertTokenizerFast.__call__] to prepare text for the model.

Please refer to the docstring of the above two methods for more information.

Source code in mindnlp/transformers/models/blip_2/processing_blip_2.py

def __call__(
    self,
    images: ImageInput = None,
    text: Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]] = None,
    add_special_tokens: bool = True,
    padding: Union[bool, str, PaddingStrategy] = False,
    truncation: Union[bool, str, TruncationStrategy] = None,
    max_length: Optional[int] = None,
    stride: int = 0,
    pad_to_multiple_of: Optional[int] = None,
    return_attention_mask: Optional[bool] = None,
    return_overflowing_tokens: bool = False,
    return_special_tokens_mask: bool = False,
    return_offsets_mapping: bool = False,
    return_token_type_ids: bool = False,
    return_length: bool = False,
    verbose: bool = True,
    return_tensors: Optional[Union[str, TensorType]] = None,
    **kwargs,
) -> BatchEncoding:
    """
    This method uses [`BlipImageProcessor.__call__`] method to prepare image(s) for the model, and
    [`BertTokenizerFast.__call__`] to prepare text for the model.

    Please refer to the docstring of the above two methods for more information.
    """
    if images is None and text is None:
        raise ValueError("You have to specify either images or text.")

    # Get only text
    if images is None:
        self.current_processor = self.tokenizer
        text_encoding = self.tokenizer(
            text=text,
            add_special_tokens=add_special_tokens,
            padding=padding,
            truncation=truncation,
            max_length=max_length,
            stride=stride,
            pad_to_multiple_of=pad_to_multiple_of,
            return_attention_mask=return_attention_mask,
            return_overflowing_tokens=return_overflowing_tokens,
            return_special_tokens_mask=return_special_tokens_mask,
            return_offsets_mapping=return_offsets_mapping,
            return_token_type_ids=return_token_type_ids,
            return_length=return_length,
            verbose=verbose,
            return_tensors=return_tensors,
            **kwargs,
        )
        return text_encoding

    # add pixel_values
    encoding_image_processor = self.image_processor(images, return_tensors=return_tensors)

    if text is not None:
        text_encoding = self.tokenizer(
            text=text,
            add_special_tokens=add_special_tokens,
            padding=padding,
            truncation=truncation,
            max_length=max_length,
            stride=stride,
            pad_to_multiple_of=pad_to_multiple_of,
            return_attention_mask=return_attention_mask,
            return_overflowing_tokens=return_overflowing_tokens,
            return_special_tokens_mask=return_special_tokens_mask,
            return_offsets_mapping=return_offsets_mapping,
            return_token_type_ids=return_token_type_ids,
            return_length=return_length,
            verbose=verbose,
            return_tensors=return_tensors,
            **kwargs,
        )
    else:
        text_encoding = None

    if text_encoding is not None:
        encoding_image_processor.update(text_encoding)

    return encoding_image_processor

mindnlp.transformers.models.blip_2.processing_blip_2.Blip2Processor.__init__(image_processor, tokenizer)

Initializes a new instance of the Blip2Processor class.

PARAMETER	DESCRIPTION
self	The instance of the class.
image_processor	An object representing the image processor to be used. It should have the necessary methods and attributes required for image processing.
tokenizer	An object representing the tokenizer to be used. t should have the necessary methods and attributes required for tokenization. The 'return_token_type_ids' attribute of the tokenizer will be set to False.

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/blip_2/processing_blip_2.py

def __init__(self, image_processor, tokenizer):
    """
    Initializes a new instance of the Blip2Processor class.

    Args:
        self: The instance of the class.
        image_processor: An object representing the image processor to be used.
            It should have the necessary methods and attributes required for image processing.
        tokenizer: An object representing the tokenizer to be used.
            t should have the necessary methods and attributes required for tokenization.
            The 'return_token_type_ids' attribute of the tokenizer will be set to False.

    Returns:
        None.

    Raises:
        None.
    """
    tokenizer.return_token_type_ids = False
    super().__init__(image_processor, tokenizer)
    self.current_processor = self.image_processor

mindnlp.transformers.models.blip_2.processing_blip_2.Blip2Processor.batch_decode(*args, **kwargs)

This method forwards all its arguments to PreTrainedTokenizer's [~PreTrainedTokenizer.batch_decode]. Please refer to the docstring of this method for more information.

Source code in mindnlp/transformers/models/blip_2/processing_blip_2.py

def batch_decode(self, *args, **kwargs):
    """
    This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please
    refer to the docstring of this method for more information.
    """
    return self.tokenizer.batch_decode(*args, **kwargs)

mindnlp.transformers.models.blip_2.processing_blip_2.Blip2Processor.decode(*args, **kwargs)

This method forwards all its arguments to PreTrainedTokenizer's [~PreTrainedTokenizer.decode]. Please refer to the docstring of this method for more information.

Source code in mindnlp/transformers/models/blip_2/processing_blip_2.py

def decode(self, *args, **kwargs):
    """
    This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to
    the docstring of this method for more information.
    """
    return self.tokenizer.decode(*args, **kwargs)