blip

mindnlp.transformers.models.blip.configuration_blip.BlipConfig

Bases: PretrainedConfig

[BlipConfig] is the configuration class to store the configuration of a [BlipModel]. It is used to instantiate a BLIP model according to the specified arguments, defining the text model and vision model configs. Instantiating a configuration with the defaults will yield a similar configuration to that of the BLIP-base Salesforce/blip-vqa-base architecture.

Configuration objects inherit from [PretrainedConfig] and can be used to control the model outputs. Read the documentation from [PretrainedConfig] for more information.

PARAMETER	DESCRIPTION
text_config	Dictionary of configuration options used to initialize [BlipTextConfig]. TYPE: `dict`, *optional* DEFAULT: None
vision_config	Dictionary of configuration options used to initialize [BlipVisionConfig]. TYPE: `dict`, *optional* DEFAULT: None
projection_dim	Dimentionality of text and vision projection layers. TYPE: `int`, *optional*, defaults to 512 DEFAULT: 512
logit_scale_init_value	The inital value of the logit_scale paramter. Default is used as per the original BLIP implementation. TYPE: `float`, *optional*, defaults to 2.6592 DEFAULT: 2.6592
image_text_hidden_size	Dimentionality of the hidden state of the image-text fusion layer. TYPE: `int`, *optional*, defaults to 256 DEFAULT: 256
label_smoothing	A float in [0.0, 1.0]. Specifies the amount of smoothing when computing the loss, where 0.0 means no smoothing. The targets become a mixture of the original ground truth and a uniform distribution as described in Rethinking the Inception Architecture for Computer Vision <https://arxiv.org/abs/1512.00567>__. Default: :math:0.0. TYPE: float, optional, *optional*, defaults to 0.0 DEFAULT: 0.0
kwargs	Dictionary of keyword arguments. TYPE: *optional* DEFAULT: {}

Example

>>> from transformers import BlipConfig, BlipModel
...
>>> # Initializing a BlipConfig with Salesforce/blip-vqa-base style configuration
>>> configuration = BlipConfig()
...
>>> # Initializing a BlipPModel (with random weights) from the Salesforce/blip-vqa-base style configuration
>>> model = BlipModel(configuration)
...
>>> # Accessing the model configuration
>>> configuration = model.config
...
>>> # We can also initialize a BlipConfig from a BlipTextConfig and a BlipVisionConfig
...
>>> # Initializing a BLIPText and BLIPVision configuration
>>> config_text = BlipTextConfig()
>>> config_vision = BlipVisionConfig()
...
>>> config = BlipConfig.from_text_vision_configs(config_text, config_vision)

Source code in mindnlp/transformers/models/blip/configuration_blip.py

class BlipConfig(PretrainedConfig):
    r"""
    [`BlipConfig`] is the configuration class to store the configuration of a [`BlipModel`]. It is used to instantiate
    a BLIP model according to the specified arguments, defining the text model and vision model configs. Instantiating
    a configuration with the defaults will yield a similar configuration to that of the BLIP-base
    [Salesforce/blip-vqa-base](https://huggingface.co/Salesforce/blip-vqa-base) architecture.

    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
    documentation from [`PretrainedConfig`] for more information.

    Args:
        text_config (`dict`, *optional*):
            Dictionary of configuration options used to initialize [`BlipTextConfig`].
        vision_config (`dict`, *optional*):
            Dictionary of configuration options used to initialize [`BlipVisionConfig`].
        projection_dim (`int`, *optional*, defaults to 512):
            Dimentionality of text and vision projection layers.
        logit_scale_init_value (`float`, *optional*, defaults to 2.6592):
            The inital value of the *logit_scale* paramter. Default is used as per the original BLIP implementation.
        image_text_hidden_size (`int`, *optional*, defaults to 256):
            Dimentionality of the hidden state of the image-text fusion layer.
        label_smoothing (float, optional, *optional*, defaults to 0.0):
            A float in [0.0, 1.0]. Specifies the amount of smoothing when computing the loss, where 0.0 means no smoothing. The targets
            become a mixture of the original ground truth and a uniform distribution as described in
            `Rethinking the Inception Architecture for Computer Vision <https://arxiv.org/abs/1512.00567>`__. Default: :math:`0.0`.
        kwargs (*optional*):
            Dictionary of keyword arguments.

    Example:
        ```python
        >>> from transformers import BlipConfig, BlipModel
        ...
        >>> # Initializing a BlipConfig with Salesforce/blip-vqa-base style configuration
        >>> configuration = BlipConfig()
        ...
        >>> # Initializing a BlipPModel (with random weights) from the Salesforce/blip-vqa-base style configuration
        >>> model = BlipModel(configuration)
        ...
        >>> # Accessing the model configuration
        >>> configuration = model.config
        ...
        >>> # We can also initialize a BlipConfig from a BlipTextConfig and a BlipVisionConfig
        ...
        >>> # Initializing a BLIPText and BLIPVision configuration
        >>> config_text = BlipTextConfig()
        >>> config_vision = BlipVisionConfig()
        ...
        >>> config = BlipConfig.from_text_vision_configs(config_text, config_vision)
        ```
    """
    model_type = "blip"

    def __init__(
        self,
        text_config=None,
        vision_config=None,
        projection_dim=512,
        logit_scale_init_value=2.6592,
        image_text_hidden_size=256,
        label_smoothing=0.0,
        **kwargs,
    ):
        """
        Initializes a BlipConfig object with the provided parameters.

        Args:
            self (BlipConfig): The instance of the BlipConfig class.
            text_config (dict, optional): Configuration parameters for text. Defaults to None.
            vision_config (dict, optional): Configuration parameters for vision. Defaults to None.
            projection_dim (int, optional): The dimension of the projection. Defaults to 512.
            logit_scale_init_value (float, optional): The initial value for logit scaling. Defaults to 2.6592.
            image_text_hidden_size (int, optional): The size of the hidden layer for image and text. Defaults to 256.
            label_smoothing (float, optional): The amount of label smoothing. Defaults to 0.0.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(**kwargs)

        if text_config is None:
            text_config = {}
            logger.info("`text_config` is `None`. Initializing the `BlipTextConfig` with default values.")

        if vision_config is None:
            vision_config = {}
            logger.info("`vision_config` is `None`. Initializing the `BlipVisionConfig` with default values.")

        self.text_config = BlipTextConfig(**text_config)
        self.vision_config = BlipVisionConfig(**vision_config)

        self.text_config.encoder_hidden_size = self.vision_config.hidden_size

        self.projection_dim = projection_dim
        self.logit_scale_init_value = logit_scale_init_value
        self.initializer_factor = 1.0
        self.initializer_range = 0.02
        self.image_text_hidden_size = image_text_hidden_size
        self.label_smoothing = label_smoothing

    @classmethod
    def from_text_vision_configs(cls, text_config: BlipTextConfig, vision_config: BlipVisionConfig, **kwargs):
        r"""
        Instantiate a [`BlipConfig`] (or a derived class) from blip text model configuration and blip vision model
        configuration.

        Returns:
            [`BlipConfig`]: An instance of a configuration object
        """
        return cls(text_config=text_config.to_dict(), vision_config=vision_config.to_dict(), **kwargs)

mindnlp.transformers.models.blip.configuration_blip.BlipConfig.__init__(text_config=None, vision_config=None, projection_dim=512, logit_scale_init_value=2.6592, image_text_hidden_size=256, label_smoothing=0.0, **kwargs)

Initializes a BlipConfig object with the provided parameters.

PARAMETER	DESCRIPTION
self	The instance of the BlipConfig class. TYPE: BlipConfig
text_config	Configuration parameters for text. Defaults to None. TYPE: dict DEFAULT: None
vision_config	Configuration parameters for vision. Defaults to None. TYPE: dict DEFAULT: None
projection_dim	The dimension of the projection. Defaults to 512. TYPE: int DEFAULT: 512
logit_scale_init_value	The initial value for logit scaling. Defaults to 2.6592. TYPE: float DEFAULT: 2.6592
image_text_hidden_size	The size of the hidden layer for image and text. Defaults to 256. TYPE: int DEFAULT: 256
label_smoothing	The amount of label smoothing. Defaults to 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/blip/configuration_blip.py

def __init__(
    self,
    text_config=None,
    vision_config=None,
    projection_dim=512,
    logit_scale_init_value=2.6592,
    image_text_hidden_size=256,
    label_smoothing=0.0,
    **kwargs,
):
    """
    Initializes a BlipConfig object with the provided parameters.

    Args:
        self (BlipConfig): The instance of the BlipConfig class.
        text_config (dict, optional): Configuration parameters for text. Defaults to None.
        vision_config (dict, optional): Configuration parameters for vision. Defaults to None.
        projection_dim (int, optional): The dimension of the projection. Defaults to 512.
        logit_scale_init_value (float, optional): The initial value for logit scaling. Defaults to 2.6592.
        image_text_hidden_size (int, optional): The size of the hidden layer for image and text. Defaults to 256.
        label_smoothing (float, optional): The amount of label smoothing. Defaults to 0.0.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(**kwargs)

    if text_config is None:
        text_config = {}
        logger.info("`text_config` is `None`. Initializing the `BlipTextConfig` with default values.")

    if vision_config is None:
        vision_config = {}
        logger.info("`vision_config` is `None`. Initializing the `BlipVisionConfig` with default values.")

    self.text_config = BlipTextConfig(**text_config)
    self.vision_config = BlipVisionConfig(**vision_config)

    self.text_config.encoder_hidden_size = self.vision_config.hidden_size

    self.projection_dim = projection_dim
    self.logit_scale_init_value = logit_scale_init_value
    self.initializer_factor = 1.0
    self.initializer_range = 0.02
    self.image_text_hidden_size = image_text_hidden_size
    self.label_smoothing = label_smoothing

mindnlp.transformers.models.blip.configuration_blip.BlipConfig.from_text_vision_configs(text_config, vision_config, **kwargs) classmethod

Instantiate a [BlipConfig] (or a derived class) from blip text model configuration and blip vision model configuration.

RETURNS	DESCRIPTION
	[BlipConfig]: An instance of a configuration object

Source code in mindnlp/transformers/models/blip/configuration_blip.py

@classmethod
def from_text_vision_configs(cls, text_config: BlipTextConfig, vision_config: BlipVisionConfig, **kwargs):
    r"""
    Instantiate a [`BlipConfig`] (or a derived class) from blip text model configuration and blip vision model
    configuration.

    Returns:
        [`BlipConfig`]: An instance of a configuration object
    """
    return cls(text_config=text_config.to_dict(), vision_config=vision_config.to_dict(), **kwargs)

mindnlp.transformers.models.blip.configuration_blip.BlipVisionConfig

Bases: PretrainedConfig

This is the configuration class to store the configuration of a [BlipVisionModel]. It is used to instantiate a BLIP vision model according to the specified arguments, defining the model architecture. Instantiating a configuration defaults will yield a similar configuration to that of the Blip-base Salesforce/blip-vqa-base architecture.

Configuration objects inherit from [PretrainedConfig] and can be used to control the model outputs. Read the documentation from [PretrainedConfig] for more information.

PARAMETER	DESCRIPTION
hidden_size	Dimensionality of the encoder layers and the pooler layer. TYPE: `int`, *optional*, defaults to 768 DEFAULT: 768
intermediate_size	Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. TYPE: `int`, *optional*, defaults to 3072 DEFAULT: 3072
num_hidden_layers	Number of hidden layers in the Transformer encoder. TYPE: `int`, *optional*, defaults to 12 DEFAULT: 12
num_attention_heads	Number of attention heads for each attention layer in the Transformer encoder. TYPE: `int`, *optional*, defaults to 12 DEFAULT: 12
image_size	The size (resolution) of each image. TYPE: `int`, *optional*, defaults to 384 DEFAULT: 384
patch_size	The size (resolution) of each patch. TYPE: `int`, *optional*, defaults to 16 DEFAULT: 16
hidden_act	The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu", "relu", "selu" and "gelu_new" `"gelu" are supported. TYPE: `str` or `function`, *optional*, defaults to `"gelu"` DEFAULT: 'gelu'
layer_norm_eps	The epsilon used by the layer normalization layers. TYPE: `float`, *optional*, defaults to 1e-5 DEFAULT: 1e-05
attention_dropout	The dropout ratio for the attention probabilities. TYPE: `float`, *optional*, defaults to 0.0 DEFAULT: 0.0
initializer_range	The standard deviation of the truncated_normal_initializer for initializing all weight matrices. TYPE: `float`, *optional*, defaults to 1e-10 DEFAULT: 1e-10

Example

>>> from transformers import BlipVisionConfig, BlipVisionModel
...
>>> # Initializing a BlipVisionConfig with Salesforce/blip-vqa-base style configuration
>>> configuration = BlipVisionConfig()
...
>>> # Initializing a BlipVisionModel (with random weights) from the Salesforce/blip-vqa-base style configuration
>>> model = BlipVisionModel(configuration)
...
>>> # Accessing the model configuration
>>> configuration = model.config

Source code in mindnlp/transformers/models/blip/configuration_blip.py

class BlipVisionConfig(PretrainedConfig):
    r"""
    This is the configuration class to store the configuration of a [`BlipVisionModel`]. It is used to instantiate a
    BLIP vision model according to the specified arguments, defining the model architecture. Instantiating a
    configuration defaults will yield a similar configuration to that of the Blip-base
    [Salesforce/blip-vqa-base](https://huggingface.co/Salesforce/blip-vqa-base) architecture.

    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
    documentation from [`PretrainedConfig`] for more information.


    Args:
        hidden_size (`int`, *optional*, defaults to 768):
            Dimensionality of the encoder layers and the pooler layer.
        intermediate_size (`int`, *optional*, defaults to 3072):
            Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder.
        num_hidden_layers (`int`, *optional*, defaults to 12):
            Number of hidden layers in the Transformer encoder.
        num_attention_heads (`int`, *optional*, defaults to 12):
            Number of attention heads for each attention layer in the Transformer encoder.
        image_size (`int`, *optional*, defaults to 384):
            The size (resolution) of each image.
        patch_size (`int`, *optional*, defaults to 16):
            The size (resolution) of each patch.
        hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`):
            The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`,
            `"relu"`, `"selu"` and `"gelu_new"` ``"gelu"` are supported.
        layer_norm_eps (`float`, *optional*, defaults to 1e-5):
            The epsilon used by the layer normalization layers.
        attention_dropout (`float`, *optional*, defaults to 0.0):
            The dropout ratio for the attention probabilities.
        initializer_range (`float`, *optional*, defaults to 1e-10):
            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.

    Example:
        ```python
        >>> from transformers import BlipVisionConfig, BlipVisionModel
        ...
        >>> # Initializing a BlipVisionConfig with Salesforce/blip-vqa-base style configuration
        >>> configuration = BlipVisionConfig()
        ...
        >>> # Initializing a BlipVisionModel (with random weights) from the Salesforce/blip-vqa-base style configuration
        >>> model = BlipVisionModel(configuration)
        ...
        >>> # Accessing the model configuration
        >>> configuration = model.config
        ```
    """
    model_type = "blip_vision_model"

    def __init__(
        self,
        hidden_size=768,
        intermediate_size=3072,
        projection_dim=512,
        num_hidden_layers=12,
        num_attention_heads=12,
        image_size=384,
        patch_size=16,
        hidden_act="gelu",
        layer_norm_eps=1e-5,
        attention_dropout=0.0,
        initializer_range=1e-10,
        **kwargs,
    ):
        '''
        Initializes a BlipVisionConfig instance.

        Args:
            self: The object instance.
            hidden_size (int, optional): The size of the hidden layers. Defaults to 768.
            intermediate_size (int, optional): The size of the intermediate layers. Defaults to 3072.
            projection_dim (int, optional): The dimension of the projected output. Defaults to 512.
            num_hidden_layers (int, optional): The number of hidden layers. Defaults to 12.
            num_attention_heads (int, optional): The number of attention heads. Defaults to 12.
            image_size (int, optional): The size of the input image. Defaults to 384.
            patch_size (int, optional): The size of the image patch. Defaults to 16.
            hidden_act (str, optional): The activation function for the hidden layers. Defaults to 'gelu'.
            layer_norm_eps (float, optional): The epsilon value for layer normalization. Defaults to 1e-05.
            attention_dropout (float, optional): The dropout rate for attention layers. Defaults to 0.0.
            initializer_range (float, optional): The range for parameter initialization. Defaults to 1e-10.

        Returns:
            None.

        Raises:
            None.
        '''
        super().__init__(**kwargs)

        self.hidden_size = hidden_size
        self.intermediate_size = intermediate_size
        self.projection_dim = projection_dim
        self.num_hidden_layers = num_hidden_layers
        self.num_attention_heads = num_attention_heads
        self.patch_size = patch_size
        self.image_size = image_size
        self.initializer_range = initializer_range
        self.attention_dropout = attention_dropout
        self.layer_norm_eps = layer_norm_eps
        self.hidden_act = hidden_act

    @classmethod
    def from_pretrained(cls, pretrained_model_name_or_path: Union[str, os.PathLike], **kwargs) -> "PretrainedConfig":
        """
        This method 'from_pretrained' in the class 'BlipVisionConfig' is used to create a new instance of the class by loading a pretrained model configuration.

        Args:
            cls (class): The class object itself, automatically passed as the first argument.
            pretrained_model_name_or_path (Union[str, os.PathLike]): A string representing the name or path of
                the pretrained model. It can be either a string or a PathLike object. This parameter is used to
                fetch the configuration dictionary for the pretrained model.

        Returns:
            PretrainedConfig: An instance of the 'PretrainedConfig' class representing the configuration of
                the pretrained model. The method returns this configuration for further use.

        Raises:
            None:
                No specific exceptions are documented to be raised by this method based on the provided code snippet.
                However, potential exceptions might include:

                - KeyError: If the 'model_type' key is missing in the configuration dictionary.
                - AttributeError: If the 'model_type' attribute is not present in the class.
                - Warning: If the model type being used is different from the expected model type, a warning message is logged.

        Note:
            It is recommended to handle exceptions that may occur during the execution of this method to
            ensure proper error handling and flow control.
        """
        config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)

        # get the vision config dict if we are loading from BlipConfig
        if config_dict.get("model_type") == "blip":
            config_dict = config_dict["vision_config"]

        if "model_type" in config_dict and hasattr(cls, "model_type") and config_dict["model_type"] != cls.model_type:
            logger.warning(
                f"You are using a model of type {config_dict['model_type']} to instantiate a model of type "
                f"{cls.model_type}. This is not supported for all configurations of models and can yield errors."
            )

        return cls.from_dict(config_dict, **kwargs)

mindnlp.transformers.models.blip.configuration_blip.BlipVisionConfig.__init__(hidden_size=768, intermediate_size=3072, projection_dim=512, num_hidden_layers=12, num_attention_heads=12, image_size=384, patch_size=16, hidden_act='gelu', layer_norm_eps=1e-05, attention_dropout=0.0, initializer_range=1e-10, **kwargs)

Initializes a BlipVisionConfig instance.

PARAMETER	DESCRIPTION
self	The object instance.
hidden_size	The size of the hidden layers. Defaults to 768. TYPE: int DEFAULT: 768
intermediate_size	The size of the intermediate layers. Defaults to 3072. TYPE: int DEFAULT: 3072
projection_dim	The dimension of the projected output. Defaults to 512. TYPE: int DEFAULT: 512
num_hidden_layers	The number of hidden layers. Defaults to 12. TYPE: int DEFAULT: 12
num_attention_heads	The number of attention heads. Defaults to 12. TYPE: int DEFAULT: 12
image_size	The size of the input image. Defaults to 384. TYPE: int DEFAULT: 384
patch_size	The size of the image patch. Defaults to 16. TYPE: int DEFAULT: 16
hidden_act	The activation function for the hidden layers. Defaults to 'gelu'. TYPE: str DEFAULT: 'gelu'
layer_norm_eps	The epsilon value for layer normalization. Defaults to 1e-05. TYPE: float DEFAULT: 1e-05
attention_dropout	The dropout rate for attention layers. Defaults to 0.0. TYPE: float DEFAULT: 0.0
initializer_range	The range for parameter initialization. Defaults to 1e-10. TYPE: float DEFAULT: 1e-10

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/blip/configuration_blip.py

def __init__(
    self,
    hidden_size=768,
    intermediate_size=3072,
    projection_dim=512,
    num_hidden_layers=12,
    num_attention_heads=12,
    image_size=384,
    patch_size=16,
    hidden_act="gelu",
    layer_norm_eps=1e-5,
    attention_dropout=0.0,
    initializer_range=1e-10,
    **kwargs,
):
    '''
    Initializes a BlipVisionConfig instance.

    Args:
        self: The object instance.
        hidden_size (int, optional): The size of the hidden layers. Defaults to 768.
        intermediate_size (int, optional): The size of the intermediate layers. Defaults to 3072.
        projection_dim (int, optional): The dimension of the projected output. Defaults to 512.
        num_hidden_layers (int, optional): The number of hidden layers. Defaults to 12.
        num_attention_heads (int, optional): The number of attention heads. Defaults to 12.
        image_size (int, optional): The size of the input image. Defaults to 384.
        patch_size (int, optional): The size of the image patch. Defaults to 16.
        hidden_act (str, optional): The activation function for the hidden layers. Defaults to 'gelu'.
        layer_norm_eps (float, optional): The epsilon value for layer normalization. Defaults to 1e-05.
        attention_dropout (float, optional): The dropout rate for attention layers. Defaults to 0.0.
        initializer_range (float, optional): The range for parameter initialization. Defaults to 1e-10.

    Returns:
        None.

    Raises:
        None.
    '''
    super().__init__(**kwargs)

    self.hidden_size = hidden_size
    self.intermediate_size = intermediate_size
    self.projection_dim = projection_dim
    self.num_hidden_layers = num_hidden_layers
    self.num_attention_heads = num_attention_heads
    self.patch_size = patch_size
    self.image_size = image_size
    self.initializer_range = initializer_range
    self.attention_dropout = attention_dropout
    self.layer_norm_eps = layer_norm_eps
    self.hidden_act = hidden_act

mindnlp.transformers.models.blip.configuration_blip.BlipVisionConfig.from_pretrained(pretrained_model_name_or_path, **kwargs) classmethod

This method 'from_pretrained' in the class 'BlipVisionConfig' is used to create a new instance of the class by loading a pretrained model configuration.

PARAMETER	DESCRIPTION
cls	The class object itself, automatically passed as the first argument. TYPE: class
pretrained_model_name_or_path	A string representing the name or path of the pretrained model. It can be either a string or a PathLike object. This parameter is used to fetch the configuration dictionary for the pretrained model. TYPE: Union[str, PathLike]

RETURNS	DESCRIPTION
PretrainedConfig	An instance of the 'PretrainedConfig' class representing the configuration of the pretrained model. The method returns this configuration for further use. TYPE: PretrainedConfig

RAISES

DESCRIPTION

None

No specific exceptions are documented to be raised by this method based on the provided code snippet. However, potential exceptions might include: KeyError: If the 'model_type' key is missing in the configuration dictionary. AttributeError: If the 'model_type' attribute is not present in the class. Warning: If the model type being used is different from the expected model type, a warning message is logged.

Note

It is recommended to handle exceptions that may occur during the execution of this method to ensure proper error handling and flow control.

Source code in mindnlp/transformers/models/blip/configuration_blip.py

@classmethod
def from_pretrained(cls, pretrained_model_name_or_path: Union[str, os.PathLike], **kwargs) -> "PretrainedConfig":
    """
    This method 'from_pretrained' in the class 'BlipVisionConfig' is used to create a new instance of the class by loading a pretrained model configuration.

    Args:
        cls (class): The class object itself, automatically passed as the first argument.
        pretrained_model_name_or_path (Union[str, os.PathLike]): A string representing the name or path of
            the pretrained model. It can be either a string or a PathLike object. This parameter is used to
            fetch the configuration dictionary for the pretrained model.

    Returns:
        PretrainedConfig: An instance of the 'PretrainedConfig' class representing the configuration of
            the pretrained model. The method returns this configuration for further use.

    Raises:
        None:
            No specific exceptions are documented to be raised by this method based on the provided code snippet.
            However, potential exceptions might include:

            - KeyError: If the 'model_type' key is missing in the configuration dictionary.
            - AttributeError: If the 'model_type' attribute is not present in the class.
            - Warning: If the model type being used is different from the expected model type, a warning message is logged.

    Note:
        It is recommended to handle exceptions that may occur during the execution of this method to
        ensure proper error handling and flow control.
    """
    config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)

    # get the vision config dict if we are loading from BlipConfig
    if config_dict.get("model_type") == "blip":
        config_dict = config_dict["vision_config"]

    if "model_type" in config_dict and hasattr(cls, "model_type") and config_dict["model_type"] != cls.model_type:
        logger.warning(
            f"You are using a model of type {config_dict['model_type']} to instantiate a model of type "
            f"{cls.model_type}. This is not supported for all configurations of models and can yield errors."
        )

    return cls.from_dict(config_dict, **kwargs)

mindnlp.transformers.models.blip.configuration_blip.BlipTextConfig

Bases: PretrainedConfig

This is the configuration class to store the configuration of a [BlipTextModel]. It is used to instantiate a BLIP text model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the BlipText used by the base architectures.

Configuration objects inherit from [PretrainedConfig] and can be used to control the model outputs. Read the documentation from [PretrainedConfig] for more information.

PARAMETER	DESCRIPTION
vocab_size	Vocabulary size of the Blip text model. Defines the number of different tokens that can be represented by the inputs_ids passed when calling [BlipModel]. TYPE: `int`, *optional*, defaults to 30524 DEFAULT: 30524
hidden_size	Dimensionality of the encoder layers and the pooler layer. TYPE: `int`, *optional*, defaults to 768 DEFAULT: 768
encoder_hidden_size	Dimensionality of the encoder layers from the vision model. TYPE: `int`, *optional*, defaults to 768 DEFAULT: 768
intermediate_size	Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. TYPE: `int`, *optional*, defaults to 3072 DEFAULT: 3072
num_hidden_layers	Number of hidden layers in the Transformer encoder. TYPE: `int`, *optional*, defaults to 12 DEFAULT: 12
num_attention_heads	Number of attention heads for each attention layer in the Transformer encoder. TYPE: `int`, *optional*, defaults to 8 DEFAULT: 8
max_position_embeddings	The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048). TYPE: `int`, *optional*, defaults to 512 DEFAULT: 512
hidden_act	The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu", "relu", "selu" and "gelu_new" `"gelu" are supported. TYPE: `str` or `function`, *optional*, defaults to `"gelu"` DEFAULT: 'gelu'
layer_norm_eps	The epsilon used by the layer normalization layers. TYPE: `float`, *optional*, defaults to 1e-12 DEFAULT: 1e-12
hidden_dropout_prob	The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. TYPE: `float`, *optional*, defaults to 0.0 DEFAULT: 0.0
attention_dropout	The dropout ratio for the attention probabilities. TYPE: `float`, *optional*, defaults to 0.0
initializer_range	The standard deviation of the truncated_normal_initializer for initializing all weight matrices. TYPE: `float`, *optional*, defaults to 0.02 DEFAULT: 0.02
bos_token_id	The id of the beginning-of-sequence token. TYPE: `int`, *optional*, defaults to 30522 DEFAULT: 30522
eos_token_id	The id of the end-of-sequence token. TYPE: `int`, *optional*, defaults to 2 DEFAULT: 2
pad_token_id	The id of the padding token. TYPE: `int`, *optional*, defaults to 0 DEFAULT: 0
sep_token_id	The id of the separator token. TYPE: `int`, *optional*, defaults to 102 DEFAULT: 102
is_decoder	Whether the model is used as a decoder. TYPE: `bool`, *optional*, defaults to `True` DEFAULT: True
use_cache	Whether or not the model should return the last key/values attentions (not used by all models). TYPE: `bool`, *optional*, defaults to `True` DEFAULT: True
label_smoothing	A float in [0.0, 1.0]. Specifies the amount of smoothing when computing the loss, where 0.0 means no smoothing. The targets become a mixture of the original ground truth and a uniform distribution as described in Rethinking the Inception Architecture for Computer Vision <https://arxiv.org/abs/1512.00567>__. Default: :math:0.0. TYPE: float, *optional* DEFAULT: 0.0

Example

>>> from transformers import BlipTextConfig, BlipTextModel
...
>>> # Initializing a BlipTextConfig with Salesforce/blip-vqa-base style configuration
>>> configuration = BlipTextConfig()
...
>>> # Initializing a BlipTextModel (with random weights) from the Salesforce/blip-vqa-base style configuration
>>> model = BlipTextModel(configuration)
...
>>> # Accessing the model configuration
>>> configuration = model.config

Source code in mindnlp/transformers/models/blip/configuration_blip.py

class BlipTextConfig(PretrainedConfig):
    r"""
    This is the configuration class to store the configuration of a [`BlipTextModel`]. It is used to instantiate a BLIP
    text model according to the specified arguments, defining the model architecture. Instantiating a configuration
    with the defaults will yield a similar configuration to that of the `BlipText` used by the [base
    architectures](https://huggingface.co/Salesforce/blip-vqa-base).

    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
    documentation from [`PretrainedConfig`] for more information.


    Args:
        vocab_size (`int`, *optional*, defaults to 30524):
            Vocabulary size of the `Blip` text model. Defines the number of different tokens that can be represented by
            the `inputs_ids` passed when calling [`BlipModel`].
        hidden_size (`int`, *optional*, defaults to 768):
            Dimensionality of the encoder layers and the pooler layer.
        encoder_hidden_size (`int`, *optional*, defaults to 768):
            Dimensionality of the encoder layers from the vision model.
        intermediate_size (`int`, *optional*, defaults to 3072):
            Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder.
        num_hidden_layers (`int`, *optional*, defaults to 12):
            Number of hidden layers in the Transformer encoder.
        num_attention_heads (`int`, *optional*, defaults to 8):
            Number of attention heads for each attention layer in the Transformer encoder.
        max_position_embeddings (`int`, *optional*, defaults to 512):
            The maximum sequence length that this model might ever be used with. Typically set this to something large
            just in case (e.g., 512 or 1024 or 2048).
        hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`):
            The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`,
            `"relu"`, `"selu"` and `"gelu_new"` ``"gelu"` are supported.
        layer_norm_eps (`float`, *optional*, defaults to 1e-12):
            The epsilon used by the layer normalization layers.
        hidden_dropout_prob (`float`, *optional*, defaults to 0.0):
            The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
        attention_dropout (`float`, *optional*, defaults to 0.0):
            The dropout ratio for the attention probabilities.
        initializer_range (`float`, *optional*, defaults to 0.02):
            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
        bos_token_id (`int`, *optional*, defaults to 30522):
            The id of the `beginning-of-sequence` token.
        eos_token_id (`int`, *optional*, defaults to 2):
            The id of the `end-of-sequence` token.
        pad_token_id (`int`, *optional*, defaults to 0):
            The id of the `padding` token.
        sep_token_id (`int`, *optional*, defaults to 102):
            The id of the `separator` token.
        is_decoder (`bool`, *optional*, defaults to `True`):
            Whether the model is used as a decoder.
        use_cache (`bool`, *optional*, defaults to `True`):
            Whether or not the model should return the last key/values attentions (not used by all models).
        label_smoothing (float, *optional*):
            A float in [0.0, 1.0]. Specifies the amount of smoothing when computing the loss, where 0.0 means no smoothing. The targets
            become a mixture of the original ground truth and a uniform distribution as described in
            `Rethinking the Inception Architecture for Computer Vision <https://arxiv.org/abs/1512.00567>`__. Default: :math:`0.0`.

    Example:
        ```python
        >>> from transformers import BlipTextConfig, BlipTextModel
        ...
        >>> # Initializing a BlipTextConfig with Salesforce/blip-vqa-base style configuration
        >>> configuration = BlipTextConfig()
        ...
        >>> # Initializing a BlipTextModel (with random weights) from the Salesforce/blip-vqa-base style configuration
        >>> model = BlipTextModel(configuration)
        ...
        >>> # Accessing the model configuration
        >>> configuration = model.config
        ```
    """
    model_type = "blip_text_model"

    def __init__(
        self,
        vocab_size=30524,
        hidden_size=768,
        encoder_hidden_size=768,
        intermediate_size=3072,
        projection_dim=768,
        num_hidden_layers=12,
        num_attention_heads=8,
        max_position_embeddings=512,
        hidden_act="gelu",
        layer_norm_eps=1e-12,
        hidden_dropout_prob=0.0,
        attention_probs_dropout_prob=0.0,
        initializer_range=0.02,
        bos_token_id=30522,
        eos_token_id=2,
        pad_token_id=0,
        sep_token_id=102,
        is_decoder=True,
        use_cache=True,
        label_smoothing=0.0,
        **kwargs,
    ):
        """
        Initializes a BlipTextConfig object with the given parameters.

        Args:
            self: The BlipTextConfig instance.
            vocab_size (int, optional): The size of the vocabulary. Default is 30524.
            hidden_size (int, optional): The size of the hidden layers. Default is 768.
            encoder_hidden_size (int, optional): The size of the encoder hidden layers. Default is 768.
            intermediate_size (int, optional): The size of the intermediate layers. Default is 3072.
            projection_dim (int, optional): The projection dimension. Default is 768.
            num_hidden_layers (int, optional): The number of hidden layers. Default is 12.
            num_attention_heads (int, optional): The number of attention heads. Default is 8.
            max_position_embeddings (int, optional): The maximum position embeddings. Default is 512.
            hidden_act (str, optional): The activation function for the hidden layers. Default is 'gelu'.
            layer_norm_eps (float, optional): The epsilon value for layer normalization. Default is 1e-12.
            hidden_dropout_prob (float, optional): The dropout probability for the hidden layers. Default is 0.0.
            attention_probs_dropout_prob (float, optional): The dropout probability for attention probabilities. Default is 0.0.
            initializer_range (float, optional): The range for weight initialization. Default is 0.02.
            bos_token_id (int, optional): The ID of the beginning of sentence token. Default is 30522.
            eos_token_id (int, optional): The ID of the end of sentence token. Default is 2.
            pad_token_id (int, optional): The ID of the padding token. Default is 0.
            sep_token_id (int, optional): The ID of the separator token. Default is 102.
            is_decoder (bool, optional): Whether the model is a decoder. Default is True.
            use_cache (bool, optional): Whether to use cache for faster decoding. Default is True.
            label_smoothing (float, optional): The label smoothing factor. Default is 0.0.

        Returns:
            None

        Raises:
            None
        """
        super().__init__(
            pad_token_id=pad_token_id,
            bos_token_id=bos_token_id,
            eos_token_id=eos_token_id,
            sep_token_id=sep_token_id,
            **kwargs,
        )

        self.vocab_size = vocab_size
        self.hidden_size = hidden_size
        self.encoder_hidden_size = encoder_hidden_size
        self.intermediate_size = intermediate_size
        self.projection_dim = projection_dim
        self.hidden_dropout_prob = hidden_dropout_prob
        self.num_hidden_layers = num_hidden_layers
        self.num_attention_heads = num_attention_heads
        self.max_position_embeddings = max_position_embeddings
        self.layer_norm_eps = layer_norm_eps
        self.hidden_act = hidden_act
        self.initializer_range = initializer_range
        self.attention_probs_dropout_prob = attention_probs_dropout_prob
        self.is_decoder = is_decoder
        self.use_cache = use_cache
        self.label_smoothing = label_smoothing

    @classmethod
    def from_pretrained(cls, pretrained_model_name_or_path: Union[str, os.PathLike], **kwargs) -> "PretrainedConfig":
        """
        This method 'from_pretrained' in the class 'BlipTextConfig' is used to instantiate a model configuration object
        based on a pretrained model name or path.

        Args:
            cls (class): The class object itself.
            pretrained_model_name_or_path (Union[str, os.PathLike]): A string representing the name of a pretrained model
                or a valid path to a pretrained model configuration file. This parameter is mandatory and required for
                initializing the configuration object.

        Returns:
            PretrainedConfig: An instance of 'PretrainedConfig' class representing the configuration settings of
                the pretrained model.
                The method returns the configuration object based on the provided pretrained model name or path.

        Raises:
            TypeError: If the provided 'pretrained_model_name_or_path' is not a string or a valid path-like object.
            KeyError: If the 'model_type' key is missing from the configuration dictionary.
            Warning: If the model type in the configuration dictionary does not match the class model type,
                a warning message is logged as this may lead to errors during instantiation.
        """
        config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)

        # get the text config dict if we are loading from BlipConfig
        if config_dict.get("model_type") == "blip":
            config_dict = config_dict["text_config"]

        if "model_type" in config_dict and hasattr(cls, "model_type") and config_dict["model_type"] != cls.model_type:
            logger.warning(
                f"You are using a model of type {config_dict['model_type']} to instantiate a model of type "
                f"{cls.model_type}. This is not supported for all configurations of models and can yield errors."
            )

        return cls.from_dict(config_dict, **kwargs)

mindnlp.transformers.models.blip.configuration_blip.BlipTextConfig.__init__(vocab_size=30524, hidden_size=768, encoder_hidden_size=768, intermediate_size=3072, projection_dim=768, num_hidden_layers=12, num_attention_heads=8, max_position_embeddings=512, hidden_act='gelu', layer_norm_eps=1e-12, hidden_dropout_prob=0.0, attention_probs_dropout_prob=0.0, initializer_range=0.02, bos_token_id=30522, eos_token_id=2, pad_token_id=0, sep_token_id=102, is_decoder=True, use_cache=True, label_smoothing=0.0, **kwargs)

Initializes a BlipTextConfig object with the given parameters.

PARAMETER	DESCRIPTION
self	The BlipTextConfig instance.
vocab_size	The size of the vocabulary. Default is 30524. TYPE: int DEFAULT: 30524
hidden_size	The size of the hidden layers. Default is 768. TYPE: int DEFAULT: 768
encoder_hidden_size	The size of the encoder hidden layers. Default is 768. TYPE: int DEFAULT: 768
intermediate_size	The size of the intermediate layers. Default is 3072. TYPE: int DEFAULT: 3072
projection_dim	The projection dimension. Default is 768. TYPE: int DEFAULT: 768
num_hidden_layers	The number of hidden layers. Default is 12. TYPE: int DEFAULT: 12
num_attention_heads	The number of attention heads. Default is 8. TYPE: int DEFAULT: 8
max_position_embeddings	The maximum position embeddings. Default is 512. TYPE: int DEFAULT: 512
hidden_act	The activation function for the hidden layers. Default is 'gelu'. TYPE: str DEFAULT: 'gelu'
layer_norm_eps	The epsilon value for layer normalization. Default is 1e-12. TYPE: float DEFAULT: 1e-12
hidden_dropout_prob	The dropout probability for the hidden layers. Default is 0.0. TYPE: float DEFAULT: 0.0
attention_probs_dropout_prob	The dropout probability for attention probabilities. Default is 0.0. TYPE: float DEFAULT: 0.0
initializer_range	The range for weight initialization. Default is 0.02. TYPE: float DEFAULT: 0.02
bos_token_id	The ID of the beginning of sentence token. Default is 30522. TYPE: int DEFAULT: 30522
eos_token_id	The ID of the end of sentence token. Default is 2. TYPE: int DEFAULT: 2
pad_token_id	The ID of the padding token. Default is 0. TYPE: int DEFAULT: 0
sep_token_id	The ID of the separator token. Default is 102. TYPE: int DEFAULT: 102
is_decoder	Whether the model is a decoder. Default is True. TYPE: bool DEFAULT: True
use_cache	Whether to use cache for faster decoding. Default is True. TYPE: bool DEFAULT: True
label_smoothing	The label smoothing factor. Default is 0.0. TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/blip/configuration_blip.py

def __init__(
    self,
    vocab_size=30524,
    hidden_size=768,
    encoder_hidden_size=768,
    intermediate_size=3072,
    projection_dim=768,
    num_hidden_layers=12,
    num_attention_heads=8,
    max_position_embeddings=512,
    hidden_act="gelu",
    layer_norm_eps=1e-12,
    hidden_dropout_prob=0.0,
    attention_probs_dropout_prob=0.0,
    initializer_range=0.02,
    bos_token_id=30522,
    eos_token_id=2,
    pad_token_id=0,
    sep_token_id=102,
    is_decoder=True,
    use_cache=True,
    label_smoothing=0.0,
    **kwargs,
):
    """
    Initializes a BlipTextConfig object with the given parameters.

    Args:
        self: The BlipTextConfig instance.
        vocab_size (int, optional): The size of the vocabulary. Default is 30524.
        hidden_size (int, optional): The size of the hidden layers. Default is 768.
        encoder_hidden_size (int, optional): The size of the encoder hidden layers. Default is 768.
        intermediate_size (int, optional): The size of the intermediate layers. Default is 3072.
        projection_dim (int, optional): The projection dimension. Default is 768.
        num_hidden_layers (int, optional): The number of hidden layers. Default is 12.
        num_attention_heads (int, optional): The number of attention heads. Default is 8.
        max_position_embeddings (int, optional): The maximum position embeddings. Default is 512.
        hidden_act (str, optional): The activation function for the hidden layers. Default is 'gelu'.
        layer_norm_eps (float, optional): The epsilon value for layer normalization. Default is 1e-12.
        hidden_dropout_prob (float, optional): The dropout probability for the hidden layers. Default is 0.0.
        attention_probs_dropout_prob (float, optional): The dropout probability for attention probabilities. Default is 0.0.
        initializer_range (float, optional): The range for weight initialization. Default is 0.02.
        bos_token_id (int, optional): The ID of the beginning of sentence token. Default is 30522.
        eos_token_id (int, optional): The ID of the end of sentence token. Default is 2.
        pad_token_id (int, optional): The ID of the padding token. Default is 0.
        sep_token_id (int, optional): The ID of the separator token. Default is 102.
        is_decoder (bool, optional): Whether the model is a decoder. Default is True.
        use_cache (bool, optional): Whether to use cache for faster decoding. Default is True.
        label_smoothing (float, optional): The label smoothing factor. Default is 0.0.

    Returns:
        None

    Raises:
        None
    """
    super().__init__(
        pad_token_id=pad_token_id,
        bos_token_id=bos_token_id,
        eos_token_id=eos_token_id,
        sep_token_id=sep_token_id,
        **kwargs,
    )

    self.vocab_size = vocab_size
    self.hidden_size = hidden_size
    self.encoder_hidden_size = encoder_hidden_size
    self.intermediate_size = intermediate_size
    self.projection_dim = projection_dim
    self.hidden_dropout_prob = hidden_dropout_prob
    self.num_hidden_layers = num_hidden_layers
    self.num_attention_heads = num_attention_heads
    self.max_position_embeddings = max_position_embeddings
    self.layer_norm_eps = layer_norm_eps
    self.hidden_act = hidden_act
    self.initializer_range = initializer_range
    self.attention_probs_dropout_prob = attention_probs_dropout_prob
    self.is_decoder = is_decoder
    self.use_cache = use_cache
    self.label_smoothing = label_smoothing

mindnlp.transformers.models.blip.configuration_blip.BlipTextConfig.from_pretrained(pretrained_model_name_or_path, **kwargs) classmethod

This method 'from_pretrained' in the class 'BlipTextConfig' is used to instantiate a model configuration object based on a pretrained model name or path.

PARAMETER	DESCRIPTION
cls	The class object itself. TYPE: class
pretrained_model_name_or_path	A string representing the name of a pretrained model or a valid path to a pretrained model configuration file. This parameter is mandatory and required for initializing the configuration object. TYPE: Union[str, PathLike]

RETURNS	DESCRIPTION
PretrainedConfig	An instance of 'PretrainedConfig' class representing the configuration settings of the pretrained model. The method returns the configuration object based on the provided pretrained model name or path. TYPE: PretrainedConfig

RAISES	DESCRIPTION
TypeError	If the provided 'pretrained_model_name_or_path' is not a string or a valid path-like object.
KeyError	If the 'model_type' key is missing from the configuration dictionary.
Warning	If the model type in the configuration dictionary does not match the class model type, a warning message is logged as this may lead to errors during instantiation.

Source code in mindnlp/transformers/models/blip/configuration_blip.py

@classmethod
def from_pretrained(cls, pretrained_model_name_or_path: Union[str, os.PathLike], **kwargs) -> "PretrainedConfig":
    """
    This method 'from_pretrained' in the class 'BlipTextConfig' is used to instantiate a model configuration object
    based on a pretrained model name or path.

    Args:
        cls (class): The class object itself.
        pretrained_model_name_or_path (Union[str, os.PathLike]): A string representing the name of a pretrained model
            or a valid path to a pretrained model configuration file. This parameter is mandatory and required for
            initializing the configuration object.

    Returns:
        PretrainedConfig: An instance of 'PretrainedConfig' class representing the configuration settings of
            the pretrained model.
            The method returns the configuration object based on the provided pretrained model name or path.

    Raises:
        TypeError: If the provided 'pretrained_model_name_or_path' is not a string or a valid path-like object.
        KeyError: If the 'model_type' key is missing from the configuration dictionary.
        Warning: If the model type in the configuration dictionary does not match the class model type,
            a warning message is logged as this may lead to errors during instantiation.
    """
    config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)

    # get the text config dict if we are loading from BlipConfig
    if config_dict.get("model_type") == "blip":
        config_dict = config_dict["text_config"]

    if "model_type" in config_dict and hasattr(cls, "model_type") and config_dict["model_type"] != cls.model_type:
        logger.warning(
            f"You are using a model of type {config_dict['model_type']} to instantiate a model of type "
            f"{cls.model_type}. This is not supported for all configurations of models and can yield errors."
        )

    return cls.from_dict(config_dict, **kwargs)

mindnlp.transformers.models.blip.modeling_blip.BlipModel

Bases: BlipPreTrainedModel

BlipModel

BlipModel is a class that represents a multimodal model for processing both text and images. It inherits from BlipPreTrainedModel and includes methods for obtaining text and image features, as well as for forwarding the model output.

Example

>>> from transformers import AutoProcessor, BlipModel
>>> model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base")
>>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
>>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")
>>> text_features = model.get_text_features(**inputs)
...
>>> from PIL import Image
>>> import requests
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(images=image, return_tensors="pt")
>>> image_features = model.get_image_features(**inputs)
...
>>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True)
>>> outputs = model(**inputs)
>>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score
>>> probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities

Source code in mindnlp/transformers/models/blip/modeling_blip.py

class BlipModel(BlipPreTrainedModel):

    """
    BlipModel

    BlipModel is a class that represents a multimodal model for processing both text and images.
    It inherits from BlipPreTrainedModel and includes methods for obtaining text and image features, as well as for
    forwarding the model output.

    Example:
        ```python
        >>> from transformers import AutoProcessor, BlipModel
        >>> model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base")
        >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
        >>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")
        >>> text_features = model.get_text_features(**inputs)
        ...
        >>> from PIL import Image
        >>> import requests
        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
        >>> image = Image.open(requests.get(url, stream=True).raw)
        >>> inputs = processor(images=image, return_tensors="pt")
        >>> image_features = model.get_image_features(**inputs)
        ...
        >>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True)
        >>> outputs = model(**inputs)
        >>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score
        >>> probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities
        ```
    """
    config_class = BlipConfig

    def __init__(self, config: BlipConfig):
        """
        Initializes an instance of the BlipModel class.

        Args:
            self: The current instance of the BlipModel class.
            config (BlipConfig): The configuration object for the BlipModel.
                It should contain the following attributes:

                - text_config (BlipTextConfig): The configuration object for the text model component of BlipModel.
                It should be of type BlipTextConfig and contain the necessary parameters for the text model.
                - vision_config (BlipVisionConfig): The configuration object for the vision model component of BlipModel.
                It should be of type BlipVisionConfig and contain the necessary parameters for the vision model.
                - projection_dim (int): The dimension of the projection space.
                - logit_scale_init_value (float): The initial value for the logit scale parameter.

        Returns:
            None

        Raises:
            ValueError: If config.text_config is not of type BlipTextConfig.
            ValueError: If config.vision_config is not of type BlipVisionConfig.

        Note:
            This method initializes the BlipModel instance by setting the projection dimension, text embedding dimension,
            vision embedding dimension, text model, vision model, visual projection, text projection, and logit scale
            attributes based on the provided configuration. It also calls the post_init method.
        """
        super().__init__(config)

        if not isinstance(config.text_config, BlipTextConfig):
            raise ValueError(
                "config.text_config is expected to be of type BlipTextConfig but is of type"
                f" {type(config.text_config)}."
            )

        if not isinstance(config.vision_config, BlipVisionConfig):
            raise ValueError(
                "config.vision_config is expected to be of type BlipVisionConfig but is of type"
                f" {type(config.vision_config)}."
            )

        text_config = config.text_config
        vision_config = config.vision_config

        self.projection_dim = config.projection_dim
        self.text_embed_dim = text_config.hidden_size
        self.vision_embed_dim = vision_config.hidden_size

        self.text_model = BlipTextModel(text_config)
        self.vision_model = BlipVisionModel(vision_config)

        self.visual_projection = nn.Linear(self.vision_embed_dim, self.projection_dim, bias=False)
        self.text_projection = nn.Linear(self.text_embed_dim, self.projection_dim, bias=False)
        self.logit_scale = Parameter(mindspore.tensor(self.config.logit_scale_init_value))

        # Initialize weights and apply final processing
        self.post_init()

    def get_text_features(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        position_ids: Optional[mindspore.Tensor] = None,
        return_dict: Optional[bool] = None,
    ) -> mindspore.Tensor:
        r"""
        Returns:
            text_features (`mindspore.Tensor` of shape `(batch_size, output_dim`): The text embeddings obtained by
            applying the projection layer to the pooled output of [`BlipTextModel`].

        Example:
            ```python
            >>> from transformers import AutoProcessor, BlipModel
            ...
            >>> model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base")
            >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
            ...
            >>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")
            >>> text_features = model.get_text_features(**inputs)
            ```
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        text_outputs = self.text_model(
            input_ids=input_ids,
            attention_mask=attention_mask,
            position_ids=position_ids,
            return_dict=return_dict,
        )

        pooled_output = text_outputs[1]
        text_features = self.text_projection(pooled_output)

        return text_features

    def get_image_features(
        self,
        pixel_values: Optional[mindspore.Tensor] = None,
        return_dict: Optional[bool] = None,
    ) -> mindspore.Tensor:
        r"""
        Returns:
            image_features (`mindspore.Tensor` of shape `(batch_size, output_dim`): The image embeddings obtained by
                applying the projection layer to the pooled output of [`BlipVisionModel`].

        Example:
            ```python
            >>> from PIL import Image
            >>> import requests
            >>> from transformers import AutoProcessor, BlipModel
            ...
            >>> model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base")
            >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
            ...
            >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
            >>> image = Image.open(requests.get(url, stream=True).raw)
            ...
            >>> inputs = processor(images=image, return_tensors="pt")
            ...
            >>> image_features = model.get_image_features(**inputs)
            ```
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        vision_outputs = self.vision_model(pixel_values=pixel_values, return_dict=return_dict)

        pooled_output = vision_outputs[1]  # pooled_output
        image_features = self.visual_projection(pooled_output)

        return image_features

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        pixel_values: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        position_ids: Optional[mindspore.Tensor] = None,
        return_loss: Optional[bool] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, BlipOutput]:
        r"""
        Returns:
            `Union[Tuple, BlipOutput]`

        Example:
            ```python
            >>> from PIL import Image
            >>> import requests
            >>> from transformers import AutoProcessor, BlipModel
            ...
            >>> model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base")
            >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
            ...
            >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
            >>> image = Image.open(requests.get(url, stream=True).raw)
            ...
            >>> inputs = processor(
            ...     text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True
            ... )
            ...
            >>> outputs = model(**inputs)
            >>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score
            >>> probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities
            ```
        """
        # Use BLIP model's config for some fields (if specified) instead of those of vision & text components.
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        vision_outputs = self.vision_model(
            pixel_values=pixel_values,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        text_outputs = self.text_model(
            input_ids=input_ids,
            attention_mask=attention_mask,
            position_ids=position_ids,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        image_embeds = vision_outputs[1]
        image_embeds = self.visual_projection(image_embeds)

        text_embeds = text_outputs[1]
        text_embeds = self.text_projection(text_embeds)

        # normalized features
        image_embeds = image_embeds / image_embeds.norm(ord=2, dim=-1, keepdim=True)
        text_embeds = text_embeds / text_embeds.norm(ord=2, dim=-1, keepdim=True)

        # cosine similarity as logits
        logit_scale = self.logit_scale.exp()
        logits_per_text = ops.matmul(text_embeds, image_embeds.t()) * logit_scale
        logits_per_image = logits_per_text.t()

        loss = None
        if return_loss:
            loss = blip_loss(logits_per_text)

        if not return_dict:
            output = (logits_per_image, logits_per_text, text_embeds, image_embeds, text_outputs, vision_outputs)
            return ((loss,) + output) if loss is not None else output

        return BlipOutput(
            loss=loss,
            logits_per_image=logits_per_image,
            logits_per_text=logits_per_text,
            text_embeds=text_embeds,
            image_embeds=image_embeds,
            text_model_output=text_outputs,
            vision_model_output=vision_outputs,
        )

mindnlp.transformers.models.blip.modeling_blip.BlipModel.__init__(config)

Initializes an instance of the BlipModel class.

PARAMETER	DESCRIPTION
self	The current instance of the BlipModel class.
config	The configuration object for the BlipModel. It should contain the following attributes: text_config (BlipTextConfig): The configuration object for the text model component of BlipModel. It should be of type BlipTextConfig and contain the necessary parameters for the text model. vision_config (BlipVisionConfig): The configuration object for the vision model component of BlipModel. It should be of type BlipVisionConfig and contain the necessary parameters for the vision model. projection_dim (int): The dimension of the projection space. logit_scale_init_value (float): The initial value for the logit scale parameter. TYPE: BlipConfig

RETURNS	DESCRIPTION
	None

RAISES	DESCRIPTION
ValueError	If config.text_config is not of type BlipTextConfig.
ValueError	If config.vision_config is not of type BlipVisionConfig.

Note

This method initializes the BlipModel instance by setting the projection dimension, text embedding dimension, vision embedding dimension, text model, vision model, visual projection, text projection, and logit scale attributes based on the provided configuration. It also calls the post_init method.

Source code in mindnlp/transformers/models/blip/modeling_blip.py

def __init__(self, config: BlipConfig):
    """
    Initializes an instance of the BlipModel class.

    Args:
        self: The current instance of the BlipModel class.
        config (BlipConfig): The configuration object for the BlipModel.
            It should contain the following attributes:

            - text_config (BlipTextConfig): The configuration object for the text model component of BlipModel.
            It should be of type BlipTextConfig and contain the necessary parameters for the text model.
            - vision_config (BlipVisionConfig): The configuration object for the vision model component of BlipModel.
            It should be of type BlipVisionConfig and contain the necessary parameters for the vision model.
            - projection_dim (int): The dimension of the projection space.
            - logit_scale_init_value (float): The initial value for the logit scale parameter.

    Returns:
        None

    Raises:
        ValueError: If config.text_config is not of type BlipTextConfig.
        ValueError: If config.vision_config is not of type BlipVisionConfig.

    Note:
        This method initializes the BlipModel instance by setting the projection dimension, text embedding dimension,
        vision embedding dimension, text model, vision model, visual projection, text projection, and logit scale
        attributes based on the provided configuration. It also calls the post_init method.
    """
    super().__init__(config)

    if not isinstance(config.text_config, BlipTextConfig):
        raise ValueError(
            "config.text_config is expected to be of type BlipTextConfig but is of type"
            f" {type(config.text_config)}."
        )

    if not isinstance(config.vision_config, BlipVisionConfig):
        raise ValueError(
            "config.vision_config is expected to be of type BlipVisionConfig but is of type"
            f" {type(config.vision_config)}."
        )

    text_config = config.text_config
    vision_config = config.vision_config

    self.projection_dim = config.projection_dim
    self.text_embed_dim = text_config.hidden_size
    self.vision_embed_dim = vision_config.hidden_size

    self.text_model = BlipTextModel(text_config)
    self.vision_model = BlipVisionModel(vision_config)

    self.visual_projection = nn.Linear(self.vision_embed_dim, self.projection_dim, bias=False)
    self.text_projection = nn.Linear(self.text_embed_dim, self.projection_dim, bias=False)
    self.logit_scale = Parameter(mindspore.tensor(self.config.logit_scale_init_value))

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.blip.modeling_blip.BlipModel.forward(input_ids=None, pixel_values=None, attention_mask=None, position_ids=None, return_loss=None, output_attentions=None, output_hidden_states=None, return_dict=None)

RETURNS	DESCRIPTION
Union[Tuple, BlipOutput]	Union[Tuple, BlipOutput]

Example

>>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, BlipModel
...
>>> model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base")
>>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
...
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
...
>>> inputs = processor(
...     text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True
... )
...
>>> outputs = model(**inputs)
>>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score
>>> probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities

Source code in mindnlp/transformers/models/blip/modeling_blip.py

def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    pixel_values: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    position_ids: Optional[mindspore.Tensor] = None,
    return_loss: Optional[bool] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple, BlipOutput]:
    r"""
    Returns:
        `Union[Tuple, BlipOutput]`

    Example:
        ```python
        >>> from PIL import Image
        >>> import requests
        >>> from transformers import AutoProcessor, BlipModel
        ...
        >>> model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base")
        >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
        ...
        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
        >>> image = Image.open(requests.get(url, stream=True).raw)
        ...
        >>> inputs = processor(
        ...     text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True
        ... )
        ...
        >>> outputs = model(**inputs)
        >>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score
        >>> probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities
        ```
    """
    # Use BLIP model's config for some fields (if specified) instead of those of vision & text components.
    output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
    output_hidden_states = (
        output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
    )
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    vision_outputs = self.vision_model(
        pixel_values=pixel_values,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    text_outputs = self.text_model(
        input_ids=input_ids,
        attention_mask=attention_mask,
        position_ids=position_ids,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    image_embeds = vision_outputs[1]
    image_embeds = self.visual_projection(image_embeds)

    text_embeds = text_outputs[1]
    text_embeds = self.text_projection(text_embeds)

    # normalized features
    image_embeds = image_embeds / image_embeds.norm(ord=2, dim=-1, keepdim=True)
    text_embeds = text_embeds / text_embeds.norm(ord=2, dim=-1, keepdim=True)

    # cosine similarity as logits
    logit_scale = self.logit_scale.exp()
    logits_per_text = ops.matmul(text_embeds, image_embeds.t()) * logit_scale
    logits_per_image = logits_per_text.t()

    loss = None
    if return_loss:
        loss = blip_loss(logits_per_text)

    if not return_dict:
        output = (logits_per_image, logits_per_text, text_embeds, image_embeds, text_outputs, vision_outputs)
        return ((loss,) + output) if loss is not None else output

    return BlipOutput(
        loss=loss,
        logits_per_image=logits_per_image,
        logits_per_text=logits_per_text,
        text_embeds=text_embeds,
        image_embeds=image_embeds,
        text_model_output=text_outputs,
        vision_model_output=vision_outputs,
    )

mindnlp.transformers.models.blip.modeling_blip.BlipModel.get_image_features(pixel_values=None, return_dict=None)

RETURNS	DESCRIPTION
image_features	The image embeddings obtained by applying the projection layer to the pooled output of [BlipVisionModel]. TYPE: `mindspore.Tensor` of shape `(batch_size, output_dim`

Example

>>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, BlipModel
...
>>> model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base")
>>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
...
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
...
>>> inputs = processor(images=image, return_tensors="pt")
...
>>> image_features = model.get_image_features(**inputs)

Source code in mindnlp/transformers/models/blip/modeling_blip.py

def get_image_features(
    self,
    pixel_values: Optional[mindspore.Tensor] = None,
    return_dict: Optional[bool] = None,
) -> mindspore.Tensor:
    r"""
    Returns:
        image_features (`mindspore.Tensor` of shape `(batch_size, output_dim`): The image embeddings obtained by
            applying the projection layer to the pooled output of [`BlipVisionModel`].

    Example:
        ```python
        >>> from PIL import Image
        >>> import requests
        >>> from transformers import AutoProcessor, BlipModel
        ...
        >>> model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base")
        >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
        ...
        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
        >>> image = Image.open(requests.get(url, stream=True).raw)
        ...
        >>> inputs = processor(images=image, return_tensors="pt")
        ...
        >>> image_features = model.get_image_features(**inputs)
        ```
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    vision_outputs = self.vision_model(pixel_values=pixel_values, return_dict=return_dict)

    pooled_output = vision_outputs[1]  # pooled_output
    image_features = self.visual_projection(pooled_output)

    return image_features

mindnlp.transformers.models.blip.modeling_blip.BlipModel.get_text_features(input_ids=None, attention_mask=None, position_ids=None, return_dict=None)

RETURNS	DESCRIPTION
text_features	The text embeddings obtained by TYPE: `mindspore.Tensor` of shape `(batch_size, output_dim`
Tensor	applying the projection layer to the pooled output of [BlipTextModel].

Example

>>> from transformers import AutoProcessor, BlipModel
...
>>> model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base")
>>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
...
>>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")
>>> text_features = model.get_text_features(**inputs)

Source code in mindnlp/transformers/models/blip/modeling_blip.py

def get_text_features(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    position_ids: Optional[mindspore.Tensor] = None,
    return_dict: Optional[bool] = None,
) -> mindspore.Tensor:
    r"""
    Returns:
        text_features (`mindspore.Tensor` of shape `(batch_size, output_dim`): The text embeddings obtained by
        applying the projection layer to the pooled output of [`BlipTextModel`].

    Example:
        ```python
        >>> from transformers import AutoProcessor, BlipModel
        ...
        >>> model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base")
        >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
        ...
        >>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")
        >>> text_features = model.get_text_features(**inputs)
        ```
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    text_outputs = self.text_model(
        input_ids=input_ids,
        attention_mask=attention_mask,
        position_ids=position_ids,
        return_dict=return_dict,
    )

    pooled_output = text_outputs[1]
    text_features = self.text_projection(pooled_output)

    return text_features

mindnlp.transformers.models.blip.modeling_blip.BlipPreTrainedModel

Bases: PreTrainedModel

An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models.

Source code in mindnlp/transformers/models/blip/modeling_blip.py

class BlipPreTrainedModel(PreTrainedModel):
    """
    An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained
    models.
    """
    config_class = BlipConfig
    base_model_prefix = "blip"
    supports_gradient_checkpointing = True

    def _init_weights(self, cell):
        """Initialize the weights"""
        factor = self.config.initializer_range
        if isinstance(cell, (nn.Conv2d, nn.Linear, nn.Embedding)):
            cell.weight.set_data(initializer(Normal(factor), cell.weight.shape, cell.weight.dtype))
            if hasattr(cell, "bias") and cell.bias is not None:
                cell.bias.set_data(initializer('zeros', cell.bias.shape, cell.bias.dtype))

        if isinstance(cell, BlipVisionEmbeddings):
            if hasattr(self.config, "vision_config"):
                factor = self.config.vision_config.initializer_range

            cell.position_embedding.set_data(initializer(TruncatedNormal(factor), cell.position_embedding.shape, cell.position_embedding.dtype))
            cell.class_embedding.set_data(initializer(TruncatedNormal(factor), cell.class_embedding.shape, cell.class_embedding.dtype))

        elif isinstance(cell, nn.LayerNorm):
            cell.weight.set_data(initializer('ones', cell.weight.shape, cell.weight.dtype))
            cell.bias.set_data(initializer('zeros', cell.bias.shape, cell.bias.dtype))

        elif isinstance(cell, nn.Linear) and cell.bias is not None:
            cell.bias.set_data(initializer('zeros', cell.bias.shape, cell.bias.dtype))

mindnlp.transformers.models.blip.modeling_blip.BlipForConditionalGeneration

Bases: BlipPreTrainedModel

A class representing the BlipForConditionalGeneration model for image captioning.

This class extends the BlipPreTrainedModel class and provides methods for initializing the model, generating image captions, and forwarding the model's architecture.

ATTRIBUTE	DESCRIPTION
vision_model	The vision model used for extracting image features. TYPE: BlipVisionModel
text_decoder	The text decoder model used for generating captions. TYPE: BlipTextLMHeadModel
decoder_input_ids	The token ID to start the decoder input sequence. TYPE: int
decoder_pad_token_id	The token ID used for padding the decoder input sequence. TYPE: int

METHOD	DESCRIPTION
__init__	Initializes the BlipForConditionalGeneration model.
get_input_embeddings	Returns the input embeddings of the vision model.
forward	Constructs the model architecture and generates image captions.
generate	Generates image captions based on the input image.

Example

>>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, BlipForConditionalGeneration
...
>>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
>>> model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")
...
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> text = "A picture of"
...
>>> inputs = processor(images=image, text=text, return_tensors="pt")
...
>>> outputs = model(**inputs)

Overrides

Source code in mindnlp/transformers/models/blip/modeling_blip.py

class BlipForConditionalGeneration(BlipPreTrainedModel):

    """
    A class representing the BlipForConditionalGeneration model for image captioning.

    This class extends the BlipPreTrainedModel class and provides methods for initializing the model,
    generating image captions, and forwarding the model's architecture.

    Attributes:
        vision_model (BlipVisionModel): The vision model used for extracting image features.
        text_decoder (BlipTextLMHeadModel): The text decoder model used for generating captions.
        decoder_input_ids (int): The token ID to start the decoder input sequence.
        decoder_pad_token_id (int): The token ID used for padding the decoder input sequence.

    Methods:
        __init__: Initializes the BlipForConditionalGeneration model.
        get_input_embeddings: Returns the input embeddings of the vision model.
        forward: Constructs the model architecture and generates image captions.
        generate: Generates image captions based on the input image.

    Example:
        ```python
        >>> from PIL import Image
        >>> import requests
        >>> from transformers import AutoProcessor, BlipForConditionalGeneration
        ...
        >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
        >>> model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")
        ...
        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
        >>> image = Image.open(requests.get(url, stream=True).raw)
        >>> text = "A picture of"
        ...
        >>> inputs = processor(images=image, text=text, return_tensors="pt")
        ...
        >>> outputs = model(**inputs)
        ```

    Overrides:
        generate: Overrides the generate function to enable the model to be used as a conditional generator.

    """
    config_class = BlipConfig
    _tied_weights_keys = ["text_decoder.cls.predictions.decoder.bias"]
    main_input_name = "pixel_values"

    def __init__(self, config: BlipConfig):
        """
        Initializes an instance of the BlipForConditionalGeneration class.

        Args:
            self (BlipForConditionalGeneration): The instance of the BlipForConditionalGeneration class.
            config (BlipConfig): An object representing the configuration settings for the Blip model.
                It contains the necessary configurations for the vision model and text decoder.
                It is expected that the config parameter is of type BlipConfig.

        Returns:
            None.

        Raises:
            TypeError: If the config parameter is not of type BlipConfig.
            ValueError: If the config parameter is missing required configuration settings.
        """
        super().__init__(config)

        self.vision_model = BlipVisionModel(config.vision_config)

        self.text_decoder = BlipTextLMHeadModel(config.text_config)

        self.decoder_input_ids = config.text_config.bos_token_id
        self.decoder_pad_token_id = config.text_config.pad_token_id

        # Initialize weights and apply final processing
        self.post_init()

    def get_input_embeddings(self) -> nn.Module:
        """
        This method returns the input embeddings for the BlipForConditionalGeneration class.

        Args:
            self (BlipForConditionalGeneration): The instance of the BlipForConditionalGeneration class.

        Returns:
            nn.Module: The input embeddings for the BlipForConditionalGeneration class. This is an instance of the nn.Module class.

        Raises:
            None.

        """
        return self.vision_model.embeddings.patch_embedding

    def forward(
        self,
        pixel_values: mindspore.Tensor,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        labels: Optional[mindspore.Tensor] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, BlipForConditionalGenerationModelOutput]:
        r"""
        Returns:
            Union[Tuple, BlipForConditionalGenerationModelOutput]

        Example:
            ```python
            >>> from PIL import Image
            >>> import requests
            >>> from transformers import AutoProcessor, BlipForConditionalGeneration
            ...
            >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
            >>> model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")
            ...
            >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
            >>> image = Image.open(requests.get(url, stream=True).raw)
            >>> text = "A picture of"
            ...
            >>> inputs = processor(images=image, text=text, return_tensors="pt")
            ...
            >>> outputs = model(**inputs)
            ```
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )

        vision_outputs = self.vision_model(
            pixel_values=pixel_values,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        image_embeds = vision_outputs[0]

        outputs = self.text_decoder(
            input_ids=input_ids,
            attention_mask=attention_mask,
            encoder_hidden_states=image_embeds,
            labels=labels,
            return_dict=return_dict,
            reduction="mean",
        )

        if not return_dict:
            outputs = (outputs[0], outputs[1], image_embeds, vision_outputs[0]) + vision_outputs[2:]
            return tuple(output for output in outputs if output is not None)

        return BlipForConditionalGenerationModelOutput(
            loss=outputs.loss,
            logits=outputs.logits,
            image_embeds=image_embeds,
            last_hidden_state=vision_outputs.last_hidden_state,
            hidden_states=vision_outputs.hidden_states,
            attentions=vision_outputs.attentions,
        )

    def generate(
        self,
        pixel_values: mindspore.Tensor,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        **generate_kwargs,
    ) -> mindspore.Tensor:
        r"""
        Overrides *generate* function to be able to use the model as a conditional generator

        Parameters:
            pixel_values (*mindspore.Tensor* of shape *(batch_size, num_channels, image_height, image_width)*:
                Input image to be processed
            input_ids (*mindspore.Tensor* of shape *(batch_size, sequence_length)*, *optional*):
                The sequence used as a prompt for the generation.
            attention_mask (*mindspore.Tensor* of shape *(batch_size, sequence_length)*, *optional*):
                Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:

        Example:
            ```python
            >>> from PIL import Image
            >>> import requests
            >>> from transformers import AutoProcessor, BlipForConditionalGeneration
            ...
            >>> model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")
            >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
            ...
            >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
            >>> image = Image.open(requests.get(url, stream=True).raw)
            ...
            >>> inputs = processor(images=image, return_tensors="pt")
            ...
            >>> outputs = model.generate(**inputs)
            >>> print(processor.decode(outputs[0], skip_special_tokens=True))
            two cats sleeping on a couch
            ```
        """
        batch_size = pixel_values.shape[0]
        vision_outputs = self.vision_model(pixel_values=pixel_values)

        image_embeds = vision_outputs[0]

        image_attention_mask = ops.ones(*image_embeds.shape[:-1], dtype=mindspore.int64)

        if isinstance(input_ids, list):
            input_ids = mindspore.Tensor(input_ids)
        elif input_ids is None:
            input_ids = (
                ops.tile((mindspore.Tensor([[self.decoder_input_ids, self.config.text_config.eos_token_id]])), (batch_size, 1))
            )

        input_ids[:, 0] = self.config.text_config.bos_token_id
        attention_mask = attention_mask[:, :-1] if attention_mask is not None else None

        outputs = self.text_decoder.generate(
            input_ids=input_ids[:, :-1],
            eos_token_id=self.config.text_config.sep_token_id,
            pad_token_id=self.config.text_config.pad_token_id,
            attention_mask=attention_mask,
            encoder_hidden_states=image_embeds,
            encoder_attention_mask=image_attention_mask,
            **generate_kwargs,
        )

        return outputs

mindnlp.transformers.models.blip.modeling_blip.BlipForConditionalGeneration.__init__(config)

Initializes an instance of the BlipForConditionalGeneration class.

PARAMETER	DESCRIPTION
self	The instance of the BlipForConditionalGeneration class. TYPE: BlipForConditionalGeneration
config	An object representing the configuration settings for the Blip model. It contains the necessary configurations for the vision model and text decoder. It is expected that the config parameter is of type BlipConfig. TYPE: BlipConfig

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
TypeError	If the config parameter is not of type BlipConfig.
ValueError	If the config parameter is missing required configuration settings.

Source code in mindnlp/transformers/models/blip/modeling_blip.py

def __init__(self, config: BlipConfig):
    """
    Initializes an instance of the BlipForConditionalGeneration class.

    Args:
        self (BlipForConditionalGeneration): The instance of the BlipForConditionalGeneration class.
        config (BlipConfig): An object representing the configuration settings for the Blip model.
            It contains the necessary configurations for the vision model and text decoder.
            It is expected that the config parameter is of type BlipConfig.

    Returns:
        None.

    Raises:
        TypeError: If the config parameter is not of type BlipConfig.
        ValueError: If the config parameter is missing required configuration settings.
    """
    super().__init__(config)

    self.vision_model = BlipVisionModel(config.vision_config)

    self.text_decoder = BlipTextLMHeadModel(config.text_config)

    self.decoder_input_ids = config.text_config.bos_token_id
    self.decoder_pad_token_id = config.text_config.pad_token_id

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.blip.modeling_blip.BlipForConditionalGeneration.forward(pixel_values, input_ids=None, attention_mask=None, output_attentions=None, output_hidden_states=None, labels=None, return_dict=None)

RETURNS	DESCRIPTION
Union[Tuple, BlipForConditionalGenerationModelOutput]	Union[Tuple, BlipForConditionalGenerationModelOutput]

Example

>>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, BlipForConditionalGeneration
...
>>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
>>> model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")
...
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> text = "A picture of"
...
>>> inputs = processor(images=image, text=text, return_tensors="pt")
...
>>> outputs = model(**inputs)

Source code in mindnlp/transformers/models/blip/modeling_blip.py

def forward(
    self,
    pixel_values: mindspore.Tensor,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    labels: Optional[mindspore.Tensor] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple, BlipForConditionalGenerationModelOutput]:
    r"""
    Returns:
        Union[Tuple, BlipForConditionalGenerationModelOutput]

    Example:
        ```python
        >>> from PIL import Image
        >>> import requests
        >>> from transformers import AutoProcessor, BlipForConditionalGeneration
        ...
        >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
        >>> model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")
        ...
        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
        >>> image = Image.open(requests.get(url, stream=True).raw)
        >>> text = "A picture of"
        ...
        >>> inputs = processor(images=image, text=text, return_tensors="pt")
        ...
        >>> outputs = model(**inputs)
        ```
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
    output_hidden_states = (
        output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
    )

    vision_outputs = self.vision_model(
        pixel_values=pixel_values,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    image_embeds = vision_outputs[0]

    outputs = self.text_decoder(
        input_ids=input_ids,
        attention_mask=attention_mask,
        encoder_hidden_states=image_embeds,
        labels=labels,
        return_dict=return_dict,
        reduction="mean",
    )

    if not return_dict:
        outputs = (outputs[0], outputs[1], image_embeds, vision_outputs[0]) + vision_outputs[2:]
        return tuple(output for output in outputs if output is not None)

    return BlipForConditionalGenerationModelOutput(
        loss=outputs.loss,
        logits=outputs.logits,
        image_embeds=image_embeds,
        last_hidden_state=vision_outputs.last_hidden_state,
        hidden_states=vision_outputs.hidden_states,
        attentions=vision_outputs.attentions,
    )

mindnlp.transformers.models.blip.modeling_blip.BlipForConditionalGeneration.generate(pixel_values, input_ids=None, attention_mask=None, **generate_kwargs)

Overrides generate function to be able to use the model as a conditional generator

PARAMETER	DESCRIPTION
pixel_values	Input image to be processed TYPE: *mindspore.Tensor* of shape *(batch_size, num_channels, image_height, image_width)*
input_ids	The sequence used as a prompt for the generation. TYPE: *mindspore.Tensor* of shape *(batch_size, sequence_length)*, *optional* DEFAULT: None
attention_mask	Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]: TYPE: *mindspore.Tensor* of shape *(batch_size, sequence_length)*, *optional* DEFAULT: None

Example

>>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, BlipForConditionalGeneration
...
>>> model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")
>>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
...
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
...
>>> inputs = processor(images=image, return_tensors="pt")
...
>>> outputs = model.generate(**inputs)
>>> print(processor.decode(outputs[0], skip_special_tokens=True))
two cats sleeping on a couch

Source code in mindnlp/transformers/models/blip/modeling_blip.py

def generate(
    self,
    pixel_values: mindspore.Tensor,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    **generate_kwargs,
) -> mindspore.Tensor:
    r"""
    Overrides *generate* function to be able to use the model as a conditional generator

    Parameters:
        pixel_values (*mindspore.Tensor* of shape *(batch_size, num_channels, image_height, image_width)*:
            Input image to be processed
        input_ids (*mindspore.Tensor* of shape *(batch_size, sequence_length)*, *optional*):
            The sequence used as a prompt for the generation.
        attention_mask (*mindspore.Tensor* of shape *(batch_size, sequence_length)*, *optional*):
            Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:

    Example:
        ```python
        >>> from PIL import Image
        >>> import requests
        >>> from transformers import AutoProcessor, BlipForConditionalGeneration
        ...
        >>> model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")
        >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
        ...
        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
        >>> image = Image.open(requests.get(url, stream=True).raw)
        ...
        >>> inputs = processor(images=image, return_tensors="pt")
        ...
        >>> outputs = model.generate(**inputs)
        >>> print(processor.decode(outputs[0], skip_special_tokens=True))
        two cats sleeping on a couch
        ```
    """
    batch_size = pixel_values.shape[0]
    vision_outputs = self.vision_model(pixel_values=pixel_values)

    image_embeds = vision_outputs[0]

    image_attention_mask = ops.ones(*image_embeds.shape[:-1], dtype=mindspore.int64)

    if isinstance(input_ids, list):
        input_ids = mindspore.Tensor(input_ids)
    elif input_ids is None:
        input_ids = (
            ops.tile((mindspore.Tensor([[self.decoder_input_ids, self.config.text_config.eos_token_id]])), (batch_size, 1))
        )

    input_ids[:, 0] = self.config.text_config.bos_token_id
    attention_mask = attention_mask[:, :-1] if attention_mask is not None else None

    outputs = self.text_decoder.generate(
        input_ids=input_ids[:, :-1],
        eos_token_id=self.config.text_config.sep_token_id,
        pad_token_id=self.config.text_config.pad_token_id,
        attention_mask=attention_mask,
        encoder_hidden_states=image_embeds,
        encoder_attention_mask=image_attention_mask,
        **generate_kwargs,
    )

    return outputs

mindnlp.transformers.models.blip.modeling_blip.BlipForConditionalGeneration.get_input_embeddings()

This method returns the input embeddings for the BlipForConditionalGeneration class.

PARAMETER	DESCRIPTION
self	The instance of the BlipForConditionalGeneration class. TYPE: BlipForConditionalGeneration

RETURNS	DESCRIPTION
Module	nn.Module: The input embeddings for the BlipForConditionalGeneration class. This is an instance of the nn.Module class.

Source code in mindnlp/transformers/models/blip/modeling_blip.py

def get_input_embeddings(self) -> nn.Module:
    """
    This method returns the input embeddings for the BlipForConditionalGeneration class.

    Args:
        self (BlipForConditionalGeneration): The instance of the BlipForConditionalGeneration class.

    Returns:
        nn.Module: The input embeddings for the BlipForConditionalGeneration class. This is an instance of the nn.Module class.

    Raises:
        None.

    """
    return self.vision_model.embeddings.patch_embedding

mindnlp.transformers.models.blip.modeling_blip.BlipForQuestionAnswering

Bases: BlipPreTrainedModel

BlipForQuestionAnswering is a class that represents a model for question answering using both text and vision inputs. It is designed to be used with the BlipPreTrainedModel base class.

This class has the following attributes:

• vision_model: An instance of the BlipVisionModel class that handles the vision inputs.
• text_encoder: An instance of the BlipTextModel class that encodes the text inputs.
• text_decoder: An instance of the BlipTextLMHeadModel class that decodes the text inputs.
• decoder_pad_token_id: The ID of the padding token used in the decoder.
• decoder_start_token_id: The ID of the start token used in the decoder.

The BlipForQuestionAnswering class provides the following methods:

1. init: Initializes the BlipForQuestionAnswering instance with the given configuration.
2. get_input_embeddings: Returns the input embeddings of the vision model.
3. forward: Constructs the model and performs the forward pass. Returns the model outputs.
4. generate: Generates text outputs based on the given input IDs and pixel values.

Please refer to the code examples in the docstring for more information on how to use the BlipForQuestionAnswering class for training and inference.

Note

This documentation is auto-generated and may not capture all the intricacies of the class implementation. For more details, please refer to the source code.

Source code in mindnlp/transformers/models/blip/modeling_blip.py

class BlipForQuestionAnswering(BlipPreTrainedModel):

    """
    BlipForQuestionAnswering is a class that represents a model for question answering using both text and vision inputs. It is designed to be used with the BlipPreTrainedModel base class.

    This class has the following attributes:

    - vision_model: An instance of the BlipVisionModel class that handles the vision inputs.
    - text_encoder: An instance of the BlipTextModel class that encodes the text inputs.
    - text_decoder: An instance of the BlipTextLMHeadModel class that decodes the text inputs.
    - decoder_pad_token_id: The ID of the padding token used in the decoder.
    - decoder_start_token_id: The ID of the start token used in the decoder.

    The BlipForQuestionAnswering class provides the following methods:

    1. __init__:
    Initializes the BlipForQuestionAnswering instance with the given configuration.
    2. get_input_embeddings:
    Returns the input embeddings of the vision model.
    3. forward:
    Constructs the model and performs the forward pass. Returns the model outputs.
    4. generate:
    Generates text outputs based on the given input IDs and pixel values.

    Please refer to the code examples in the docstring for more information on how to use the BlipForQuestionAnswering
    class for training and inference.

    Note:
        This documentation is auto-generated and may not capture all the intricacies of the class implementation.
        For more details, please refer to the source code.
    """
    config_class = BlipConfig
    _tied_weights_keys = ["text_decoder.cls.predictions.decoder.bias"]

    def __init__(self, config: BlipConfig):
        """
        Initializes an instance of BlipForQuestionAnswering.

        Args:
            self: The instance of the class.
            config (BlipConfig): An instance of BlipConfig containing the configuration for the model.

        Returns:
            None.

        Raises:
            None
        """
        super().__init__(config)

        self.vision_model = BlipVisionModel(config.vision_config)

        self.text_encoder = BlipTextModel(config.text_config, add_pooling_layer=False)

        self.text_decoder = BlipTextLMHeadModel(config.text_config)

        self.decoder_pad_token_id = config.text_config.pad_token_id
        self.decoder_start_token_id = config.text_config.bos_token_id

        # Initialize weights and apply final processing
        self.post_init()

    def get_input_embeddings(self) -> nn.Module:
        """
        This method returns the input embeddings from the vision model for question answering.

        Args:
            self (BlipForQuestionAnswering): The instance of the BlipForQuestionAnswering class.

        Returns:
            nn.Module: The input embeddings from the vision model, which is of type nn.Module.

        Raises:
            None
        """
        return self.vision_model.embeddings.patch_embedding

    def forward(
        self,
        input_ids: mindspore.Tensor,
        pixel_values: mindspore.Tensor,
        decoder_input_ids: Optional[mindspore.Tensor] = None,
        decoder_attention_mask: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        labels: Optional[mindspore.Tensor] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, BlipTextVisionModelOutput]:
        r"""
        Returns:
            `Union[Tuple, BlipTextVisionModelOutput]`

        Example:
            ```python
            >>> from PIL import Image
            >>> import requests
            >>> from transformers import AutoProcessor, BlipForQuestionAnswering
            ...
            >>> model = BlipForQuestionAnswering.from_pretrained("Salesforce/blip-vqa-base")
            >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-vqa-base")
            ...
            >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
            >>> image = Image.open(requests.get(url, stream=True).raw)
            ...
            >>> # training
            >>> text = "How many cats are in the picture?"
            >>> label = "2"
            >>> inputs = processor(images=image, text=text, return_tensors="pt")
            >>> labels = processor(text=label, return_tensors="pt").input_ids
            ...
            >>> inputs["labels"] = labels
            >>> outputs = model(**inputs)
            >>> loss = outputs.loss
            >>> loss.backward()
            ...
            >>> # inference
            >>> text = "How many cats are in the picture?"
            >>> inputs = processor(images=image, text=text, return_tensors="pt")
            >>> outputs = model.generate(**inputs)
            >>> print(processor.decode(outputs[0], skip_special_tokens=True))
            2
            ```
        """
        if labels is None and decoder_input_ids is None:
            raise ValueError(
                "Either `decoder_input_ids` or `labels` should be passed when calling `forward` with"
                " `BlipForQuestionAnswering`. if you are training the model make sure that `labels` is passed, if you"
                " are using the model for inference make sure that `decoder_input_ids` is passed or call `generate`"
            )

        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )

        vision_outputs = self.vision_model(
            pixel_values=pixel_values,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        image_embeds = vision_outputs[0]
        image_attention_mask = ops.ones(*image_embeds.shape[:-1], dtype=mindspore.int64)

        question_embeds = self.text_encoder(
            input_ids=input_ids,
            attention_mask=attention_mask,
            encoder_hidden_states=image_embeds,
            encoder_attention_mask=image_attention_mask,
            return_dict=return_dict,
        )

        if labels is not None and decoder_input_ids is None:
            # labels are already shifted right, see: https://github.com/huggingface/transformers/pull/23153
            decoder_input_ids = labels

        question_embeds = question_embeds[0] if not return_dict else question_embeds.last_hidden_state

        answer_output = self.text_decoder(
            input_ids=decoder_input_ids,
            attention_mask=decoder_attention_mask,
            encoder_hidden_states=question_embeds,
            encoder_attention_mask=attention_mask,
            labels=labels,
            return_dict=return_dict,
            reduction="mean",
        )

        if labels is not None:
            decoder_loss = answer_output.loss.mean() if return_dict else answer_output[0].mean()
        else:
            decoder_loss = None

        if not return_dict:
            outputs = (decoder_loss, image_embeds, vision_outputs[0]) + vision_outputs[2:]
            return tuple(output for output in outputs if output is not None)

        return BlipTextVisionModelOutput(
            loss=decoder_loss,
            image_embeds=image_embeds,
            last_hidden_state=vision_outputs.last_hidden_state,
            hidden_states=vision_outputs.hidden_states,
            attentions=vision_outputs.attentions,
        )

    def generate(
        self,
        input_ids: mindspore.Tensor,
        pixel_values: mindspore.Tensor,
        attention_mask: Optional[mindspore.Tensor] = None,
        **generate_kwargs,
    ) -> mindspore.Tensor:
        r"""
        Overrides *generate* function to be able to use the model as a conditional generator

        Parameters:
            input_ids (*mindspore.Tensor* of shape *(batch_size, sequence_length)*):
                The sequence used as a prompt for the generation.
            pixel_values (*mindspore.Tensor* of shape *(batch_size, num_channels, image_height, image_width)*:
                Input image to be processed
            attention_mask (*mindspore.Tensor* of shape *(batch_size, sequence_length)*, *optional*):
                Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`. `1` for
                tokens that are NOT MASKED, `0` for MASKED tokens.
            **generate_kwargs:
                Additional arguments passed to the *generate* function of the decoder

        Example:
            ```python
            >>> from PIL import Image
            >>> import requests
            >>> from transformers import AutoProcessor, BlipForQuestionAnswering
            ...
            >>> model = BlipForQuestionAnswering.from_pretrained("Salesforce/blip-vqa-base")
            >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-vqa-base")
            ...
            >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
            >>> image = Image.open(requests.get(url, stream=True).raw)
            >>> text = "How many cats are in the picture?"
            ...
            >>> inputs = processor(images=image, text=text, return_tensors="pt")
            ...
            >>> outputs = model.generate(**inputs)
            >>> print(processor.decode(outputs[0], skip_special_tokens=True))
            2
            ```
        """
        vision_outputs = self.vision_model(pixel_values=pixel_values)

        image_embeds = vision_outputs[0]

        image_attention_mask = ops.ones(*image_embeds.shape[:-1], dtype=mindspore.int64)

        if isinstance(input_ids, list):
            input_ids = mindspore.Tensor(input_ids)

        question_outputs = self.text_encoder(
            input_ids=input_ids,
            attention_mask=attention_mask,
            encoder_hidden_states=image_embeds,
            encoder_attention_mask=image_attention_mask,
            return_dict=False,
        )

        question_embeds = question_outputs[0]

        question_attention_mask = ops.ones(*question_embeds.shape[:-1], dtype=mindspore.int64)

        bos_ids = ops.full(
            (question_embeds.shape[0], 1), fill_value=self.decoder_start_token_id
        )

        outputs = self.text_decoder.generate(
            input_ids=bos_ids,
            eos_token_id=self.config.text_config.sep_token_id,
            pad_token_id=self.config.text_config.pad_token_id,
            encoder_hidden_states=question_embeds,
            encoder_attention_mask=question_attention_mask,
            **generate_kwargs,
        )

        return outputs

mindnlp.transformers.models.blip.modeling_blip.BlipForQuestionAnswering.__init__(config)

Initializes an instance of BlipForQuestionAnswering.

PARAMETER	DESCRIPTION
self	The instance of the class.
config	An instance of BlipConfig containing the configuration for the model. TYPE: BlipConfig

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/blip/modeling_blip.py

def __init__(self, config: BlipConfig):
    """
    Initializes an instance of BlipForQuestionAnswering.

    Args:
        self: The instance of the class.
        config (BlipConfig): An instance of BlipConfig containing the configuration for the model.

    Returns:
        None.

    Raises:
        None
    """
    super().__init__(config)

    self.vision_model = BlipVisionModel(config.vision_config)

    self.text_encoder = BlipTextModel(config.text_config, add_pooling_layer=False)

    self.text_decoder = BlipTextLMHeadModel(config.text_config)

    self.decoder_pad_token_id = config.text_config.pad_token_id
    self.decoder_start_token_id = config.text_config.bos_token_id

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.blip.modeling_blip.BlipForQuestionAnswering.forward(input_ids, pixel_values, decoder_input_ids=None, decoder_attention_mask=None, attention_mask=None, output_attentions=None, output_hidden_states=None, labels=None, return_dict=None)

RETURNS	DESCRIPTION
Union[Tuple, BlipTextVisionModelOutput]	Union[Tuple, BlipTextVisionModelOutput]

Example

>>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, BlipForQuestionAnswering
...
>>> model = BlipForQuestionAnswering.from_pretrained("Salesforce/blip-vqa-base")
>>> processor = AutoProcessor.from_pretrained("Salesforce/blip-vqa-base")
...
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
...
>>> # training
>>> text = "How many cats are in the picture?"
>>> label = "2"
>>> inputs = processor(images=image, text=text, return_tensors="pt")
>>> labels = processor(text=label, return_tensors="pt").input_ids
...
>>> inputs["labels"] = labels
>>> outputs = model(**inputs)
>>> loss = outputs.loss
>>> loss.backward()
...
>>> # inference
>>> text = "How many cats are in the picture?"
>>> inputs = processor(images=image, text=text, return_tensors="pt")
>>> outputs = model.generate(**inputs)
>>> print(processor.decode(outputs[0], skip_special_tokens=True))
2

Source code in mindnlp/transformers/models/blip/modeling_blip.py

def forward(
    self,
    input_ids: mindspore.Tensor,
    pixel_values: mindspore.Tensor,
    decoder_input_ids: Optional[mindspore.Tensor] = None,
    decoder_attention_mask: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    labels: Optional[mindspore.Tensor] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple, BlipTextVisionModelOutput]:
    r"""
    Returns:
        `Union[Tuple, BlipTextVisionModelOutput]`

    Example:
        ```python
        >>> from PIL import Image
        >>> import requests
        >>> from transformers import AutoProcessor, BlipForQuestionAnswering
        ...
        >>> model = BlipForQuestionAnswering.from_pretrained("Salesforce/blip-vqa-base")
        >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-vqa-base")
        ...
        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
        >>> image = Image.open(requests.get(url, stream=True).raw)
        ...
        >>> # training
        >>> text = "How many cats are in the picture?"
        >>> label = "2"
        >>> inputs = processor(images=image, text=text, return_tensors="pt")
        >>> labels = processor(text=label, return_tensors="pt").input_ids
        ...
        >>> inputs["labels"] = labels
        >>> outputs = model(**inputs)
        >>> loss = outputs.loss
        >>> loss.backward()
        ...
        >>> # inference
        >>> text = "How many cats are in the picture?"
        >>> inputs = processor(images=image, text=text, return_tensors="pt")
        >>> outputs = model.generate(**inputs)
        >>> print(processor.decode(outputs[0], skip_special_tokens=True))
        2
        ```
    """
    if labels is None and decoder_input_ids is None:
        raise ValueError(
            "Either `decoder_input_ids` or `labels` should be passed when calling `forward` with"
            " `BlipForQuestionAnswering`. if you are training the model make sure that `labels` is passed, if you"
            " are using the model for inference make sure that `decoder_input_ids` is passed or call `generate`"
        )

    return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
    output_hidden_states = (
        output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
    )

    vision_outputs = self.vision_model(
        pixel_values=pixel_values,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    image_embeds = vision_outputs[0]
    image_attention_mask = ops.ones(*image_embeds.shape[:-1], dtype=mindspore.int64)

    question_embeds = self.text_encoder(
        input_ids=input_ids,
        attention_mask=attention_mask,
        encoder_hidden_states=image_embeds,
        encoder_attention_mask=image_attention_mask,
        return_dict=return_dict,
    )

    if labels is not None and decoder_input_ids is None:
        # labels are already shifted right, see: https://github.com/huggingface/transformers/pull/23153
        decoder_input_ids = labels

    question_embeds = question_embeds[0] if not return_dict else question_embeds.last_hidden_state

    answer_output = self.text_decoder(
        input_ids=decoder_input_ids,
        attention_mask=decoder_attention_mask,
        encoder_hidden_states=question_embeds,
        encoder_attention_mask=attention_mask,
        labels=labels,
        return_dict=return_dict,
        reduction="mean",
    )

    if labels is not None:
        decoder_loss = answer_output.loss.mean() if return_dict else answer_output[0].mean()
    else:
        decoder_loss = None

    if not return_dict:
        outputs = (decoder_loss, image_embeds, vision_outputs[0]) + vision_outputs[2:]
        return tuple(output for output in outputs if output is not None)

    return BlipTextVisionModelOutput(
        loss=decoder_loss,
        image_embeds=image_embeds,
        last_hidden_state=vision_outputs.last_hidden_state,
        hidden_states=vision_outputs.hidden_states,
        attentions=vision_outputs.attentions,
    )

mindnlp.transformers.models.blip.modeling_blip.BlipForQuestionAnswering.generate(input_ids, pixel_values, attention_mask=None, **generate_kwargs)

Overrides generate function to be able to use the model as a conditional generator

PARAMETER	DESCRIPTION
input_ids	The sequence used as a prompt for the generation. TYPE: *mindspore.Tensor* of shape *(batch_size, sequence_length)*
pixel_values	Input image to be processed TYPE: *mindspore.Tensor* of shape *(batch_size, num_channels, image_height, image_width)*
attention_mask	Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]. 1 for tokens that are NOT MASKED, 0 for MASKED tokens. TYPE: *mindspore.Tensor* of shape *(batch_size, sequence_length)*, *optional* DEFAULT: None
**generate_kwargs	Additional arguments passed to the generate function of the decoder DEFAULT: {}

Example

>>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, BlipForQuestionAnswering
...
>>> model = BlipForQuestionAnswering.from_pretrained("Salesforce/blip-vqa-base")
>>> processor = AutoProcessor.from_pretrained("Salesforce/blip-vqa-base")
...
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> text = "How many cats are in the picture?"
...
>>> inputs = processor(images=image, text=text, return_tensors="pt")
...
>>> outputs = model.generate(**inputs)
>>> print(processor.decode(outputs[0], skip_special_tokens=True))
2

Source code in mindnlp/transformers/models/blip/modeling_blip.py

def generate(
    self,
    input_ids: mindspore.Tensor,
    pixel_values: mindspore.Tensor,
    attention_mask: Optional[mindspore.Tensor] = None,
    **generate_kwargs,
) -> mindspore.Tensor:
    r"""
    Overrides *generate* function to be able to use the model as a conditional generator

    Parameters:
        input_ids (*mindspore.Tensor* of shape *(batch_size, sequence_length)*):
            The sequence used as a prompt for the generation.
        pixel_values (*mindspore.Tensor* of shape *(batch_size, num_channels, image_height, image_width)*:
            Input image to be processed
        attention_mask (*mindspore.Tensor* of shape *(batch_size, sequence_length)*, *optional*):
            Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`. `1` for
            tokens that are NOT MASKED, `0` for MASKED tokens.
        **generate_kwargs:
            Additional arguments passed to the *generate* function of the decoder

    Example:
        ```python
        >>> from PIL import Image
        >>> import requests
        >>> from transformers import AutoProcessor, BlipForQuestionAnswering
        ...
        >>> model = BlipForQuestionAnswering.from_pretrained("Salesforce/blip-vqa-base")
        >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-vqa-base")
        ...
        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
        >>> image = Image.open(requests.get(url, stream=True).raw)
        >>> text = "How many cats are in the picture?"
        ...
        >>> inputs = processor(images=image, text=text, return_tensors="pt")
        ...
        >>> outputs = model.generate(**inputs)
        >>> print(processor.decode(outputs[0], skip_special_tokens=True))
        2
        ```
    """
    vision_outputs = self.vision_model(pixel_values=pixel_values)

    image_embeds = vision_outputs[0]

    image_attention_mask = ops.ones(*image_embeds.shape[:-1], dtype=mindspore.int64)

    if isinstance(input_ids, list):
        input_ids = mindspore.Tensor(input_ids)

    question_outputs = self.text_encoder(
        input_ids=input_ids,
        attention_mask=attention_mask,
        encoder_hidden_states=image_embeds,
        encoder_attention_mask=image_attention_mask,
        return_dict=False,
    )

    question_embeds = question_outputs[0]

    question_attention_mask = ops.ones(*question_embeds.shape[:-1], dtype=mindspore.int64)

    bos_ids = ops.full(
        (question_embeds.shape[0], 1), fill_value=self.decoder_start_token_id
    )

    outputs = self.text_decoder.generate(
        input_ids=bos_ids,
        eos_token_id=self.config.text_config.sep_token_id,
        pad_token_id=self.config.text_config.pad_token_id,
        encoder_hidden_states=question_embeds,
        encoder_attention_mask=question_attention_mask,
        **generate_kwargs,
    )

    return outputs

mindnlp.transformers.models.blip.modeling_blip.BlipForQuestionAnswering.get_input_embeddings()

This method returns the input embeddings from the vision model for question answering.

PARAMETER	DESCRIPTION
self	The instance of the BlipForQuestionAnswering class. TYPE: BlipForQuestionAnswering

RETURNS	DESCRIPTION
Module	nn.Module: The input embeddings from the vision model, which is of type nn.Module.

Source code in mindnlp/transformers/models/blip/modeling_blip.py

def get_input_embeddings(self) -> nn.Module:
    """
    This method returns the input embeddings from the vision model for question answering.

    Args:
        self (BlipForQuestionAnswering): The instance of the BlipForQuestionAnswering class.

    Returns:
        nn.Module: The input embeddings from the vision model, which is of type nn.Module.

    Raises:
        None
    """
    return self.vision_model.embeddings.patch_embedding

mindnlp.transformers.models.blip.modeling_blip.BlipVisionModel

Bases: BlipPreTrainedModel

A class representing the BlipVisionModel for vision tasks.

This class inherits from the BlipPreTrainedModel and provides methods to forward the model, get input embeddings, and return the outputs.

ATTRIBUTE	DESCRIPTION
config	The configuration for the BlipVisionModel. TYPE: BlipVisionConfig
embeddings	The embeddings layer for the BlipVisionModel. TYPE: BlipVisionEmbeddings
encoder	The encoder layer for the BlipVisionModel. TYPE: BlipEncoder
post_layernorm	The post layer normalization layer for the BlipVisionModel. TYPE: LayerNorm

METHOD	DESCRIPTION
__init__	Initializes the BlipVisionModel with the given configuration.
forward	Constructs the BlipVisionModel and returns the model outputs.
get_input_embeddings	Returns the input embeddings for the BlipVisionModel.

Source code in mindnlp/transformers/models/blip/modeling_blip.py

class BlipVisionModel(BlipPreTrainedModel):

    """
    A class representing the BlipVisionModel for vision tasks.

    This class inherits from the BlipPreTrainedModel and provides methods to forward the model, get input embeddings, and return the outputs.

    Attributes:
        config (BlipVisionConfig): The configuration for the BlipVisionModel.
        embeddings (BlipVisionEmbeddings): The embeddings layer for the BlipVisionModel.
        encoder (BlipEncoder): The encoder layer for the BlipVisionModel.
        post_layernorm (nn.LayerNorm): The post layer normalization layer for the BlipVisionModel.

    Methods:
        __init__: Initializes the BlipVisionModel with the given configuration.
        forward: Constructs the BlipVisionModel and returns the model outputs.
        get_input_embeddings: Returns the input embeddings for the BlipVisionModel.
    """
    main_input_name = "pixel_values"
    config_class = BlipVisionConfig

    def __init__(self, config: BlipVisionConfig):
        """
        Initializes a new instance of the BlipVisionModel class.

        Args:
            self: The object itself.
            config (BlipVisionConfig): The configuration object that holds all the necessary parameters for the model.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(config)
        self.config = config
        embed_dim = config.hidden_size

        self.embeddings = BlipVisionEmbeddings(config)
        self.encoder = BlipEncoder(config)
        self.post_layernorm = nn.LayerNorm(embed_dim, eps=config.layer_norm_eps)

        self.post_init()

    def forward(
        self,
        pixel_values: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, BaseModelOutputWithPooling]:
        r"""
        Returns:
            Union[Tuple, BaseModelOutputWithPooling]
        """
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        if pixel_values is None:
            raise ValueError("You have to specify pixel_values")

        hidden_states = self.embeddings(pixel_values)

        encoder_outputs = self.encoder(
            inputs_embeds=hidden_states,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        last_hidden_state = encoder_outputs[0]
        last_hidden_state = self.post_layernorm(last_hidden_state)

        pooled_output = last_hidden_state[:, 0, :]
        pooled_output = self.post_layernorm(pooled_output)

        if not return_dict:
            return (last_hidden_state, pooled_output) + encoder_outputs[1:]

        return BaseModelOutputWithPooling(
            last_hidden_state=last_hidden_state,
            pooler_output=pooled_output,
            hidden_states=encoder_outputs.hidden_states,
            attentions=encoder_outputs.attentions,
        )

    def get_input_embeddings(self):
        """
        Returns the input embeddings from the BlipVisionModel.

        Args:
            self (BlipVisionModel): An instance of the BlipVisionModel class.

        Returns:
            None.

        Raises:
            None.
        """
        return self.embeddings

mindnlp.transformers.models.blip.modeling_blip.BlipVisionModel.__init__(config)

Initializes a new instance of the BlipVisionModel class.

PARAMETER	DESCRIPTION
self	The object itself.
config	The configuration object that holds all the necessary parameters for the model. TYPE: BlipVisionConfig

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/blip/modeling_blip.py

def __init__(self, config: BlipVisionConfig):
    """
    Initializes a new instance of the BlipVisionModel class.

    Args:
        self: The object itself.
        config (BlipVisionConfig): The configuration object that holds all the necessary parameters for the model.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(config)
    self.config = config
    embed_dim = config.hidden_size

    self.embeddings = BlipVisionEmbeddings(config)
    self.encoder = BlipEncoder(config)
    self.post_layernorm = nn.LayerNorm(embed_dim, eps=config.layer_norm_eps)

    self.post_init()

mindnlp.transformers.models.blip.modeling_blip.BlipVisionModel.forward(pixel_values=None, output_attentions=None, output_hidden_states=None, return_dict=None)

RETURNS	DESCRIPTION
Union[Tuple, BaseModelOutputWithPooling]	Union[Tuple, BaseModelOutputWithPooling]

Source code in mindnlp/transformers/models/blip/modeling_blip.py

def forward(
    self,
    pixel_values: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple, BaseModelOutputWithPooling]:
    r"""
    Returns:
        Union[Tuple, BaseModelOutputWithPooling]
    """
    output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
    output_hidden_states = (
        output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
    )
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    if pixel_values is None:
        raise ValueError("You have to specify pixel_values")

    hidden_states = self.embeddings(pixel_values)

    encoder_outputs = self.encoder(
        inputs_embeds=hidden_states,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    last_hidden_state = encoder_outputs[0]
    last_hidden_state = self.post_layernorm(last_hidden_state)

    pooled_output = last_hidden_state[:, 0, :]
    pooled_output = self.post_layernorm(pooled_output)

    if not return_dict:
        return (last_hidden_state, pooled_output) + encoder_outputs[1:]

    return BaseModelOutputWithPooling(
        last_hidden_state=last_hidden_state,
        pooler_output=pooled_output,
        hidden_states=encoder_outputs.hidden_states,
        attentions=encoder_outputs.attentions,
    )

mindnlp.transformers.models.blip.modeling_blip.BlipVisionModel.get_input_embeddings()

Returns the input embeddings from the BlipVisionModel.

PARAMETER	DESCRIPTION
self	An instance of the BlipVisionModel class. TYPE: BlipVisionModel

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/blip/modeling_blip.py

def get_input_embeddings(self):
    """
    Returns the input embeddings from the BlipVisionModel.

    Args:
        self (BlipVisionModel): An instance of the BlipVisionModel class.

    Returns:
        None.

    Raises:
        None.
    """
    return self.embeddings

mindnlp.transformers.models.blip.modeling_blip.BlipTextModel

Bases: BlipTextPreTrainedModel

The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in Attention is all you need by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. argument and is_decoder set to True; an encoder_hidden_states is then expected as an input to the forward pass.

Source code in mindnlp/transformers/models/blip/modeling_blip_text.py

class BlipTextModel(BlipTextPreTrainedModel):
    """
    The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of
    cross-attention is added between the self-attention layers, following the architecture described in [Attention is
    all you need](https://arxiv.org/abs/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit,
    Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. argument and `is_decoder` set to `True`; an
    `encoder_hidden_states` is then expected as an input to the forward pass.
    """
    def __init__(self, config, add_pooling_layer=True):
        """
        Initializes a BlipTextModel object.

        Args:
            self: The object instance itself.
            config (dict): A dictionary containing configuration parameters for the BlipTextModel.
            add_pooling_layer (bool): A flag indicating whether to add a pooling layer to the model. Default is True.

        Returns:
            None.

        Raises:
            TypeError: If the config parameter is not provided or not of type dict.
            ValueError: If the config dictionary is missing required keys or has invalid values.
            RuntimeError: If an issue occurs during the initialization process.
        """
        super().__init__(config)
        self.config = config

        self.embeddings = BlipTextEmbeddings(config)
        self.encoder = BlipTextEncoder(config)
        self.pooler = BlipTextPooler(config) if add_pooling_layer else None

        self.post_init()

    def get_input_embeddings(self):
        """
        This method returns the input embeddings from the BlipTextModel.

        Args:
            self (BlipTextModel): The instance of the BlipTextModel class.

        Returns:
            None: This method returns the input embeddings from the BlipTextModel.
                The input embeddings are retrieved from the word_embeddings attribute of the embeddings.

        Raises:
            None
        """
        return self.embeddings.word_embeddings

    def set_input_embeddings(self, value):
        """
        Sets the input embeddings for the BlipTextModel.

        Args:
            self (BlipTextModel): The instance of the BlipTextModel class.
            value (torch.Tensor): The input embeddings to be set for the BlipTextModel.
                It should be a tensor of shape (vocab_size, embedding_dim).

        Returns:
            None.

        Raises:
            None.
        """
        self.embeddings.word_embeddings = value

    # Copied from transformers.models.bert.modeling_bert.BertModel._prune_heads
    def _prune_heads(self, heads_to_prune):
        """
        Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base
        class PreTrainedModel
        """
        for layer, heads in heads_to_prune.items():
            self.encoder.layer[layer].attention.prune_heads(heads)

    def get_extended_attention_mask(
        self, attention_mask: mindspore.Tensor, input_shape: Tuple[int], is_decoder: bool
    ) -> mindspore.Tensor:
        """
        Makes broadcastable attention and causal masks so that future and masked tokens are ignored.

        Arguments:
            attention_mask (`mindspore.Tensor`):
                Mask with ones indicating tokens to attend to, zeros for tokens to ignore.
            input_shape (`Tuple[int]`):
                The shape of the input to the model.

        Returns:
            `mindspore.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`.
        """
        # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length]
        # ourselves in which case we just need to make it broadcastable to all heads.
        if attention_mask.ndim == 3:
            extended_attention_mask = attention_mask[:, None, :, :]
        elif attention_mask.ndim == 2:
            # Provided a padding mask of dimensions [batch_size, seq_length]
            # - if the model is a decoder, apply a causal mask in addition to the padding mask
            # - if the model is an encoder, make the mask broadcastable to [batch_size, num_heads, seq_length, seq_length]
            if is_decoder:
                batch_size, seq_length = input_shape

                seq_ids = ops.arange(seq_length)
                causal_mask = ops.tile(seq_ids[None, None, :], (batch_size, seq_length, 1)) <= seq_ids[None, :, None]
                # in case past_key_values are used we need to add a prefix ones mask to the causal mask
                # causal and attention masks must have same type with pytorch version < 1.3
                causal_mask = causal_mask.to(attention_mask.dtype)

                if causal_mask.shape[1] < attention_mask.shape[1]:
                    prefix_seq_len = attention_mask.shape[1] - causal_mask.shape[1]
                    causal_mask = ops.cat(
                        [
                            ops.ones(
                                batch_size, seq_length, prefix_seq_len, dtype=causal_mask.dtype
                            ),
                            causal_mask,
                        ],
                        dim=-1,
                    )

                extended_attention_mask = causal_mask[:, None, :, :] * attention_mask[:, None, None, :]
            else:
                extended_attention_mask = attention_mask[:, None, None, :]
        else:
            raise ValueError(
                "Wrong shape for input_ids (shape {}) or attention_mask (shape {})".format(
                    input_shape, attention_mask.shape
                )
            )

        # Since attention_mask is 1.0 for positions we want to attend and 0.0 for
        # masked positions, this operation will create a tensor which is 0.0 for
        # positions we want to attend and -10000.0 for masked positions.
        # Since we are adding it to the raw scores before the softmax, this is
        # effectively the same as removing these entirely.
        extended_attention_mask = extended_attention_mask.to(dtype=self.dtype)  # fp16 compatibility
        extended_attention_mask = (1.0 - extended_attention_mask) * -10000.0
        return extended_attention_mask

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        position_ids: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        encoder_embeds: Optional[mindspore.Tensor] = None,
        encoder_hidden_states: Optional[mindspore.Tensor] = None,
        encoder_attention_mask: Optional[mindspore.Tensor] = None,
        past_key_values: Optional[List[mindspore.Tensor]] = None,
        use_cache: Optional[bool] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
        is_decoder: Optional[bool] = False,
    ) -> Union[Tuple[mindspore.Tensor], BaseModelOutputWithPoolingAndCrossAttentions]:
        r"""
        Args:
            encoder_hidden_states  (`mindspore.Tensor`, *optional*):
                Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if
                the model is configured as a decoder.
            encoder_attention_mask (`mindspore.Tensor`, *optional*):
                Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in
                the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`:

                - 1 for tokens that are **not masked**,
                - 0 for tokens that are **masked**.
            past_key_values (`tuple(tuple(mindspore.Tensor))`, *optional*):
                Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding.
                If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that
                don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all
                `decoder_input_ids` of shape `(batch_size, sequence_length)`.
            use_cache (`bool`, *optional*):
                If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see
                `past_key_values`).
        """
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        if is_decoder:
            use_cache = use_cache if use_cache is not None else self.config.use_cache
        else:
            use_cache = False

        if input_ids is not None and inputs_embeds is not None:
            raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
        elif input_ids is not None:
            self.warn_if_padding_and_no_attention_mask(input_ids, attention_mask)
            input_shape = input_ids.shape
            batch_size, seq_length = input_shape
        elif inputs_embeds is not None:
            input_shape = inputs_embeds.shape[:-1]
            batch_size, seq_length = input_shape
        elif encoder_embeds is not None:
            input_shape = encoder_embeds.shape[:-1]
            batch_size, seq_length = input_shape
        else:
            raise ValueError("You have to specify either input_ids or inputs_embeds or encoder_embeds")

        # past_key_values_length
        past_key_values_length = past_key_values[0][0].shape[2] if past_key_values is not None else 0

        if attention_mask is None:
            attention_mask = ops.ones(batch_size, seq_length + past_key_values_length)

        # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length]
        # ourselves in which case we just need to make it broadcastable to all heads.
        extended_attention_mask: mindspore.Tensor = self.get_extended_attention_mask(
            attention_mask, input_shape, is_decoder
        )

        # If a 2D or 3D attention mask is provided for the cross-attention
        # we need to make broadcastable to [batch_size, num_heads, seq_length, seq_length]
        if encoder_hidden_states is not None:
            if isinstance(encoder_hidden_states, list):
                encoder_batch_size, encoder_sequence_length, _ = encoder_hidden_states[0].shape
            else:
                encoder_batch_size, encoder_sequence_length, _ = encoder_hidden_states.shape
            encoder_hidden_shape = (encoder_batch_size, encoder_sequence_length)

            if isinstance(encoder_attention_mask, list):
                encoder_extended_attention_mask = [self.invert_attention_mask(mask) for mask in encoder_attention_mask]
            elif encoder_attention_mask is None:
                encoder_attention_mask = ops.ones(*encoder_hidden_shape)
                encoder_extended_attention_mask = self.invert_attention_mask(encoder_attention_mask)
            else:
                encoder_extended_attention_mask = self.invert_attention_mask(encoder_attention_mask)
        else:
            encoder_extended_attention_mask = None

        # Prepare head mask if needed
        # 1.0 in head_mask indicate we keep the head
        # attention_probs has shape bsz x n_heads x N x N
        # input head_mask has shape [num_heads] or [num_hidden_layers x num_heads]
        # and head_mask is converted to shape [num_hidden_layers x batch x num_heads x seq_length x seq_length]
        head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)

        if encoder_embeds is None:
            embedding_output = self.embeddings(
                input_ids=input_ids,
                position_ids=position_ids,
                inputs_embeds=inputs_embeds,
                past_key_values_length=past_key_values_length,
            )
        else:
            embedding_output = encoder_embeds

        encoder_outputs = self.encoder(
            embedding_output,
            attention_mask=extended_attention_mask,
            head_mask=head_mask,
            encoder_hidden_states=encoder_hidden_states,
            encoder_attention_mask=encoder_extended_attention_mask,
            past_key_values=past_key_values,
            use_cache=use_cache,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )
        sequence_output = encoder_outputs[0]
        pooled_output = self.pooler(sequence_output) if self.pooler is not None else None

        if not return_dict:
            return (sequence_output, pooled_output) + encoder_outputs[1:]

        return BaseModelOutputWithPoolingAndCrossAttentions(
            last_hidden_state=sequence_output,
            pooler_output=pooled_output,
            past_key_values=encoder_outputs.past_key_values,
            hidden_states=encoder_outputs.hidden_states,
            attentions=encoder_outputs.attentions,
            cross_attentions=encoder_outputs.cross_attentions,
        )

mindnlp.transformers.models.blip.modeling_blip.BlipTextModel.__init__(config, add_pooling_layer=True)

Initializes a BlipTextModel object.

PARAMETER	DESCRIPTION
self	The object instance itself.
config	A dictionary containing configuration parameters for the BlipTextModel. TYPE: dict
add_pooling_layer	A flag indicating whether to add a pooling layer to the model. Default is True. TYPE: bool DEFAULT: True

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
TypeError	If the config parameter is not provided or not of type dict.
ValueError	If the config dictionary is missing required keys or has invalid values.
RuntimeError	If an issue occurs during the initialization process.

Source code in mindnlp/transformers/models/blip/modeling_blip_text.py

def __init__(self, config, add_pooling_layer=True):
    """
    Initializes a BlipTextModel object.

    Args:
        self: The object instance itself.
        config (dict): A dictionary containing configuration parameters for the BlipTextModel.
        add_pooling_layer (bool): A flag indicating whether to add a pooling layer to the model. Default is True.

    Returns:
        None.

    Raises:
        TypeError: If the config parameter is not provided or not of type dict.
        ValueError: If the config dictionary is missing required keys or has invalid values.
        RuntimeError: If an issue occurs during the initialization process.
    """
    super().__init__(config)
    self.config = config

    self.embeddings = BlipTextEmbeddings(config)
    self.encoder = BlipTextEncoder(config)
    self.pooler = BlipTextPooler(config) if add_pooling_layer else None

    self.post_init()

mindnlp.transformers.models.blip.modeling_blip.BlipTextModel.forward(input_ids=None, attention_mask=None, position_ids=None, head_mask=None, inputs_embeds=None, encoder_embeds=None, encoder_hidden_states=None, encoder_attention_mask=None, past_key_values=None, use_cache=None, output_attentions=None, output_hidden_states=None, return_dict=None, is_decoder=False)

PARAMETER	DESCRIPTION
encoder_hidden_states	Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. TYPE: (`mindspore.Tensor`, *optional* DEFAULT: None
encoder_attention_mask	Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in [0, 1]: 1 for tokens that are not masked, 0 for tokens that are masked. TYPE: `mindspore.Tensor`, *optional* DEFAULT: None
past_key_values	Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that don't have their past key value states given to this model) of shape (batch_size, 1) instead of all decoder_input_ids of shape (batch_size, sequence_length). TYPE: `tuple(tuple(mindspore.Tensor))`, *optional* DEFAULT: None
use_cache	If set to True, past_key_values key value states are returned and can be used to speed up decoding (see past_key_values). TYPE: `bool`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/blip/modeling_blip_text.py

def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    position_ids: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    inputs_embeds: Optional[mindspore.Tensor] = None,
    encoder_embeds: Optional[mindspore.Tensor] = None,
    encoder_hidden_states: Optional[mindspore.Tensor] = None,
    encoder_attention_mask: Optional[mindspore.Tensor] = None,
    past_key_values: Optional[List[mindspore.Tensor]] = None,
    use_cache: Optional[bool] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
    is_decoder: Optional[bool] = False,
) -> Union[Tuple[mindspore.Tensor], BaseModelOutputWithPoolingAndCrossAttentions]:
    r"""
    Args:
        encoder_hidden_states  (`mindspore.Tensor`, *optional*):
            Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if
            the model is configured as a decoder.
        encoder_attention_mask (`mindspore.Tensor`, *optional*):
            Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in
            the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`:

            - 1 for tokens that are **not masked**,
            - 0 for tokens that are **masked**.
        past_key_values (`tuple(tuple(mindspore.Tensor))`, *optional*):
            Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding.
            If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that
            don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all
            `decoder_input_ids` of shape `(batch_size, sequence_length)`.
        use_cache (`bool`, *optional*):
            If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see
            `past_key_values`).
    """
    output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
    output_hidden_states = (
        output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
    )
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    if is_decoder:
        use_cache = use_cache if use_cache is not None else self.config.use_cache
    else:
        use_cache = False

    if input_ids is not None and inputs_embeds is not None:
        raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
    elif input_ids is not None:
        self.warn_if_padding_and_no_attention_mask(input_ids, attention_mask)
        input_shape = input_ids.shape
        batch_size, seq_length = input_shape
    elif inputs_embeds is not None:
        input_shape = inputs_embeds.shape[:-1]
        batch_size, seq_length = input_shape
    elif encoder_embeds is not None:
        input_shape = encoder_embeds.shape[:-1]
        batch_size, seq_length = input_shape
    else:
        raise ValueError("You have to specify either input_ids or inputs_embeds or encoder_embeds")

    # past_key_values_length
    past_key_values_length = past_key_values[0][0].shape[2] if past_key_values is not None else 0

    if attention_mask is None:
        attention_mask = ops.ones(batch_size, seq_length + past_key_values_length)

    # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length]
    # ourselves in which case we just need to make it broadcastable to all heads.
    extended_attention_mask: mindspore.Tensor = self.get_extended_attention_mask(
        attention_mask, input_shape, is_decoder
    )

    # If a 2D or 3D attention mask is provided for the cross-attention
    # we need to make broadcastable to [batch_size, num_heads, seq_length, seq_length]
    if encoder_hidden_states is not None:
        if isinstance(encoder_hidden_states, list):
            encoder_batch_size, encoder_sequence_length, _ = encoder_hidden_states[0].shape
        else:
            encoder_batch_size, encoder_sequence_length, _ = encoder_hidden_states.shape
        encoder_hidden_shape = (encoder_batch_size, encoder_sequence_length)

        if isinstance(encoder_attention_mask, list):
            encoder_extended_attention_mask = [self.invert_attention_mask(mask) for mask in encoder_attention_mask]
        elif encoder_attention_mask is None:
            encoder_attention_mask = ops.ones(*encoder_hidden_shape)
            encoder_extended_attention_mask = self.invert_attention_mask(encoder_attention_mask)
        else:
            encoder_extended_attention_mask = self.invert_attention_mask(encoder_attention_mask)
    else:
        encoder_extended_attention_mask = None

    # Prepare head mask if needed
    # 1.0 in head_mask indicate we keep the head
    # attention_probs has shape bsz x n_heads x N x N
    # input head_mask has shape [num_heads] or [num_hidden_layers x num_heads]
    # and head_mask is converted to shape [num_hidden_layers x batch x num_heads x seq_length x seq_length]
    head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)

    if encoder_embeds is None:
        embedding_output = self.embeddings(
            input_ids=input_ids,
            position_ids=position_ids,
            inputs_embeds=inputs_embeds,
            past_key_values_length=past_key_values_length,
        )
    else:
        embedding_output = encoder_embeds

    encoder_outputs = self.encoder(
        embedding_output,
        attention_mask=extended_attention_mask,
        head_mask=head_mask,
        encoder_hidden_states=encoder_hidden_states,
        encoder_attention_mask=encoder_extended_attention_mask,
        past_key_values=past_key_values,
        use_cache=use_cache,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )
    sequence_output = encoder_outputs[0]
    pooled_output = self.pooler(sequence_output) if self.pooler is not None else None

    if not return_dict:
        return (sequence_output, pooled_output) + encoder_outputs[1:]

    return BaseModelOutputWithPoolingAndCrossAttentions(
        last_hidden_state=sequence_output,
        pooler_output=pooled_output,
        past_key_values=encoder_outputs.past_key_values,
        hidden_states=encoder_outputs.hidden_states,
        attentions=encoder_outputs.attentions,
        cross_attentions=encoder_outputs.cross_attentions,
    )

mindnlp.transformers.models.blip.modeling_blip.BlipTextModel.get_extended_attention_mask(attention_mask, input_shape, is_decoder)

Makes broadcastable attention and causal masks so that future and masked tokens are ignored.

PARAMETER	DESCRIPTION
attention_mask	Mask with ones indicating tokens to attend to, zeros for tokens to ignore. TYPE: `mindspore.Tensor`
input_shape	The shape of the input to the model. TYPE: `Tuple[int]`

RETURNS	DESCRIPTION
Tensor	mindspore.Tensor The extended attention mask, with a the same dtype as attention_mask.dtype.

Source code in mindnlp/transformers/models/blip/modeling_blip_text.py

def get_extended_attention_mask(
    self, attention_mask: mindspore.Tensor, input_shape: Tuple[int], is_decoder: bool
) -> mindspore.Tensor:
    """
    Makes broadcastable attention and causal masks so that future and masked tokens are ignored.

    Arguments:
        attention_mask (`mindspore.Tensor`):
            Mask with ones indicating tokens to attend to, zeros for tokens to ignore.
        input_shape (`Tuple[int]`):
            The shape of the input to the model.

    Returns:
        `mindspore.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`.
    """
    # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length]
    # ourselves in which case we just need to make it broadcastable to all heads.
    if attention_mask.ndim == 3:
        extended_attention_mask = attention_mask[:, None, :, :]
    elif attention_mask.ndim == 2:
        # Provided a padding mask of dimensions [batch_size, seq_length]
        # - if the model is a decoder, apply a causal mask in addition to the padding mask
        # - if the model is an encoder, make the mask broadcastable to [batch_size, num_heads, seq_length, seq_length]
        if is_decoder:
            batch_size, seq_length = input_shape

            seq_ids = ops.arange(seq_length)
            causal_mask = ops.tile(seq_ids[None, None, :], (batch_size, seq_length, 1)) <= seq_ids[None, :, None]
            # in case past_key_values are used we need to add a prefix ones mask to the causal mask
            # causal and attention masks must have same type with pytorch version < 1.3
            causal_mask = causal_mask.to(attention_mask.dtype)

            if causal_mask.shape[1] < attention_mask.shape[1]:
                prefix_seq_len = attention_mask.shape[1] - causal_mask.shape[1]
                causal_mask = ops.cat(
                    [
                        ops.ones(
                            batch_size, seq_length, prefix_seq_len, dtype=causal_mask.dtype
                        ),
                        causal_mask,
                    ],
                    dim=-1,
                )

            extended_attention_mask = causal_mask[:, None, :, :] * attention_mask[:, None, None, :]
        else:
            extended_attention_mask = attention_mask[:, None, None, :]
    else:
        raise ValueError(
            "Wrong shape for input_ids (shape {}) or attention_mask (shape {})".format(
                input_shape, attention_mask.shape
            )
        )

    # Since attention_mask is 1.0 for positions we want to attend and 0.0 for
    # masked positions, this operation will create a tensor which is 0.0 for
    # positions we want to attend and -10000.0 for masked positions.
    # Since we are adding it to the raw scores before the softmax, this is
    # effectively the same as removing these entirely.
    extended_attention_mask = extended_attention_mask.to(dtype=self.dtype)  # fp16 compatibility
    extended_attention_mask = (1.0 - extended_attention_mask) * -10000.0
    return extended_attention_mask

mindnlp.transformers.models.blip.modeling_blip.BlipTextModel.get_input_embeddings()

This method returns the input embeddings from the BlipTextModel.

PARAMETER	DESCRIPTION
self	The instance of the BlipTextModel class. TYPE: BlipTextModel

RETURNS	DESCRIPTION
None	This method returns the input embeddings from the BlipTextModel. The input embeddings are retrieved from the word_embeddings attribute of the embeddings.

Source code in mindnlp/transformers/models/blip/modeling_blip_text.py

def get_input_embeddings(self):
    """
    This method returns the input embeddings from the BlipTextModel.

    Args:
        self (BlipTextModel): The instance of the BlipTextModel class.

    Returns:
        None: This method returns the input embeddings from the BlipTextModel.
            The input embeddings are retrieved from the word_embeddings attribute of the embeddings.

    Raises:
        None
    """
    return self.embeddings.word_embeddings

mindnlp.transformers.models.blip.modeling_blip.BlipTextModel.set_input_embeddings(value)

Sets the input embeddings for the BlipTextModel.

PARAMETER	DESCRIPTION
self	The instance of the BlipTextModel class. TYPE: BlipTextModel
value	The input embeddings to be set for the BlipTextModel. It should be a tensor of shape (vocab_size, embedding_dim). TYPE: Tensor

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/blip/modeling_blip_text.py

def set_input_embeddings(self, value):
    """
    Sets the input embeddings for the BlipTextModel.

    Args:
        self (BlipTextModel): The instance of the BlipTextModel class.
        value (torch.Tensor): The input embeddings to be set for the BlipTextModel.
            It should be a tensor of shape (vocab_size, embedding_dim).

    Returns:
        None.

    Raises:
        None.
    """
    self.embeddings.word_embeddings = value

mindnlp.transformers.models.blip.modeling_blip.BlipForImageTextRetrieval

Bases: BlipPreTrainedModel

BlipForImageTextRetrieval is a class that implements a model for image-text retrieval tasks. It is designed to retrieve relevant text based on input images and vice versa. This class inherits from BlipPreTrainedModel.

The class's forwardor initializes the model with the provided configuration. It sets up the vision model, text encoder, projection layers, and other necessary components for image-text retrieval.

The 'get_input_embeddings' method returns the patch embeddings from the vision model.

The 'forward' method takes input image and text tensors and forwards the output based on the specified parameters. It utilizes the vision model to extract image features and the text encoder to process input text. Depending on the 'use_itm_head' parameter, the method either computes the similarity score between image and text features or uses the image and text projections for matching.

The method also handles optional parameters for controlling the output format and behavior. It provides examples on how to use the BlipForImageTextRetrieval class for image-text retrieval tasks.

Note

This docstring is a high-level overview and does not include method signatures or detailed implementation details.

Source code in mindnlp/transformers/models/blip/modeling_blip.py

class BlipForImageTextRetrieval(BlipPreTrainedModel):

    """
    BlipForImageTextRetrieval is a class that implements a model for image-text retrieval tasks.
    It is designed to retrieve relevant text based on input images and vice versa. This class inherits from
    BlipPreTrainedModel.

    The class's forwardor initializes the model with the provided configuration.
    It sets up the vision model, text encoder, projection layers, and other necessary components for image-text retrieval.

    The 'get_input_embeddings' method returns the patch embeddings from the vision model.

    The 'forward' method takes input image and text tensors and forwards the output based on the specified parameters.
    It utilizes the vision model to extract image features and the text encoder to process input text.
    Depending on the 'use_itm_head' parameter, the method either computes the similarity score between image and text
    features or uses the image and text projections for matching.

    The method also handles optional parameters for controlling the output format and behavior. It provides examples on
    how to use the BlipForImageTextRetrieval class for image-text retrieval tasks.

    Note:
        This docstring is a high-level overview and does not include method signatures or detailed implementation details.
    """
    config_class = BlipConfig

    def __init__(self, config: BlipConfig):
        """
        Initializes an instance of the BlipForImageTextRetrieval class.

        Args:
            self: The instance of the class itself.
            config (BlipConfig):
                The configuration object containing various settings for the BlipForImageTextRetrieval model.

        Returns:
            None

        Raises:
            None
        """
        super().__init__(config)

        self.vision_model = BlipVisionModel(config.vision_config)

        self.text_encoder = BlipTextModel(config.text_config, add_pooling_layer=False)

        # vision projection layer
        self.vision_proj = nn.Linear(config.vision_config.hidden_size, config.image_text_hidden_size)

        # text projection layer
        self.text_proj = nn.Linear(config.text_config.hidden_size, config.image_text_hidden_size)

        # image text matching head
        self.itm_head = nn.Linear(config.text_config.hidden_size, 2)

        self.decoder_pad_token_id = (
            config.text_config.pad_token_id
            if not hasattr(config, "decoder_pad_token_id")
            else config.decoder_pad_token_id
        )
        self.decoder_start_token_id = (
            config.text_config.bos_token_id
            if not hasattr(config, "decoder_start_token_id")
            else config.decoder_start_token_id
        )

        # Initialize weights and apply final processing
        self.post_init()

    def get_input_embeddings(self) -> nn.Module:
        """
        Method to get the input embeddings from the vision model for image-text retrieval.

        Args:
            self (BlipForImageTextRetrieval): The instance of the BlipForImageTextRetrieval class.
                This parameter is required to access the vision model and its embeddings.

        Returns:
            nn.Module: A neural network cell representing the input embeddings obtained from the vision model.
                These embeddings are used for matching image features with text features in the retrieval process.

        Raises:
            None
        """
        return self.vision_model.embeddings.patch_embedding

    def forward(
        self,
        input_ids: mindspore.Tensor,
        pixel_values: mindspore.Tensor,
        use_itm_head: Optional[bool] = True,
        attention_mask: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, BlipTextVisionModelOutput]:
        r"""
        Returns:
            `Union[Tuple, BlipTextVisionModelOutput]`

        Example:
            ```python
            >>> from PIL import Image
            >>> import requests
            >>> from transformers import AutoProcessor, BlipForImageTextRetrieval
            ...
            >>> model = BlipForImageTextRetrieval.from_pretrained("Salesforce/blip-itm-base-coco")
            >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-itm-base-coco")
            ...
            >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
            >>> image = Image.open(requests.get(url, stream=True).raw)
            >>> text = "an image of a cat"
            ...
            >>> inputs = processor(images=image, text=text, return_tensors="pt")
            >>> outputs = model(**inputs)
            ```
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )

        vision_outputs = self.vision_model(
            pixel_values=pixel_values,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        image_embeds = vision_outputs[0]
        image_atts = ops.ones(*image_embeds.shape[:-1], dtype=mindspore.int64)

        if use_itm_head:
            question_embeds = self.text_encoder(
                input_ids=input_ids,
                attention_mask=attention_mask,
                encoder_hidden_states=image_embeds,
                encoder_attention_mask=image_atts,
                return_dict=return_dict,
            )
            question_embeds = question_embeds[0] if not return_dict else question_embeds.last_hidden_state

            output = self.itm_head(question_embeds[:, 0, :])
        else:
            question_embeds = self.text_encoder(
                input_ids=input_ids,
                attention_mask=attention_mask,
                return_dict=return_dict,
            )
            question_embeds = question_embeds[0] if not return_dict else question_embeds.last_hidden_state

            image_feat = normalize(self.vision_proj(image_embeds[:, 0, :]), dim=-1)
            text_feat = normalize(self.text_proj(question_embeds[:, 0, :]), dim=-1)

            output = image_feat @ text_feat.t()

        if not return_dict:
            outputs = (output, vision_outputs[0]) + vision_outputs[2:] + (question_embeds,)
            return tuple(output for output in outputs if output is not None)

        return BlipImageTextMatchingModelOutput(
            itm_score=output,
            last_hidden_state=vision_outputs.last_hidden_state,
            hidden_states=vision_outputs.hidden_states,
            attentions=vision_outputs.attentions,
            question_embeds=question_embeds,
        )

mindnlp.transformers.models.blip.modeling_blip.BlipForImageTextRetrieval.__init__(config)

Initializes an instance of the BlipForImageTextRetrieval class.

PARAMETER	DESCRIPTION
self	The instance of the class itself.
config	The configuration object containing various settings for the BlipForImageTextRetrieval model. TYPE: BlipConfig

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/blip/modeling_blip.py

def __init__(self, config: BlipConfig):
    """
    Initializes an instance of the BlipForImageTextRetrieval class.

    Args:
        self: The instance of the class itself.
        config (BlipConfig):
            The configuration object containing various settings for the BlipForImageTextRetrieval model.

    Returns:
        None

    Raises:
        None
    """
    super().__init__(config)

    self.vision_model = BlipVisionModel(config.vision_config)

    self.text_encoder = BlipTextModel(config.text_config, add_pooling_layer=False)

    # vision projection layer
    self.vision_proj = nn.Linear(config.vision_config.hidden_size, config.image_text_hidden_size)

    # text projection layer
    self.text_proj = nn.Linear(config.text_config.hidden_size, config.image_text_hidden_size)

    # image text matching head
    self.itm_head = nn.Linear(config.text_config.hidden_size, 2)

    self.decoder_pad_token_id = (
        config.text_config.pad_token_id
        if not hasattr(config, "decoder_pad_token_id")
        else config.decoder_pad_token_id
    )
    self.decoder_start_token_id = (
        config.text_config.bos_token_id
        if not hasattr(config, "decoder_start_token_id")
        else config.decoder_start_token_id
    )

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.blip.modeling_blip.BlipForImageTextRetrieval.forward(input_ids, pixel_values, use_itm_head=True, attention_mask=None, output_attentions=None, output_hidden_states=None, return_dict=None)

RETURNS	DESCRIPTION
Union[Tuple, BlipTextVisionModelOutput]	Union[Tuple, BlipTextVisionModelOutput]

Example

>>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, BlipForImageTextRetrieval
...
>>> model = BlipForImageTextRetrieval.from_pretrained("Salesforce/blip-itm-base-coco")
>>> processor = AutoProcessor.from_pretrained("Salesforce/blip-itm-base-coco")
...
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> text = "an image of a cat"
...
>>> inputs = processor(images=image, text=text, return_tensors="pt")
>>> outputs = model(**inputs)

Source code in mindnlp/transformers/models/blip/modeling_blip.py

def forward(
    self,
    input_ids: mindspore.Tensor,
    pixel_values: mindspore.Tensor,
    use_itm_head: Optional[bool] = True,
    attention_mask: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple, BlipTextVisionModelOutput]:
    r"""
    Returns:
        `Union[Tuple, BlipTextVisionModelOutput]`

    Example:
        ```python
        >>> from PIL import Image
        >>> import requests
        >>> from transformers import AutoProcessor, BlipForImageTextRetrieval
        ...
        >>> model = BlipForImageTextRetrieval.from_pretrained("Salesforce/blip-itm-base-coco")
        >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-itm-base-coco")
        ...
        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
        >>> image = Image.open(requests.get(url, stream=True).raw)
        >>> text = "an image of a cat"
        ...
        >>> inputs = processor(images=image, text=text, return_tensors="pt")
        >>> outputs = model(**inputs)
        ```
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
    output_hidden_states = (
        output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
    )

    vision_outputs = self.vision_model(
        pixel_values=pixel_values,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    image_embeds = vision_outputs[0]
    image_atts = ops.ones(*image_embeds.shape[:-1], dtype=mindspore.int64)

    if use_itm_head:
        question_embeds = self.text_encoder(
            input_ids=input_ids,
            attention_mask=attention_mask,
            encoder_hidden_states=image_embeds,
            encoder_attention_mask=image_atts,
            return_dict=return_dict,
        )
        question_embeds = question_embeds[0] if not return_dict else question_embeds.last_hidden_state

        output = self.itm_head(question_embeds[:, 0, :])
    else:
        question_embeds = self.text_encoder(
            input_ids=input_ids,
            attention_mask=attention_mask,
            return_dict=return_dict,
        )
        question_embeds = question_embeds[0] if not return_dict else question_embeds.last_hidden_state

        image_feat = normalize(self.vision_proj(image_embeds[:, 0, :]), dim=-1)
        text_feat = normalize(self.text_proj(question_embeds[:, 0, :]), dim=-1)

        output = image_feat @ text_feat.t()

    if not return_dict:
        outputs = (output, vision_outputs[0]) + vision_outputs[2:] + (question_embeds,)
        return tuple(output for output in outputs if output is not None)

    return BlipImageTextMatchingModelOutput(
        itm_score=output,
        last_hidden_state=vision_outputs.last_hidden_state,
        hidden_states=vision_outputs.hidden_states,
        attentions=vision_outputs.attentions,
        question_embeds=question_embeds,
    )

mindnlp.transformers.models.blip.modeling_blip.BlipForImageTextRetrieval.get_input_embeddings()

Method to get the input embeddings from the vision model for image-text retrieval.

PARAMETER	DESCRIPTION
self	The instance of the BlipForImageTextRetrieval class. This parameter is required to access the vision model and its embeddings. TYPE: BlipForImageTextRetrieval

RETURNS	DESCRIPTION
Module	nn.Module: A neural network cell representing the input embeddings obtained from the vision model. These embeddings are used for matching image features with text features in the retrieval process.

Source code in mindnlp/transformers/models/blip/modeling_blip.py

def get_input_embeddings(self) -> nn.Module:
    """
    Method to get the input embeddings from the vision model for image-text retrieval.

    Args:
        self (BlipForImageTextRetrieval): The instance of the BlipForImageTextRetrieval class.
            This parameter is required to access the vision model and its embeddings.

    Returns:
        nn.Module: A neural network cell representing the input embeddings obtained from the vision model.
            These embeddings are used for matching image features with text features in the retrieval process.

    Raises:
        None
    """
    return self.vision_model.embeddings.patch_embedding

mindnlp.transformers.models.blip.image_processing_blip.BlipImageProcessor

Bases: BaseImageProcessor

Constructs a BLIP image processor.

PARAMETER	DESCRIPTION
do_resize	Whether to resize the image's (height, width) dimensions to the specified size. Can be overridden by the do_resize parameter in the preprocess method. TYPE: `bool`, *optional*, defaults to `True` DEFAULT: True
size	384, "width": 384}): Size of the output image after resizing. Can be overridden by thesizeparameter in thepreprocess` method. TYPE: `dict`, *optional*, defaults to `{"height" DEFAULT: None
resample	Resampling filter to use if resizing the image. Only has an effect if do_resize is set to True. Can be overridden by the resample parameter in the preprocess method. TYPE: `PILImageResampling`, *optional*, defaults to `Resampling.BICUBIC` DEFAULT: BICUBIC
do_rescale	Whether to rescale the image by the specified scale rescale_factor. Can be overridden by the do_rescale parameter in the preprocess method. TYPE: `bool`, *optional*, defaults to `True` DEFAULT: True
rescale_factor	Scale factor to use if rescaling the image. Only has an effect if do_rescale is set to True. Can be overridden by the rescale_factor parameter in the preprocess method. TYPE: `int` or `float`, *optional*, defaults to `1/255` DEFAULT: 1 / 255
do_normalize	Whether to normalize the image. Can be overridden by the do_normalize parameter in the preprocess method. Can be overridden by the do_normalize parameter in the preprocess method. TYPE: `bool`, *optional*, defaults to `True` DEFAULT: True
image_mean	Mean to use if normalizing the image. This is a float or list of floats the length of the number of channels in the image. Can be overridden by the image_mean parameter in the preprocess method. Can be overridden by the image_mean parameter in the preprocess method. TYPE: `float` or `List[float]`, *optional*, defaults to `IMAGENET_STANDARD_MEAN` DEFAULT: None
image_std	Standard deviation to use if normalizing the image. This is a float or list of floats the length of the number of channels in the image. Can be overridden by the image_std parameter in the preprocess method. Can be overridden by the image_std parameter in the preprocess method. TYPE: `float` or `List[float]`, *optional*, defaults to `IMAGENET_STANDARD_STD` DEFAULT: None
do_convert_rgb	Whether to convert the image to RGB. TYPE: `bool`, *optional*, defaults to `True` DEFAULT: True

Source code in mindnlp/transformers/models/blip/image_processing_blip.py

class BlipImageProcessor(BaseImageProcessor):
    r"""
    Constructs a BLIP image processor.

    Args:
        do_resize (`bool`, *optional*, defaults to `True`):
            Whether to resize the image's (height, width) dimensions to the specified `size`. Can be overridden by the
            `do_resize` parameter in the `preprocess` method.
        size (`dict`, *optional*, defaults to `{"height": 384, "width": 384}`):
            Size of the output image after resizing. Can be overridden by the `size` parameter in the `preprocess`
            method.
        resample (`PILImageResampling`, *optional*, defaults to `Resampling.BICUBIC`):
            Resampling filter to use if resizing the image. Only has an effect if `do_resize` is set to `True`. Can be
            overridden by the `resample` parameter in the `preprocess` method.
        do_rescale (`bool`, *optional*, defaults to `True`):
            Whether to rescale the image by the specified scale `rescale_factor`. Can be overridden by the
            `do_rescale` parameter in the `preprocess` method.
        rescale_factor (`int` or `float`, *optional*, defaults to `1/255`):
            Scale factor to use if rescaling the image. Only has an effect if `do_rescale` is set to `True`. Can be
            overridden by the `rescale_factor` parameter in the `preprocess` method.
        do_normalize (`bool`, *optional*, defaults to `True`):
            Whether to normalize the image. Can be overridden by the `do_normalize` parameter in the `preprocess`
            method. Can be overridden by the `do_normalize` parameter in the `preprocess` method.
        image_mean (`float` or `List[float]`, *optional*, defaults to `IMAGENET_STANDARD_MEAN`):
            Mean to use if normalizing the image. This is a float or list of floats the length of the number of
            channels in the image. Can be overridden by the `image_mean` parameter in the `preprocess` method. Can be
            overridden by the `image_mean` parameter in the `preprocess` method.
        image_std (`float` or `List[float]`, *optional*, defaults to `IMAGENET_STANDARD_STD`):
            Standard deviation to use if normalizing the image. This is a float or list of floats the length of the
            number of channels in the image. Can be overridden by the `image_std` parameter in the `preprocess` method.
            Can be overridden by the `image_std` parameter in the `preprocess` method.
        do_convert_rgb (`bool`, *optional*, defaults to `True`):
            Whether to convert the image to RGB.
    """
    model_input_names = ["pixel_values"]

    def __init__(
        self,
        do_resize: bool = True,
        size: Dict[str, int] = None,
        resample: PILImageResampling = PILImageResampling.BICUBIC,
        do_rescale: bool = True,
        rescale_factor: Union[int, float] = 1 / 255,
        do_normalize: bool = True,
        image_mean: Optional[Union[float, List[float]]] = None,
        image_std: Optional[Union[float, List[float]]] = None,
        do_convert_rgb: bool = True,
        **kwargs,
    ) -> None:
        """
        Initializes a BlipImageProcessor object.

        Args:
            self: The BlipImageProcessor instance.
            do_resize (bool, optional): Specifies whether to resize the image. Defaults to True.
            size (Dict[str, int], optional): Specifies the desired height and width of the image.
                Defaults to {'height': 384, 'width': 384}.
            resample (PILImageResampling, optional): Specifies the resampling method for resizing the image.
                Defaults to PILImageResampling.BICUBIC.
            do_rescale (bool, optional): Specifies whether to rescale the image. Defaults to True.
            rescale_factor (Union[int, float], optional): Specifies the rescale factor for the image.
                Defaults to 1 / 255.
            do_normalize (bool, optional): Specifies whether to normalize the image. Defaults to True.
            image_mean (Optional[Union[float, List[float]]], optional): Specifies the mean value for image normalization.
                Defaults to None.
            image_std (Optional[Union[float, List[float]]], optional):
                Specifies the standard deviation value for image normalization. Defaults to None.
            do_convert_rgb (bool, optional): Specifies whether to convert the image to RGB format. Defaults to True.
            **kwargs: Additional keyword arguments.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(**kwargs)
        size = size if size is not None else {"height": 384, "width": 384}
        size = get_size_dict(size, default_to_square=True)

        self.do_resize = do_resize
        self.size = size
        self.resample = resample
        self.do_rescale = do_rescale
        self.rescale_factor = rescale_factor
        self.do_normalize = do_normalize
        self.image_mean = image_mean if image_mean is not None else OPENAI_CLIP_MEAN
        self.image_std = image_std if image_std is not None else OPENAI_CLIP_STD
        self.do_convert_rgb = do_convert_rgb
        self._valid_processor_keys = [
            "images",
            "do_resize",
            "size",
            "resample",
            "do_rescale",
            "rescale_factor",
            "do_normalize",
            "image_mean",
            "image_std",
            "do_convert_rgb",
            "return_tensors",
            "data_format",
            "input_data_format",
        ]

    # Copied from transformers.models.vit.image_processing_vit.ViTImageProcessor.resize with PILImageResampling.BILINEAR->PILImageResampling.BICUBIC
    def resize(
        self,
        image: np.ndarray,
        size: Dict[str, int],
        resample: PILImageResampling = PILImageResampling.BICUBIC,
        data_format: Optional[Union[str, ChannelDimension]] = None,
        input_data_format: Optional[Union[str, ChannelDimension]] = None,
        **kwargs,
    ) -> np.ndarray:
        """
        Resize an image to `(size["height"], size["width"])`.

        Args:
            image (`np.ndarray`):
                Image to resize.
            size (`Dict[str, int]`):
                Dictionary in the format `{"height": int, "width": int}` specifying the size of the output image.
            resample (`PILImageResampling`, *optional*, defaults to `PILImageResampling.BICUBIC`):
                `PILImageResampling` filter to use when resizing the image e.g. `PILImageResampling.BICUBIC`.
            data_format (`ChannelDimension` or `str`, *optional*):
                The channel dimension format for the output image. If unset, the channel dimension format of the input
                image is used. Can be one of:

                - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
                - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
                - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.
            input_data_format (`ChannelDimension` or `str`, *optional*):
                The channel dimension format for the input image. If unset, the channel dimension format is inferred
                from the input image. Can be one of:

                - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
                - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
                - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.

        Returns:
            `np.ndarray`: The resized image.
        """
        size = get_size_dict(size)
        if "height" not in size or "width" not in size:
            raise ValueError(f"The `size` dictionary must contain the keys `height` and `width`. Got {size.keys()}")
        output_size = (size["height"], size["width"])
        return resize(
            image,
            size=output_size,
            resample=resample,
            data_format=data_format,
            input_data_format=input_data_format,
            **kwargs,
        )

    def preprocess(
        self,
        images: ImageInput,
        do_resize: Optional[bool] = None,
        size: Optional[Dict[str, int]] = None,
        resample: PILImageResampling = None,
        do_rescale: Optional[bool] = None,
        rescale_factor: Optional[float] = None,
        do_normalize: Optional[bool] = None,
        image_mean: Optional[Union[float, List[float]]] = None,
        image_std: Optional[Union[float, List[float]]] = None,
        return_tensors: Optional[Union[str, TensorType]] = None,
        do_convert_rgb: bool = None,
        data_format: ChannelDimension = ChannelDimension.FIRST,
        input_data_format: Optional[Union[str, ChannelDimension]] = None,
        **kwargs,
    ) -> PIL.Image.Image:
        """
        Preprocess an image or batch of images.

        Args:
            images (`ImageInput`):
                Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If
                passing in images with pixel values between 0 and 1, set `do_rescale=False`.
            do_resize (`bool`, *optional*, defaults to `self.do_resize`):
                Whether to resize the image.
            size (`Dict[str, int]`, *optional*, defaults to `self.size`):
                Controls the size of the image after `resize`. The shortest edge of the image is resized to
                `size["shortest_edge"]` whilst preserving the aspect ratio. If the longest edge of this resized image
                is > `int(size["shortest_edge"] * (1333 / 800))`, then the image is resized again to make the longest
                edge equal to `int(size["shortest_edge"] * (1333 / 800))`.
            resample (`PILImageResampling`, *optional*, defaults to `self.resample`):
                Resampling filter to use if resizing the image. Only has an effect if `do_resize` is set to `True`.
            do_rescale (`bool`, *optional*, defaults to `self.do_rescale`):
                Whether to rescale the image values between [0 - 1].
            rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`):
                Rescale factor to rescale the image by if `do_rescale` is set to `True`.
            do_normalize (`bool`, *optional*, defaults to `self.do_normalize`):
                Whether to normalize the image.
            image_mean (`float` or `List[float]`, *optional*, defaults to `self.image_mean`):
                Image mean to normalize the image by if `do_normalize` is set to `True`.
            image_std (`float` or `List[float]`, *optional*, defaults to `self.image_std`):
                Image standard deviation to normalize the image by if `do_normalize` is set to `True`.
            do_convert_rgb (`bool`, *optional*, defaults to `self.do_convert_rgb`):
                Whether to convert the image to RGB.
            return_tensors (`str` or `TensorType`, *optional*):
                The type of tensors to return. Can be one of:

                - Unset: Return a list of `np.ndarray`.
                - `TensorType.TENSORFLOW` or `'tf'`: Return a batch of type `tf.Tensor`.
                - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`.
                - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`.
                - `TensorType.JAX` or `'jax'`: Return a batch of type `jax.numpy.ndarray`.
            data_format (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`):
                The channel dimension format for the output image. Can be one of:

                - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
                - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
                - Unset: Use the channel dimension format of the input image.
            input_data_format (`ChannelDimension` or `str`, *optional*):
                The channel dimension format for the input image. If unset, the channel dimension format is inferred
                from the input image. Can be one of:

                - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
                - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
                - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.
        """
        do_resize = do_resize if do_resize is not None else self.do_resize
        resample = resample if resample is not None else self.resample
        do_rescale = do_rescale if do_rescale is not None else self.do_rescale
        rescale_factor = rescale_factor if rescale_factor is not None else self.rescale_factor
        do_normalize = do_normalize if do_normalize is not None else self.do_normalize
        image_mean = image_mean if image_mean is not None else self.image_mean
        image_std = image_std if image_std is not None else self.image_std
        do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb

        size = size if size is not None else self.size
        size = get_size_dict(size, default_to_square=False)

        images = make_list_of_images(images)

        validate_kwargs(captured_kwargs=kwargs.keys(), valid_processor_keys=self._valid_processor_keys)

        if not valid_images(images):
            raise ValueError(
                "Invalid image type. Must be of type PIL.Image.Image, numpy.ndarray, "
                "torch.Tensor, tf.Tensor or jax.ndarray."
            )

        validate_preprocess_arguments(
            do_rescale=do_rescale,
            rescale_factor=rescale_factor,
            do_normalize=do_normalize,
            image_mean=image_mean,
            image_std=image_std,
            do_resize=do_resize,
            size=size,
            resample=resample,
        )
        # PIL RGBA images are converted to RGB
        if do_convert_rgb:
            images = [convert_to_rgb(image) for image in images]

        # All transformations expect numpy arrays.
        images = [to_numpy_array(image) for image in images]

        if is_scaled_image(images[0]) and do_rescale:
            logger.warning_once(
                "It looks like you are trying to rescale already rescaled images. If the input"
                " images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again."
            )

        if input_data_format is None:
            # We assume that all images have the same channel dimension format.
            input_data_format = infer_channel_dimension_format(images[0])

        if do_resize:
            images = [
                self.resize(image=image, size=size, resample=resample, input_data_format=input_data_format)
                for image in images
            ]

        if do_rescale:
            images = [
                self.rescale(image=image, scale=rescale_factor, input_data_format=input_data_format)
                for image in images
            ]

        if do_normalize:
            images = [
                self.normalize(image=image, mean=image_mean, std=image_std, input_data_format=input_data_format)
                for image in images
            ]

        images = [
            to_channel_dimension_format(image, data_format, input_channel_dim=input_data_format) for image in images
        ]

        encoded_outputs = BatchFeature(data={"pixel_values": images}, tensor_type=return_tensors)

        return encoded_outputs

mindnlp.transformers.models.blip.image_processing_blip.BlipImageProcessor.__init__(do_resize=True, size=None, resample=PILImageResampling.BICUBIC, do_rescale=True, rescale_factor=1 / 255, do_normalize=True, image_mean=None, image_std=None, do_convert_rgb=True, **kwargs)

Initializes a BlipImageProcessor object.

PARAMETER	DESCRIPTION
self	The BlipImageProcessor instance.
do_resize	Specifies whether to resize the image. Defaults to True. TYPE: bool DEFAULT: True
size	Specifies the desired height and width of the image. Defaults to {'height': 384, 'width': 384}. TYPE: Dict[str, int] DEFAULT: None
resample	Specifies the resampling method for resizing the image. Defaults to PILImageResampling.BICUBIC. TYPE: PILImageResampling DEFAULT: BICUBIC
do_rescale	Specifies whether to rescale the image. Defaults to True. TYPE: bool DEFAULT: True
rescale_factor	Specifies the rescale factor for the image. Defaults to 1 / 255. TYPE: Union[int, float] DEFAULT: 1 / 255
do_normalize	Specifies whether to normalize the image. Defaults to True. TYPE: bool DEFAULT: True
image_mean	Specifies the mean value for image normalization. Defaults to None. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
image_std	Specifies the standard deviation value for image normalization. Defaults to None. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
do_convert_rgb	Specifies whether to convert the image to RGB format. Defaults to True. TYPE: bool DEFAULT: True
**kwargs	Additional keyword arguments. DEFAULT: {}

RETURNS	DESCRIPTION
None	None.

Source code in mindnlp/transformers/models/blip/image_processing_blip.py

def __init__(
    self,
    do_resize: bool = True,
    size: Dict[str, int] = None,
    resample: PILImageResampling = PILImageResampling.BICUBIC,
    do_rescale: bool = True,
    rescale_factor: Union[int, float] = 1 / 255,
    do_normalize: bool = True,
    image_mean: Optional[Union[float, List[float]]] = None,
    image_std: Optional[Union[float, List[float]]] = None,
    do_convert_rgb: bool = True,
    **kwargs,
) -> None:
    """
    Initializes a BlipImageProcessor object.

    Args:
        self: The BlipImageProcessor instance.
        do_resize (bool, optional): Specifies whether to resize the image. Defaults to True.
        size (Dict[str, int], optional): Specifies the desired height and width of the image.
            Defaults to {'height': 384, 'width': 384}.
        resample (PILImageResampling, optional): Specifies the resampling method for resizing the image.
            Defaults to PILImageResampling.BICUBIC.
        do_rescale (bool, optional): Specifies whether to rescale the image. Defaults to True.
        rescale_factor (Union[int, float], optional): Specifies the rescale factor for the image.
            Defaults to 1 / 255.
        do_normalize (bool, optional): Specifies whether to normalize the image. Defaults to True.
        image_mean (Optional[Union[float, List[float]]], optional): Specifies the mean value for image normalization.
            Defaults to None.
        image_std (Optional[Union[float, List[float]]], optional):
            Specifies the standard deviation value for image normalization. Defaults to None.
        do_convert_rgb (bool, optional): Specifies whether to convert the image to RGB format. Defaults to True.
        **kwargs: Additional keyword arguments.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(**kwargs)
    size = size if size is not None else {"height": 384, "width": 384}
    size = get_size_dict(size, default_to_square=True)

    self.do_resize = do_resize
    self.size = size
    self.resample = resample
    self.do_rescale = do_rescale
    self.rescale_factor = rescale_factor
    self.do_normalize = do_normalize
    self.image_mean = image_mean if image_mean is not None else OPENAI_CLIP_MEAN
    self.image_std = image_std if image_std is not None else OPENAI_CLIP_STD
    self.do_convert_rgb = do_convert_rgb
    self._valid_processor_keys = [
        "images",
        "do_resize",
        "size",
        "resample",
        "do_rescale",
        "rescale_factor",
        "do_normalize",
        "image_mean",
        "image_std",
        "do_convert_rgb",
        "return_tensors",
        "data_format",
        "input_data_format",
    ]

mindnlp.transformers.models.blip.image_processing_blip.BlipImageProcessor.preprocess(images, do_resize=None, size=None, resample=None, do_rescale=None, rescale_factor=None, do_normalize=None, image_mean=None, image_std=None, return_tensors=None, do_convert_rgb=None, data_format=ChannelDimension.FIRST, input_data_format=None, **kwargs)

Preprocess an image or batch of images.

PARAMETER	DESCRIPTION
images	Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If passing in images with pixel values between 0 and 1, set do_rescale=False. TYPE: `ImageInput`
do_resize	Whether to resize the image. TYPE: `bool`, *optional*, defaults to `self.do_resize` DEFAULT: None
size	Controls the size of the image after resize. The shortest edge of the image is resized to size["shortest_edge"] whilst preserving the aspect ratio. If the longest edge of this resized image is > int(size["shortest_edge"] * (1333 / 800)), then the image is resized again to make the longest edge equal to int(size["shortest_edge"] * (1333 / 800)). TYPE: `Dict[str, int]`, *optional*, defaults to `self.size` DEFAULT: None
resample	Resampling filter to use if resizing the image. Only has an effect if do_resize is set to True. TYPE: `PILImageResampling`, *optional*, defaults to `self.resample` DEFAULT: None
do_rescale	Whether to rescale the image values between [0 - 1]. TYPE: `bool`, *optional*, defaults to `self.do_rescale` DEFAULT: None
rescale_factor	Rescale factor to rescale the image by if do_rescale is set to True. TYPE: `float`, *optional*, defaults to `self.rescale_factor` DEFAULT: None
do_normalize	Whether to normalize the image. TYPE: `bool`, *optional*, defaults to `self.do_normalize` DEFAULT: None
image_mean	Image mean to normalize the image by if do_normalize is set to True. TYPE: `float` or `List[float]`, *optional*, defaults to `self.image_mean` DEFAULT: None
image_std	Image standard deviation to normalize the image by if do_normalize is set to True. TYPE: `float` or `List[float]`, *optional*, defaults to `self.image_std` DEFAULT: None
do_convert_rgb	Whether to convert the image to RGB. TYPE: `bool`, *optional*, defaults to `self.do_convert_rgb` DEFAULT: None
return_tensors	The type of tensors to return. Can be one of: Unset: Return a list of np.ndarray. TensorType.TENSORFLOW or 'tf': Return a batch of type tf.Tensor. TensorType.PYTORCH or 'pt': Return a batch of type torch.Tensor. TensorType.NUMPY or 'np': Return a batch of type np.ndarray. TensorType.JAX or 'jax': Return a batch of type jax.numpy.ndarray. TYPE: `str` or `TensorType`, *optional* DEFAULT: None
data_format	The channel dimension format for the output image. Can be one of: "channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format. "channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format. Unset: Use the channel dimension format of the input image. TYPE: `ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST` DEFAULT: FIRST
input_data_format	The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: "channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format. "channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format. "none" or ChannelDimension.NONE: image in (height, width) format. TYPE: `ChannelDimension` or `str`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/blip/image_processing_blip.py

def preprocess(
    self,
    images: ImageInput,
    do_resize: Optional[bool] = None,
    size: Optional[Dict[str, int]] = None,
    resample: PILImageResampling = None,
    do_rescale: Optional[bool] = None,
    rescale_factor: Optional[float] = None,
    do_normalize: Optional[bool] = None,
    image_mean: Optional[Union[float, List[float]]] = None,
    image_std: Optional[Union[float, List[float]]] = None,
    return_tensors: Optional[Union[str, TensorType]] = None,
    do_convert_rgb: bool = None,
    data_format: ChannelDimension = ChannelDimension.FIRST,
    input_data_format: Optional[Union[str, ChannelDimension]] = None,
    **kwargs,
) -> PIL.Image.Image:
    """
    Preprocess an image or batch of images.

    Args:
        images (`ImageInput`):
            Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If
            passing in images with pixel values between 0 and 1, set `do_rescale=False`.
        do_resize (`bool`, *optional*, defaults to `self.do_resize`):
            Whether to resize the image.
        size (`Dict[str, int]`, *optional*, defaults to `self.size`):
            Controls the size of the image after `resize`. The shortest edge of the image is resized to
            `size["shortest_edge"]` whilst preserving the aspect ratio. If the longest edge of this resized image
            is > `int(size["shortest_edge"] * (1333 / 800))`, then the image is resized again to make the longest
            edge equal to `int(size["shortest_edge"] * (1333 / 800))`.
        resample (`PILImageResampling`, *optional*, defaults to `self.resample`):
            Resampling filter to use if resizing the image. Only has an effect if `do_resize` is set to `True`.
        do_rescale (`bool`, *optional*, defaults to `self.do_rescale`):
            Whether to rescale the image values between [0 - 1].
        rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`):
            Rescale factor to rescale the image by if `do_rescale` is set to `True`.
        do_normalize (`bool`, *optional*, defaults to `self.do_normalize`):
            Whether to normalize the image.
        image_mean (`float` or `List[float]`, *optional*, defaults to `self.image_mean`):
            Image mean to normalize the image by if `do_normalize` is set to `True`.
        image_std (`float` or `List[float]`, *optional*, defaults to `self.image_std`):
            Image standard deviation to normalize the image by if `do_normalize` is set to `True`.
        do_convert_rgb (`bool`, *optional*, defaults to `self.do_convert_rgb`):
            Whether to convert the image to RGB.
        return_tensors (`str` or `TensorType`, *optional*):
            The type of tensors to return. Can be one of:

            - Unset: Return a list of `np.ndarray`.
            - `TensorType.TENSORFLOW` or `'tf'`: Return a batch of type `tf.Tensor`.
            - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`.
            - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`.
            - `TensorType.JAX` or `'jax'`: Return a batch of type `jax.numpy.ndarray`.
        data_format (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`):
            The channel dimension format for the output image. Can be one of:

            - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
            - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
            - Unset: Use the channel dimension format of the input image.
        input_data_format (`ChannelDimension` or `str`, *optional*):
            The channel dimension format for the input image. If unset, the channel dimension format is inferred
            from the input image. Can be one of:

            - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
            - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
            - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.
    """
    do_resize = do_resize if do_resize is not None else self.do_resize
    resample = resample if resample is not None else self.resample
    do_rescale = do_rescale if do_rescale is not None else self.do_rescale
    rescale_factor = rescale_factor if rescale_factor is not None else self.rescale_factor
    do_normalize = do_normalize if do_normalize is not None else self.do_normalize
    image_mean = image_mean if image_mean is not None else self.image_mean
    image_std = image_std if image_std is not None else self.image_std
    do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb

    size = size if size is not None else self.size
    size = get_size_dict(size, default_to_square=False)

    images = make_list_of_images(images)

    validate_kwargs(captured_kwargs=kwargs.keys(), valid_processor_keys=self._valid_processor_keys)

    if not valid_images(images):
        raise ValueError(
            "Invalid image type. Must be of type PIL.Image.Image, numpy.ndarray, "
            "torch.Tensor, tf.Tensor or jax.ndarray."
        )

    validate_preprocess_arguments(
        do_rescale=do_rescale,
        rescale_factor=rescale_factor,
        do_normalize=do_normalize,
        image_mean=image_mean,
        image_std=image_std,
        do_resize=do_resize,
        size=size,
        resample=resample,
    )
    # PIL RGBA images are converted to RGB
    if do_convert_rgb:
        images = [convert_to_rgb(image) for image in images]

    # All transformations expect numpy arrays.
    images = [to_numpy_array(image) for image in images]

    if is_scaled_image(images[0]) and do_rescale:
        logger.warning_once(
            "It looks like you are trying to rescale already rescaled images. If the input"
            " images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again."
        )

    if input_data_format is None:
        # We assume that all images have the same channel dimension format.
        input_data_format = infer_channel_dimension_format(images[0])

    if do_resize:
        images = [
            self.resize(image=image, size=size, resample=resample, input_data_format=input_data_format)
            for image in images
        ]

    if do_rescale:
        images = [
            self.rescale(image=image, scale=rescale_factor, input_data_format=input_data_format)
            for image in images
        ]

    if do_normalize:
        images = [
            self.normalize(image=image, mean=image_mean, std=image_std, input_data_format=input_data_format)
            for image in images
        ]

    images = [
        to_channel_dimension_format(image, data_format, input_channel_dim=input_data_format) for image in images
    ]

    encoded_outputs = BatchFeature(data={"pixel_values": images}, tensor_type=return_tensors)

    return encoded_outputs

mindnlp.transformers.models.blip.image_processing_blip.BlipImageProcessor.resize(image, size, resample=PILImageResampling.BICUBIC, data_format=None, input_data_format=None, **kwargs)

Resize an image to (size["height"], size["width"]).

PARAMETER	DESCRIPTION
image	Image to resize. TYPE: `np.ndarray`
size	Dictionary in the format {"height": int, "width": int} specifying the size of the output image. TYPE: `Dict[str, int]`
resample	PILImageResampling filter to use when resizing the image e.g. PILImageResampling.BICUBIC. TYPE: `PILImageResampling`, *optional*, defaults to `PILImageResampling.BICUBIC` DEFAULT: BICUBIC
data_format	The channel dimension format for the output image. If unset, the channel dimension format of the input image is used. Can be one of: "channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format. "channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format. "none" or ChannelDimension.NONE: image in (height, width) format. TYPE: `ChannelDimension` or `str`, *optional* DEFAULT: None
input_data_format	The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: "channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format. "channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format. "none" or ChannelDimension.NONE: image in (height, width) format. TYPE: `ChannelDimension` or `str`, *optional* DEFAULT: None

RETURNS	DESCRIPTION
ndarray	np.ndarray: The resized image.

Source code in mindnlp/transformers/models/blip/image_processing_blip.py

def resize(
    self,
    image: np.ndarray,
    size: Dict[str, int],
    resample: PILImageResampling = PILImageResampling.BICUBIC,
    data_format: Optional[Union[str, ChannelDimension]] = None,
    input_data_format: Optional[Union[str, ChannelDimension]] = None,
    **kwargs,
) -> np.ndarray:
    """
    Resize an image to `(size["height"], size["width"])`.

    Args:
        image (`np.ndarray`):
            Image to resize.
        size (`Dict[str, int]`):
            Dictionary in the format `{"height": int, "width": int}` specifying the size of the output image.
        resample (`PILImageResampling`, *optional*, defaults to `PILImageResampling.BICUBIC`):
            `PILImageResampling` filter to use when resizing the image e.g. `PILImageResampling.BICUBIC`.
        data_format (`ChannelDimension` or `str`, *optional*):
            The channel dimension format for the output image. If unset, the channel dimension format of the input
            image is used. Can be one of:

            - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
            - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
            - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.
        input_data_format (`ChannelDimension` or `str`, *optional*):
            The channel dimension format for the input image. If unset, the channel dimension format is inferred
            from the input image. Can be one of:

            - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
            - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
            - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.

    Returns:
        `np.ndarray`: The resized image.
    """
    size = get_size_dict(size)
    if "height" not in size or "width" not in size:
        raise ValueError(f"The `size` dictionary must contain the keys `height` and `width`. Got {size.keys()}")
    output_size = (size["height"], size["width"])
    return resize(
        image,
        size=output_size,
        resample=resample,
        data_format=data_format,
        input_data_format=input_data_format,
        **kwargs,
    )

mindnlp.transformers.models.blip.processing_blip.BlipProcessor

Bases: ProcessorMixin

Constructs a BLIP processor which wraps a BERT tokenizer and BLIP image processor into a single processor.

[BlipProcessor] offers all the functionalities of [BlipImageProcessor] and [BertTokenizerFast]. See the docstring of [~BlipProcessor.__call__] and [~BlipProcessor.decode] for more information.

PARAMETER	DESCRIPTION
image_processor	An instance of [BlipImageProcessor]. The image processor is a required input. TYPE: `BlipImageProcessor`
tokenizer	An instance of ['BertTokenizerFast`]. The tokenizer is a required input. TYPE: `BertTokenizerFast`

Source code in mindnlp/transformers/models/blip/processing_blip.py

class BlipProcessor(ProcessorMixin):
    r"""
    Constructs a BLIP processor which wraps a BERT tokenizer and BLIP image processor into a single processor.

    [`BlipProcessor`] offers all the functionalities of [`BlipImageProcessor`] and [`BertTokenizerFast`]. See the
    docstring of [`~BlipProcessor.__call__`] and [`~BlipProcessor.decode`] for more information.

    Args:
        image_processor (`BlipImageProcessor`):
            An instance of [`BlipImageProcessor`]. The image processor is a required input.
        tokenizer (`BertTokenizerFast`):
            An instance of ['BertTokenizerFast`]. The tokenizer is a required input.
    """
    attributes = ["image_processor", "tokenizer"]
    image_processor_class = "BlipImageProcessor"
    tokenizer_class = ("BertTokenizer", "BertTokenizerFast")

    def __init__(self, image_processor, tokenizer):
        """
        Initializes a BlipProcessor instance.

        Args:
            self: The BlipProcessor instance itself.
            image_processor:
                An object representing the image processor used for processing images.

                - Type: object
                - Purpose: It processes the input images.
            tokenizer:
                An object representing the tokenizer used for tokenization.

                - Type: object
                - Purpose: It tokenizes the input data.
                - Restrictions: The 'return_token_type_ids' attribute of the tokenizer should be set to False.

        Returns:
            None.

        Raises:
            None.
        """
        tokenizer.return_token_type_ids = False
        super().__init__(image_processor, tokenizer)
        self.current_processor = self.image_processor

    def __call__(
        self,
        images: ImageInput = None,
        text: Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]] = None,
        add_special_tokens: bool = True,
        padding: Union[bool, str, PaddingStrategy] = False,
        truncation: Union[bool, str, TruncationStrategy] = None,
        max_length: Optional[int] = None,
        stride: int = 0,
        pad_to_multiple_of: Optional[int] = None,
        return_attention_mask: Optional[bool] = None,
        return_overflowing_tokens: bool = False,
        return_special_tokens_mask: bool = False,
        return_offsets_mapping: bool = False,
        return_token_type_ids: bool = False,
        return_length: bool = False,
        verbose: bool = True,
        return_tensors: Optional[Union[str, TensorType]] = None,
        **kwargs,
    ) -> BatchEncoding:
        """
        This method uses [`BlipImageProcessor.__call__`] method to prepare image(s) for the model, and
        [`BertTokenizerFast.__call__`] to prepare text for the model.

        Please refer to the docstring of the above two methods for more information.
        """
        if images is None and text is None:
            raise ValueError("You have to specify either images or text.")

        # Get only text
        if images is None:
            self.current_processor = self.tokenizer
            text_encoding = self.tokenizer(
                text=text,
                add_special_tokens=add_special_tokens,
                padding=padding,
                truncation=truncation,
                max_length=max_length,
                stride=stride,
                pad_to_multiple_of=pad_to_multiple_of,
                return_attention_mask=return_attention_mask,
                return_overflowing_tokens=return_overflowing_tokens,
                return_special_tokens_mask=return_special_tokens_mask,
                return_offsets_mapping=return_offsets_mapping,
                return_token_type_ids=return_token_type_ids,
                return_length=return_length,
                verbose=verbose,
                return_tensors=return_tensors,
                **kwargs,
            )
            return text_encoding

        # add pixel_values
        encoding_image_processor = self.image_processor(images, return_tensors=return_tensors)

        if text is not None:
            text_encoding = self.tokenizer(
                text=text,
                add_special_tokens=add_special_tokens,
                padding=padding,
                truncation=truncation,
                max_length=max_length,
                stride=stride,
                pad_to_multiple_of=pad_to_multiple_of,
                return_attention_mask=return_attention_mask,
                return_overflowing_tokens=return_overflowing_tokens,
                return_special_tokens_mask=return_special_tokens_mask,
                return_offsets_mapping=return_offsets_mapping,
                return_token_type_ids=return_token_type_ids,
                return_length=return_length,
                verbose=verbose,
                return_tensors=return_tensors,
                **kwargs,
            )
        else:
            text_encoding = None

        if text_encoding is not None:
            encoding_image_processor.update(text_encoding)

        return encoding_image_processor

    def batch_decode(self, *args, **kwargs):
        """
        This method forwards all its arguments to BertTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please
        refer to the docstring of this method for more information.
        """
        return self.tokenizer.batch_decode(*args, **kwargs)

    def decode(self, *args, **kwargs):
        """
        This method forwards all its arguments to BertTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to
        the docstring of this method for more information.
        """
        return self.tokenizer.decode(*args, **kwargs)

    @property
    def model_input_names(self):
        """
        This method, model_input_names, in the BlipProcessor class, retrieves the unique model input names from
        the tokenizer and image processor.

        Args:
            self (BlipProcessor): The instance of the BlipProcessor class.
                This parameter is required to access the attributes of the BlipProcessor instance.

        Returns:
            list: A list of unique model input names derived from the tokenizer and image processor.
                The list does not contain any duplicate names and is ordered as per the appearance in the
                combined tokenizer and image processor input names.

        Raises:
            None
        """
        tokenizer_input_names = self.tokenizer.model_input_names
        image_processor_input_names = self.image_processor.model_input_names
        return list(dict.fromkeys(tokenizer_input_names + image_processor_input_names))

mindnlp.transformers.models.blip.processing_blip.BlipProcessor.model_input_names property

This method, model_input_names, in the BlipProcessor class, retrieves the unique model input names from the tokenizer and image processor.

PARAMETER	DESCRIPTION
self	The instance of the BlipProcessor class. This parameter is required to access the attributes of the BlipProcessor instance. TYPE: BlipProcessor

RETURNS	DESCRIPTION
list	A list of unique model input names derived from the tokenizer and image processor. The list does not contain any duplicate names and is ordered as per the appearance in the combined tokenizer and image processor input names.

mindnlp.transformers.models.blip.processing_blip.BlipProcessor.__call__(images=None, text=None, add_special_tokens=True, padding=False, truncation=None, max_length=None, stride=0, pad_to_multiple_of=None, return_attention_mask=None, return_overflowing_tokens=False, return_special_tokens_mask=False, return_offsets_mapping=False, return_token_type_ids=False, return_length=False, verbose=True, return_tensors=None, **kwargs)

This method uses [BlipImageProcessor.__call__] method to prepare image(s) for the model, and [BertTokenizerFast.__call__] to prepare text for the model.

Please refer to the docstring of the above two methods for more information.

Source code in mindnlp/transformers/models/blip/processing_blip.py

def __call__(
    self,
    images: ImageInput = None,
    text: Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]] = None,
    add_special_tokens: bool = True,
    padding: Union[bool, str, PaddingStrategy] = False,
    truncation: Union[bool, str, TruncationStrategy] = None,
    max_length: Optional[int] = None,
    stride: int = 0,
    pad_to_multiple_of: Optional[int] = None,
    return_attention_mask: Optional[bool] = None,
    return_overflowing_tokens: bool = False,
    return_special_tokens_mask: bool = False,
    return_offsets_mapping: bool = False,
    return_token_type_ids: bool = False,
    return_length: bool = False,
    verbose: bool = True,
    return_tensors: Optional[Union[str, TensorType]] = None,
    **kwargs,
) -> BatchEncoding:
    """
    This method uses [`BlipImageProcessor.__call__`] method to prepare image(s) for the model, and
    [`BertTokenizerFast.__call__`] to prepare text for the model.

    Please refer to the docstring of the above two methods for more information.
    """
    if images is None and text is None:
        raise ValueError("You have to specify either images or text.")

    # Get only text
    if images is None:
        self.current_processor = self.tokenizer
        text_encoding = self.tokenizer(
            text=text,
            add_special_tokens=add_special_tokens,
            padding=padding,
            truncation=truncation,
            max_length=max_length,
            stride=stride,
            pad_to_multiple_of=pad_to_multiple_of,
            return_attention_mask=return_attention_mask,
            return_overflowing_tokens=return_overflowing_tokens,
            return_special_tokens_mask=return_special_tokens_mask,
            return_offsets_mapping=return_offsets_mapping,
            return_token_type_ids=return_token_type_ids,
            return_length=return_length,
            verbose=verbose,
            return_tensors=return_tensors,
            **kwargs,
        )
        return text_encoding

    # add pixel_values
    encoding_image_processor = self.image_processor(images, return_tensors=return_tensors)

    if text is not None:
        text_encoding = self.tokenizer(
            text=text,
            add_special_tokens=add_special_tokens,
            padding=padding,
            truncation=truncation,
            max_length=max_length,
            stride=stride,
            pad_to_multiple_of=pad_to_multiple_of,
            return_attention_mask=return_attention_mask,
            return_overflowing_tokens=return_overflowing_tokens,
            return_special_tokens_mask=return_special_tokens_mask,
            return_offsets_mapping=return_offsets_mapping,
            return_token_type_ids=return_token_type_ids,
            return_length=return_length,
            verbose=verbose,
            return_tensors=return_tensors,
            **kwargs,
        )
    else:
        text_encoding = None

    if text_encoding is not None:
        encoding_image_processor.update(text_encoding)

    return encoding_image_processor

mindnlp.transformers.models.blip.processing_blip.BlipProcessor.__init__(image_processor, tokenizer)

Initializes a BlipProcessor instance.

PARAMETER	DESCRIPTION
self	The BlipProcessor instance itself.
image_processor	An object representing the image processor used for processing images. Type: object Purpose: It processes the input images.
tokenizer	An object representing the tokenizer used for tokenization. Type: object Purpose: It tokenizes the input data. Restrictions: The 'return_token_type_ids' attribute of the tokenizer should be set to False.

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/blip/processing_blip.py

def __init__(self, image_processor, tokenizer):
    """
    Initializes a BlipProcessor instance.

    Args:
        self: The BlipProcessor instance itself.
        image_processor:
            An object representing the image processor used for processing images.

            - Type: object
            - Purpose: It processes the input images.
        tokenizer:
            An object representing the tokenizer used for tokenization.

            - Type: object
            - Purpose: It tokenizes the input data.
            - Restrictions: The 'return_token_type_ids' attribute of the tokenizer should be set to False.

    Returns:
        None.

    Raises:
        None.
    """
    tokenizer.return_token_type_ids = False
    super().__init__(image_processor, tokenizer)
    self.current_processor = self.image_processor

mindnlp.transformers.models.blip.processing_blip.BlipProcessor.batch_decode(*args, **kwargs)

This method forwards all its arguments to BertTokenizerFast's [~PreTrainedTokenizer.batch_decode]. Please refer to the docstring of this method for more information.

Source code in mindnlp/transformers/models/blip/processing_blip.py

def batch_decode(self, *args, **kwargs):
    """
    This method forwards all its arguments to BertTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please
    refer to the docstring of this method for more information.
    """
    return self.tokenizer.batch_decode(*args, **kwargs)

mindnlp.transformers.models.blip.processing_blip.BlipProcessor.decode(*args, **kwargs)

This method forwards all its arguments to BertTokenizerFast's [~PreTrainedTokenizer.decode]. Please refer to the docstring of this method for more information.

Source code in mindnlp/transformers/models/blip/processing_blip.py

def decode(self, *args, **kwargs):
    """
    This method forwards all its arguments to BertTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to
    the docstring of this method for more information.
    """
    return self.tokenizer.decode(*args, **kwargs)