autoformer

mindnlp.transformers.models.autoformer.configuration_autoformer.AutoformerConfig

Bases: PretrainedConfig

This is the configuration class to store the configuration of an [AutoformerModel]. It is used to instantiate an Autoformer model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Autoformer [huggingface/autoformer-tourism-monthly] architecture.

Configuration objects inherit from [PreTrainedConfig] can be used to control the model outputs. Read the documentation from [PreTrainedConfig] for more information.

PARAMETER	DESCRIPTION
prediction_length	The prediction length for the decoder. In other words, the prediction horizon of the model. TYPE: `int` DEFAULT: None
context_length	The context length for the encoder. If unset, the context length will be the same as the prediction_length. TYPE: `int`, *optional*, defaults to `prediction_length` DEFAULT: None
distribution_output	The distribution emission head for the model. Could be either "student_t", "normal" or "negative_binomial". TYPE: `string`, *optional*, defaults to `"student_t"` DEFAULT: 'student_t'
loss	The loss function for the model corresponding to the distribution_output head. For parametric distributions it is the negative log likelihood (nll) - which currently is the only supported one. TYPE: `string`, *optional*, defaults to `"nll"` DEFAULT: 'nll'
input_size	The size of the target variable which by default is 1 for univariate targets. Would be > 1 in case of multivariate targets. TYPE: `int`, *optional*, defaults to 1 DEFAULT: 1
lags_sequence	The lags of the input time series as covariates often dictated by the frequency. Default is [1, 2, 3, 4, 5, 6, 7]. TYPE: `list[int]`, *optional*, defaults to `[1, 2, 3, 4, 5, 6, 7]` DEFAULT: [1, 2, 3, 4, 5, 6, 7]
scaling	Whether to scale the input targets. TYPE: `bool`, *optional* defaults to `True` DEFAULT: True
num_time_features	The number of time features in the input time series. TYPE: `int`, *optional*, defaults to 0 DEFAULT: 0
num_dynamic_real_features	The number of dynamic real valued features. TYPE: `int`, *optional*, defaults to 0 DEFAULT: 0
num_static_categorical_features	The number of static categorical features. TYPE: `int`, *optional*, defaults to 0 DEFAULT: 0
num_static_real_features	The number of static real valued features. TYPE: `int`, *optional*, defaults to 0 DEFAULT: 0
cardinality	The cardinality (number of different values) for each of the static categorical features. Should be a list of integers, having the same length as num_static_categorical_features. Cannot be None if num_static_categorical_features is > 0. TYPE: `list[int]`, *optional* DEFAULT: None
embedding_dimension	The dimension of the embedding for each of the static categorical features. Should be a list of integers, having the same length as num_static_categorical_features. Cannot be None if num_static_categorical_features is > 0. TYPE: `list[int]`, *optional* DEFAULT: None
d_model	Dimensionality of the transformer layers. TYPE: `int`, *optional*, defaults to 64 DEFAULT: 64
encoder_layers	Number of encoder layers. TYPE: `int`, *optional*, defaults to 2 DEFAULT: 2
decoder_layers	Number of decoder layers. TYPE: `int`, *optional*, defaults to 2 DEFAULT: 2
encoder_attention_heads	Number of attention heads for each attention layer in the Transformer encoder. TYPE: `int`, *optional*, defaults to 2 DEFAULT: 2
decoder_attention_heads	Number of attention heads for each attention layer in the Transformer decoder. TYPE: `int`, *optional*, defaults to 2 DEFAULT: 2
encoder_ffn_dim	Dimension of the "intermediate" (often named feed-forward) layer in encoder. TYPE: `int`, *optional*, defaults to 32 DEFAULT: 32
decoder_ffn_dim	Dimension of the "intermediate" (often named feed-forward) layer in decoder. TYPE: `int`, *optional*, defaults to 32 DEFAULT: 32
activation_function	The non-linear activation function (function or string) in the encoder and decoder. If string, "gelu" and "relu" are supported. TYPE: `str` or `function`, *optional*, defaults to `"gelu"` DEFAULT: 'gelu'
dropout	The dropout probability for all fully connected layers in the encoder, and decoder. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
encoder_layerdrop	The dropout probability for the attention and fully connected layers for each encoder layer. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
decoder_layerdrop	The dropout probability for the attention and fully connected layers for each decoder layer. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
attention_dropout	The dropout probability for the attention probabilities. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
activation_dropout	The dropout probability used between the two layers of the feed-forward networks. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
num_parallel_samples	The number of samples to generate in parallel for each time step of inference. TYPE: `int`, *optional*, defaults to 100 DEFAULT: 100
init_std	The standard deviation of the truncated normal weight initialization distribution. TYPE: `float`, *optional*, defaults to 0.02 DEFAULT: 0.02
use_cache	Whether to use the past key/values attentions (if applicable to the model) to speed up decoding. TYPE: `bool`, *optional*, defaults to `True` DEFAULT: True
label_length	Start token length of the Autoformer decoder, which is used for direct multi-step prediction (i.e. non-autoregressive generation). TYPE: `int`, *optional*, defaults to 10 DEFAULT: 10
moving_average	The window size of the moving average. In practice, it's the kernel size in AvgPool1d of the Decomposition Layer. TYPE: `int`, defaults to 25 DEFAULT: 25
autocorrelation_factor	"Attention" (i.e. AutoCorrelation mechanism) factor which is used to find top k autocorrelations delays. It's recommended in the paper to set it to a number between 1 and 5. TYPE: `int`, defaults to 3 DEFAULT: 3

Example

```python

from transformers import AutoformerConfig, AutoformerModel ...

Initializing a default Autoformer configuration

configuration = AutoformerConfig() ...

Randomly initializing a model (with random weights) from the configuration

model = AutoformerModel(configuration) ...

Accessing the model configuration

configuration = model.config ```

Source code in mindnlp/transformers/models/autoformer/configuration_autoformer.py

class AutoformerConfig(PretrainedConfig):
    r"""
    This is the configuration class to store the configuration of an [`AutoformerModel`]. It is used to instantiate an
    Autoformer model according to the specified arguments, defining the model architecture. Instantiating a
    configuration with the defaults will yield a similar configuration to that of the Autoformer
    [huggingface/autoformer-tourism-monthly] architecture.

    Configuration objects inherit from [`PreTrainedConfig`] can be used to control the model outputs. Read the
    documentation from [`PreTrainedConfig`] for more information.

    Args:
        prediction_length (`int`):
            The prediction length for the decoder. In other words, the prediction horizon of the model.
        context_length (`int`, *optional*, defaults to `prediction_length`):
            The context length for the encoder. If unset, the context length will be the same as the
            `prediction_length`.
        distribution_output (`string`, *optional*, defaults to `"student_t"`):
            The distribution emission head for the model. Could be either "student_t", "normal" or "negative_binomial".
        loss (`string`, *optional*, defaults to `"nll"`):
            The loss function for the model corresponding to the `distribution_output` head. For parametric
            distributions it is the negative log likelihood (nll) - which currently is the only supported one.
        input_size (`int`, *optional*, defaults to 1):
            The size of the target variable which by default is 1 for univariate targets. Would be > 1 in case of
            multivariate targets.
        lags_sequence (`list[int]`, *optional*, defaults to `[1, 2, 3, 4, 5, 6, 7]`):
            The lags of the input time series as covariates often dictated by the frequency. Default is `[1, 2, 3, 4,
            5, 6, 7]`.
        scaling (`bool`, *optional* defaults to `True`):
            Whether to scale the input targets.
        num_time_features (`int`, *optional*, defaults to 0):
            The number of time features in the input time series.
        num_dynamic_real_features (`int`, *optional*, defaults to 0):
            The number of dynamic real valued features.
        num_static_categorical_features (`int`, *optional*, defaults to 0):
            The number of static categorical features.
        num_static_real_features (`int`, *optional*, defaults to 0):
            The number of static real valued features.
        cardinality (`list[int]`, *optional*):
            The cardinality (number of different values) for each of the static categorical features. Should be a list
            of integers, having the same length as `num_static_categorical_features`. Cannot be `None` if
            `num_static_categorical_features` is > 0.
        embedding_dimension (`list[int]`, *optional*):
            The dimension of the embedding for each of the static categorical features. Should be a list of integers,
            having the same length as `num_static_categorical_features`. Cannot be `None` if
            `num_static_categorical_features` is > 0.
        d_model (`int`, *optional*, defaults to 64):
            Dimensionality of the transformer layers.
        encoder_layers (`int`, *optional*, defaults to 2):
            Number of encoder layers.
        decoder_layers (`int`, *optional*, defaults to 2):
            Number of decoder layers.
        encoder_attention_heads (`int`, *optional*, defaults to 2):
            Number of attention heads for each attention layer in the Transformer encoder.
        decoder_attention_heads (`int`, *optional*, defaults to 2):
            Number of attention heads for each attention layer in the Transformer decoder.
        encoder_ffn_dim (`int`, *optional*, defaults to 32):
            Dimension of the "intermediate" (often named feed-forward) layer in encoder.
        decoder_ffn_dim (`int`, *optional*, defaults to 32):
            Dimension of the "intermediate" (often named feed-forward) layer in decoder.
        activation_function (`str` or `function`, *optional*, defaults to `"gelu"`):
            The non-linear activation function (function or string) in the encoder and decoder. If string, `"gelu"` and
            `"relu"` are supported.
        dropout (`float`, *optional*, defaults to 0.1):
            The dropout probability for all fully connected layers in the encoder, and decoder.
        encoder_layerdrop (`float`, *optional*, defaults to 0.1):
            The dropout probability for the attention and fully connected layers for each encoder layer.
        decoder_layerdrop (`float`, *optional*, defaults to 0.1):
            The dropout probability for the attention and fully connected layers for each decoder layer.
        attention_dropout (`float`, *optional*, defaults to 0.1):
            The dropout probability for the attention probabilities.
        activation_dropout (`float`, *optional*, defaults to 0.1):
            The dropout probability used between the two layers of the feed-forward networks.
        num_parallel_samples (`int`, *optional*, defaults to 100):
            The number of samples to generate in parallel for each time step of inference.
        init_std (`float`, *optional*, defaults to 0.02):
            The standard deviation of the truncated normal weight initialization distribution.
        use_cache (`bool`, *optional*, defaults to `True`):
            Whether to use the past key/values attentions (if applicable to the model) to speed up decoding.
        label_length (`int`, *optional*, defaults to 10):
            Start token length of the Autoformer decoder, which is used for direct multi-step prediction (i.e.
            non-autoregressive generation).
        moving_average (`int`, defaults to 25):
            The window size of the moving average. In practice, it's the kernel size in AvgPool1d of the Decomposition
            Layer.
        autocorrelation_factor (`int`, defaults to 3):
            "Attention" (i.e. AutoCorrelation mechanism) factor which is used to find top k autocorrelations delays.
            It's recommended in the paper to set it to a number between 1 and 5.

    Example:
        ```python
        >>> from transformers import AutoformerConfig, AutoformerModel
        ...
        >>> # Initializing a default Autoformer configuration
        >>> configuration = AutoformerConfig()
        ...
        >>> # Randomly initializing a model (with random weights) from the configuration
        >>> model = AutoformerModel(configuration)
        ...
        >>> # Accessing the model configuration
        >>> configuration = model.config
            ```
    """
    model_type = "autoformer"
    attribute_map = {
        "hidden_size": "d_model",
        "num_attention_heads": "encoder_attention_heads",
        "num_hidden_layers": "encoder_layers",
    }

    def __init__(
        self,
        prediction_length: Optional[int] = None,
        context_length: Optional[int] = None,
        distribution_output: str = "student_t",
        loss: str = "nll",
        input_size: int = 1,
        lags_sequence: List[int] = [1, 2, 3, 4, 5, 6, 7],
        # todo #pylint: disable=dangerous-default-value
        scaling: bool = True,
        num_time_features: int = 0,
        num_dynamic_real_features: int = 0,
        num_static_categorical_features: int = 0,
        num_static_real_features: int = 0,
        cardinality: Optional[List[int]] = None,
        embedding_dimension: Optional[List[int]] = None,
        d_model: int = 64,
        encoder_attention_heads: int = 2,
        decoder_attention_heads: int = 2,
        encoder_layers: int = 2,
        decoder_layers: int = 2,
        encoder_ffn_dim: int = 32,
        decoder_ffn_dim: int = 32,
        activation_function: str = "gelu",
        dropout: float = 0.1,
        encoder_layerdrop: float = 0.1,
        decoder_layerdrop: float = 0.1,
        attention_dropout: float = 0.1,
        activation_dropout: float = 0.1,
        num_parallel_samples: int = 100,
        init_std: float = 0.02,
        use_cache: bool = True,
        is_encoder_decoder=True,
        # Autoformer arguments
        label_length: int = 10,
        moving_average: int = 25,
        autocorrelation_factor: int = 3,
        **kwargs,
    ):
        """
        Initialize the AutoformerConfig.

        Args:
            prediction_length (Optional[int]): The number of time steps to predict into the future. Default is None.
            context_length (Optional[int]): The number of time steps to use as a context for prediction. Default is None.
            distribution_output (str): The type of output distribution to use. Default is 'student_t'.
            loss (str): The loss function to use. Default is 'nll'.
            input_size (int): The size of the input. Default is 1.
            lags_sequence (List[int]): The sequence of lagged time steps to use as inputs. Default is [1, 2, 3, 4, 5, 6, 7].
            scaling (bool): Whether to scale the input data. Default is True.
            num_time_features (int): The number of time features. Default is 0.
            num_dynamic_real_features (int): The number of dynamic real features. Default is 0.
            num_static_categorical_features (int): The number of static categorical features. Default is 0.
            num_static_real_features (int): The number of static real features. Default is 0.
            cardinality (Optional[List[int]]): The cardinality of each static categorical feature. Default is None.
            embedding_dimension (Optional[List[int]]): The embedding dimension of each static categorical feature. Default is None.
            d_model (int): The dimension of the model. Default is 64.
            encoder_attention_heads (int): The number of attention heads in the encoder. Default is 2.
            decoder_attention_heads (int): The number of attention heads in the decoder. Default is 2.
            encoder_layers (int): The number of layers in the encoder. Default is 2.
            decoder_layers (int): The number of layers in the decoder. Default is 2.
            encoder_ffn_dim (int): The dimension of the feed-forward network in the encoder. Default is 32.
            decoder_ffn_dim (int): The dimension of the feed-forward network in the decoder. Default is 32.
            activation_function (str): The activation function to use. Default is 'gelu'.
            dropout (float): The dropout rate. Default is 0.1.
            encoder_layerdrop (float): The layerdrop rate in the encoder. Default is 0.1.
            decoder_layerdrop (float): The layerdrop rate in the decoder. Default is 0.1.
            attention_dropout (float): The dropout rate for attention layers. Default is 0.1.
            activation_dropout (float): The dropout rate for activation layers. Default is 0.1.
            num_parallel_samples (int): The number of parallel samples. Default is 100.
            init_std (float): The standard deviation for weight initialization. Default is 0.02.
            use_cache (bool): Whether to use cache. Default is True.
            is_encoder_decoder (bool): Whether the model is an encoder-decoder model. Default is True.
            label_length (int): The length of the label. Default is 10.
            moving_average (int): The moving average factor. Default is 25.
            autocorrelation_factor (int): The autocorrelation factor. Default is 3.

        Returns:
            None

        Raises:
            ValueError:
                If `cardinality` is not None and `num_static_categorical_features` is greater than 0,
                but the lengths of `cardinality` and `num_static_categorical_features` do not match.
            ValueError:
                If `embedding_dimension` is not None and `num_static_categorical_features` is greater than 0,
                but the lengths of `embedding_dimension` and `num_static_categorical_features` do not match.
        """
        # time series specific configuration
        self.prediction_length = prediction_length
        self.context_length = context_length if context_length is not None else prediction_length
        self.distribution_output = distribution_output
        self.loss = loss
        self.input_size = input_size
        self.num_time_features = num_time_features
        self.lags_sequence = lags_sequence
        self.scaling = scaling
        self.num_dynamic_real_features = num_dynamic_real_features
        self.num_static_real_features = num_static_real_features
        self.num_static_categorical_features = num_static_categorical_features
        if cardinality is not None and num_static_categorical_features > 0:
            if len(cardinality) != num_static_categorical_features:
                raise ValueError(
                    "The cardinality should be a list of the same length as `num_static_categorical_features`"
                )
            self.cardinality = cardinality
        else:
            self.cardinality = [0]
        if embedding_dimension is not None and num_static_categorical_features > 0:
            if len(embedding_dimension) != num_static_categorical_features:
                raise ValueError(
                    "The embedding dimension should be a list of the same length as `num_static_categorical_features`"
                )
            self.embedding_dimension = embedding_dimension
        else:
            self.embedding_dimension = [min(50, (cat + 1) // 2) for cat in self.cardinality]
        self.num_parallel_samples = num_parallel_samples

        # Transformer architecture configuration
        self.feature_size = input_size * len(self.lags_sequence) + self._number_of_features
        self.d_model = d_model
        self.encoder_attention_heads = encoder_attention_heads
        self.decoder_attention_heads = decoder_attention_heads
        self.encoder_ffn_dim = encoder_ffn_dim
        self.decoder_ffn_dim = decoder_ffn_dim
        self.encoder_layers = encoder_layers
        self.decoder_layers = decoder_layers

        self.dropout = dropout
        self.attention_dropout = attention_dropout
        self.activation_dropout = activation_dropout
        self.encoder_layerdrop = encoder_layerdrop
        self.decoder_layerdrop = decoder_layerdrop

        self.activation_function = activation_function
        self.init_std = init_std

        self.use_cache = use_cache

        # Autoformer
        self.label_length = label_length
        self.moving_average = moving_average
        self.autocorrelation_factor = autocorrelation_factor

        super().__init__(is_encoder_decoder=is_encoder_decoder, **kwargs)

    @property
    def _number_of_features(self) -> int:
        """
        Returns the total number of features used in the Autoformer model.

        Args:
            self: An instance of the AutoformerConfig class.

        Returns:
            int: The total number of features used in the Autoformer model.

        Raises:
            None.
        """
        return (
            sum(self.embedding_dimension)
            + self.num_dynamic_real_features
            + self.num_time_features
            + self.num_static_real_features
            + self.input_size * 2  # the log1p(abs(loc)) and log(scale) features
        )

mindnlp.transformers.models.autoformer.configuration_autoformer.AutoformerConfig.__init__(prediction_length=None, context_length=None, distribution_output='student_t', loss='nll', input_size=1, lags_sequence=[1, 2, 3, 4, 5, 6, 7], scaling=True, num_time_features=0, num_dynamic_real_features=0, num_static_categorical_features=0, num_static_real_features=0, cardinality=None, embedding_dimension=None, d_model=64, encoder_attention_heads=2, decoder_attention_heads=2, encoder_layers=2, decoder_layers=2, encoder_ffn_dim=32, decoder_ffn_dim=32, activation_function='gelu', dropout=0.1, encoder_layerdrop=0.1, decoder_layerdrop=0.1, attention_dropout=0.1, activation_dropout=0.1, num_parallel_samples=100, init_std=0.02, use_cache=True, is_encoder_decoder=True, label_length=10, moving_average=25, autocorrelation_factor=3, **kwargs)

Initialize the AutoformerConfig.

PARAMETER	DESCRIPTION
prediction_length	The number of time steps to predict into the future. Default is None. TYPE: Optional[int] DEFAULT: None
context_length	The number of time steps to use as a context for prediction. Default is None. TYPE: Optional[int] DEFAULT: None
distribution_output	The type of output distribution to use. Default is 'student_t'. TYPE: str DEFAULT: 'student_t'
loss	The loss function to use. Default is 'nll'. TYPE: str DEFAULT: 'nll'
input_size	The size of the input. Default is 1. TYPE: int DEFAULT: 1
lags_sequence	The sequence of lagged time steps to use as inputs. Default is [1, 2, 3, 4, 5, 6, 7]. TYPE: List[int] DEFAULT: [1, 2, 3, 4, 5, 6, 7]
scaling	Whether to scale the input data. Default is True. TYPE: bool DEFAULT: True
num_time_features	The number of time features. Default is 0. TYPE: int DEFAULT: 0
num_dynamic_real_features	The number of dynamic real features. Default is 0. TYPE: int DEFAULT: 0
num_static_categorical_features	The number of static categorical features. Default is 0. TYPE: int DEFAULT: 0
num_static_real_features	The number of static real features. Default is 0. TYPE: int DEFAULT: 0
cardinality	The cardinality of each static categorical feature. Default is None. TYPE: Optional[List[int]] DEFAULT: None
embedding_dimension	The embedding dimension of each static categorical feature. Default is None. TYPE: Optional[List[int]] DEFAULT: None
d_model	The dimension of the model. Default is 64. TYPE: int DEFAULT: 64
encoder_attention_heads	The number of attention heads in the encoder. Default is 2. TYPE: int DEFAULT: 2
decoder_attention_heads	The number of attention heads in the decoder. Default is 2. TYPE: int DEFAULT: 2
encoder_layers	The number of layers in the encoder. Default is 2. TYPE: int DEFAULT: 2
decoder_layers	The number of layers in the decoder. Default is 2. TYPE: int DEFAULT: 2
encoder_ffn_dim	The dimension of the feed-forward network in the encoder. Default is 32. TYPE: int DEFAULT: 32
decoder_ffn_dim	The dimension of the feed-forward network in the decoder. Default is 32. TYPE: int DEFAULT: 32
activation_function	The activation function to use. Default is 'gelu'. TYPE: str DEFAULT: 'gelu'
dropout	The dropout rate. Default is 0.1. TYPE: float DEFAULT: 0.1
encoder_layerdrop	The layerdrop rate in the encoder. Default is 0.1. TYPE: float DEFAULT: 0.1
decoder_layerdrop	The layerdrop rate in the decoder. Default is 0.1. TYPE: float DEFAULT: 0.1
attention_dropout	The dropout rate for attention layers. Default is 0.1. TYPE: float DEFAULT: 0.1
activation_dropout	The dropout rate for activation layers. Default is 0.1. TYPE: float DEFAULT: 0.1
num_parallel_samples	The number of parallel samples. Default is 100. TYPE: int DEFAULT: 100
init_std	The standard deviation for weight initialization. Default is 0.02. TYPE: float DEFAULT: 0.02
use_cache	Whether to use cache. Default is True. TYPE: bool DEFAULT: True
is_encoder_decoder	Whether the model is an encoder-decoder model. Default is True. TYPE: bool DEFAULT: True
label_length	The length of the label. Default is 10. TYPE: int DEFAULT: 10
moving_average	The moving average factor. Default is 25. TYPE: int DEFAULT: 25
autocorrelation_factor	The autocorrelation factor. Default is 3. TYPE: int DEFAULT: 3

RETURNS	DESCRIPTION
	None

RAISES	DESCRIPTION
ValueError	If cardinality is not None and num_static_categorical_features is greater than 0, but the lengths of cardinality and num_static_categorical_features do not match.
ValueError	If embedding_dimension is not None and num_static_categorical_features is greater than 0, but the lengths of embedding_dimension and num_static_categorical_features do not match.

Source code in mindnlp/transformers/models/autoformer/configuration_autoformer.py

def __init__(
    self,
    prediction_length: Optional[int] = None,
    context_length: Optional[int] = None,
    distribution_output: str = "student_t",
    loss: str = "nll",
    input_size: int = 1,
    lags_sequence: List[int] = [1, 2, 3, 4, 5, 6, 7],
    # todo #pylint: disable=dangerous-default-value
    scaling: bool = True,
    num_time_features: int = 0,
    num_dynamic_real_features: int = 0,
    num_static_categorical_features: int = 0,
    num_static_real_features: int = 0,
    cardinality: Optional[List[int]] = None,
    embedding_dimension: Optional[List[int]] = None,
    d_model: int = 64,
    encoder_attention_heads: int = 2,
    decoder_attention_heads: int = 2,
    encoder_layers: int = 2,
    decoder_layers: int = 2,
    encoder_ffn_dim: int = 32,
    decoder_ffn_dim: int = 32,
    activation_function: str = "gelu",
    dropout: float = 0.1,
    encoder_layerdrop: float = 0.1,
    decoder_layerdrop: float = 0.1,
    attention_dropout: float = 0.1,
    activation_dropout: float = 0.1,
    num_parallel_samples: int = 100,
    init_std: float = 0.02,
    use_cache: bool = True,
    is_encoder_decoder=True,
    # Autoformer arguments
    label_length: int = 10,
    moving_average: int = 25,
    autocorrelation_factor: int = 3,
    **kwargs,
):
    """
    Initialize the AutoformerConfig.

    Args:
        prediction_length (Optional[int]): The number of time steps to predict into the future. Default is None.
        context_length (Optional[int]): The number of time steps to use as a context for prediction. Default is None.
        distribution_output (str): The type of output distribution to use. Default is 'student_t'.
        loss (str): The loss function to use. Default is 'nll'.
        input_size (int): The size of the input. Default is 1.
        lags_sequence (List[int]): The sequence of lagged time steps to use as inputs. Default is [1, 2, 3, 4, 5, 6, 7].
        scaling (bool): Whether to scale the input data. Default is True.
        num_time_features (int): The number of time features. Default is 0.
        num_dynamic_real_features (int): The number of dynamic real features. Default is 0.
        num_static_categorical_features (int): The number of static categorical features. Default is 0.
        num_static_real_features (int): The number of static real features. Default is 0.
        cardinality (Optional[List[int]]): The cardinality of each static categorical feature. Default is None.
        embedding_dimension (Optional[List[int]]): The embedding dimension of each static categorical feature. Default is None.
        d_model (int): The dimension of the model. Default is 64.
        encoder_attention_heads (int): The number of attention heads in the encoder. Default is 2.
        decoder_attention_heads (int): The number of attention heads in the decoder. Default is 2.
        encoder_layers (int): The number of layers in the encoder. Default is 2.
        decoder_layers (int): The number of layers in the decoder. Default is 2.
        encoder_ffn_dim (int): The dimension of the feed-forward network in the encoder. Default is 32.
        decoder_ffn_dim (int): The dimension of the feed-forward network in the decoder. Default is 32.
        activation_function (str): The activation function to use. Default is 'gelu'.
        dropout (float): The dropout rate. Default is 0.1.
        encoder_layerdrop (float): The layerdrop rate in the encoder. Default is 0.1.
        decoder_layerdrop (float): The layerdrop rate in the decoder. Default is 0.1.
        attention_dropout (float): The dropout rate for attention layers. Default is 0.1.
        activation_dropout (float): The dropout rate for activation layers. Default is 0.1.
        num_parallel_samples (int): The number of parallel samples. Default is 100.
        init_std (float): The standard deviation for weight initialization. Default is 0.02.
        use_cache (bool): Whether to use cache. Default is True.
        is_encoder_decoder (bool): Whether the model is an encoder-decoder model. Default is True.
        label_length (int): The length of the label. Default is 10.
        moving_average (int): The moving average factor. Default is 25.
        autocorrelation_factor (int): The autocorrelation factor. Default is 3.

    Returns:
        None

    Raises:
        ValueError:
            If `cardinality` is not None and `num_static_categorical_features` is greater than 0,
            but the lengths of `cardinality` and `num_static_categorical_features` do not match.
        ValueError:
            If `embedding_dimension` is not None and `num_static_categorical_features` is greater than 0,
            but the lengths of `embedding_dimension` and `num_static_categorical_features` do not match.
    """
    # time series specific configuration
    self.prediction_length = prediction_length
    self.context_length = context_length if context_length is not None else prediction_length
    self.distribution_output = distribution_output
    self.loss = loss
    self.input_size = input_size
    self.num_time_features = num_time_features
    self.lags_sequence = lags_sequence
    self.scaling = scaling
    self.num_dynamic_real_features = num_dynamic_real_features
    self.num_static_real_features = num_static_real_features
    self.num_static_categorical_features = num_static_categorical_features
    if cardinality is not None and num_static_categorical_features > 0:
        if len(cardinality) != num_static_categorical_features:
            raise ValueError(
                "The cardinality should be a list of the same length as `num_static_categorical_features`"
            )
        self.cardinality = cardinality
    else:
        self.cardinality = [0]
    if embedding_dimension is not None and num_static_categorical_features > 0:
        if len(embedding_dimension) != num_static_categorical_features:
            raise ValueError(
                "The embedding dimension should be a list of the same length as `num_static_categorical_features`"
            )
        self.embedding_dimension = embedding_dimension
    else:
        self.embedding_dimension = [min(50, (cat + 1) // 2) for cat in self.cardinality]
    self.num_parallel_samples = num_parallel_samples

    # Transformer architecture configuration
    self.feature_size = input_size * len(self.lags_sequence) + self._number_of_features
    self.d_model = d_model
    self.encoder_attention_heads = encoder_attention_heads
    self.decoder_attention_heads = decoder_attention_heads
    self.encoder_ffn_dim = encoder_ffn_dim
    self.decoder_ffn_dim = decoder_ffn_dim
    self.encoder_layers = encoder_layers
    self.decoder_layers = decoder_layers

    self.dropout = dropout
    self.attention_dropout = attention_dropout
    self.activation_dropout = activation_dropout
    self.encoder_layerdrop = encoder_layerdrop
    self.decoder_layerdrop = decoder_layerdrop

    self.activation_function = activation_function
    self.init_std = init_std

    self.use_cache = use_cache

    # Autoformer
    self.label_length = label_length
    self.moving_average = moving_average
    self.autocorrelation_factor = autocorrelation_factor

    super().__init__(is_encoder_decoder=is_encoder_decoder, **kwargs)

mindnlp.transformers.models.autoformer.modeling_autoformer.AUTOFORMER_PRETRAINED_MODEL_ARCHIVE_LIST = ['huggingface/autoformer-tourism-monthly'] module-attribute

mindnlp.transformers.models.autoformer.modeling_autoformer.AutoformerForPrediction

Bases: AutoformerPreTrainedModel

todo add docstring

Source code in mindnlp/transformers/models/autoformer/modeling_autoformer.py

class AutoformerForPrediction(AutoformerPreTrainedModel):
    r"""
    # todo add docstring
    """
    def __init__(self, config: AutoformerConfig):
        """
        Initializes an instance of AutoformerForPrediction.

        Args:
            self: The instance of the class.
            config (AutoformerConfig): An object containing the configuration settings for AutoformerForPrediction.

        Returns:
            None.

        Raises:
            ValueError: If the 'config.distribution_output' is not 'student_t' or 'normal'.
            ValueError: If the 'config.loss' is not 'nll'.
        """
        super().__init__(config)
        self.model = AutoformerModel(config)
        if config.distribution_output == "student_t":
            self.distribution_output = StudentTOutput(dim=config.input_size)
        elif config.distribution_output == "normal":
            self.distribution_output = NormalOutput(dim=config.input_size)
        #elif config.distribution_output == "negative_binomial":
        #    self.distribution_output = NegativeBinomialOutput(dim=config.input_size)
        else:
            raise ValueError(f"Unknown distribution output {config.distribution_output}")

        self.parameter_projection = self.distribution_output.get_parameter_projection(self.model.config.feature_size)
        self.target_shape = self.distribution_output.event_shape

        if config.loss == "nll":
            self.loss = nll
        else:
            raise ValueError(f"Unknown loss function {config.loss}")

        # Initialize weights of distribution_output and apply final processing
        self.post_init()

    def output_params(self, decoder_output):
        r"""
        #todo add docstring
        """
        return self.parameter_projection(decoder_output[:, -self.config.prediction_length :, :])

    def get_encoder(self):
        r"""
        #todo add docstring
        """
        return self.model.get_encoder()

    def get_decoder(self):
        r"""
        #todo add docstring
        """
        return self.model.get_decoder()

    def output_distribution(self, params, loc=None, scale=None, trailing_n=None):
        r"""
        #todo add docstring
        """
        sliced_params = params
        if trailing_n is not None:
            sliced_params = [p[:, -trailing_n:] for p in params]
        return self.distribution_output.distribution(sliced_params, loc=loc, scale=scale)

    def forward(
        self,
        past_values: mindspore.Tensor,
        past_time_features: mindspore.Tensor,
        past_observed_mask: mindspore.Tensor,
        static_categorical_features: Optional[mindspore.Tensor] = None,
        static_real_features: Optional[mindspore.Tensor] = None,
        future_values: Optional[mindspore.Tensor] = None,
        future_time_features: Optional[mindspore.Tensor] = None,
        future_observed_mask: Optional[mindspore.Tensor] = None,
        decoder_attention_mask: Optional[mindspore.Tensor] = None,
        head_mask: Optional[mindspore.Tensor] = None,
        decoder_head_mask: Optional[mindspore.Tensor] = None,
        cross_attn_head_mask: Optional[mindspore.Tensor] = None,
        encoder_outputs: Optional[List[mindspore.Tensor]] = None,
        past_key_values: Optional[List[mindspore.Tensor]] = None,
        output_hidden_states: Optional[bool] = None,
        output_attentions: Optional[bool] = None,
        use_cache: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Seq2SeqTSPredictionOutput, Tuple]:
        r"""
        Returns:
            Union[Seq2SeqTSPredictionOutput, Tuple]

        Example:
            ```python
            >>> from huggingface_hub import hf_hub_download
            ...
            >>> from transformers import AutoformerForPrediction
            ...
            >>> file = hf_hub_download(
            ...     repo_id="hf-internal-testing/tourism-monthly-batch", filename="train-batch.pt", repo_type="dataset"
            ... )
            >>> batch = torch.load(file)
            ...
            >>> model = AutoformerForPrediction.from_pretrained("huggingface/autoformer-tourism-monthly")
            ...
            >>> # during training, one provides both past and future values
            >>> # as well as possible additional features
            >>> outputs = model(
            ...     past_values=batch["past_values"],
            ...     past_time_features=batch["past_time_features"],
            ...     past_observed_mask=batch["past_observed_mask"],
            ...     static_categorical_features=batch["static_categorical_features"],
            ...     static_real_features=batch["static_real_features"],
            ...     future_values=batch["future_values"],
            ...     future_time_features=batch["future_time_features"],
            ... )
            ...
            >>> loss = outputs.loss
            >>> loss.backward()
            ...
            >>> # during inference, one only provides past values
            >>> # as well as possible additional features
            >>> # the model autoregressively generates future values
            >>> outputs = model.generate(
            ...     past_values=batch["past_values"],
            ...     past_time_features=batch["past_time_features"],
            ...     past_observed_mask=batch["past_observed_mask"],
            ...     static_categorical_features=batch["static_categorical_features"],
            ...     static_real_features=batch["static_real_features"],
            ...     future_time_features=batch["future_time_features"],
            ... )
            ...
            >>> mean_prediction = outputs.sequences.mean(dim=1)
            ```
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
        if future_values is not None:
            use_cache = False

        outputs = self.model(
            past_values=past_values,
            past_time_features=past_time_features,
            past_observed_mask=past_observed_mask,
            static_categorical_features=static_categorical_features,
            static_real_features=static_real_features,
            future_values=future_values,
            future_time_features=future_time_features,
            decoder_attention_mask=decoder_attention_mask,
            head_mask=head_mask,
            decoder_head_mask=decoder_head_mask,
            cross_attn_head_mask=cross_attn_head_mask,
            encoder_outputs=encoder_outputs,
            past_key_values=past_key_values,
            output_hidden_states=output_hidden_states,
            output_attentions=output_attentions,
            use_cache=use_cache,
        )

        prediction_loss = None
        params = None
        if future_values is not None:
            # outputs.last_hidden_state and trend
            # loc is 4rd last and scale is 3rd last output
            params = self.output_params(outputs[0] + outputs[1])
            distribution = self.output_distribution(params, loc=outputs[-3], scale=outputs[-2])

            loss = self.loss(distribution, future_values)

            if future_observed_mask is None:
                future_observed_mask = ops.ones_like(future_values)

            if len(self.target_shape) == 0:
                loss_weights = future_observed_mask
            else:
                loss_weights, _ = future_observed_mask.min(
                    dim=-1, keepdims=False)

            prediction_loss = weighted_average(loss, weights=loss_weights)

        if not return_dict:
            outputs = ((params,) + outputs[2:]) if params is not None else outputs[2:]
            return ((prediction_loss,) + outputs) if prediction_loss is not None else outputs

        return Seq2SeqTSPredictionOutput(
            loss=prediction_loss,
            params=params,
            past_key_values=outputs.past_key_values,
            decoder_hidden_states=outputs.decoder_hidden_states,
            decoder_attentions=outputs.decoder_attentions,
            cross_attentions=outputs.cross_attentions,
            encoder_last_hidden_state=outputs.encoder_last_hidden_state,
            encoder_hidden_states=outputs.encoder_hidden_states,
            encoder_attentions=outputs.encoder_attentions,
            loc=outputs.loc,
            scale=outputs.scale,
            static_features=outputs.static_features,
        )

    def generate(
        self,
        past_values: mindspore.Tensor,
        past_time_features: mindspore.Tensor,
        future_time_features: mindspore.Tensor,
        past_observed_mask: Optional[mindspore.Tensor] = None,
        static_categorical_features: Optional[mindspore.Tensor] = None,
        static_real_features: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
    ) -> SampleTSPredictionOutput:
        r"""
        Greedily generate sequences of sample predictions from a model with a probability distribution head.

        Parameters:
            past_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)` or `(batch_size, sequence_length, input_size)`):
                Past values of the time series, that serve as context in order to predict the future. The sequence size
                of this tensor must be larger than the `context_length` of the model, since the model will use the
                larger size to forward lag features, i.e. additional values from the past which are added in order to
                serve as "extra context".

                The `sequence_length` here is equal to `config.context_length` + `max(config.lags_sequence)`, which if
                no `lags_sequence` is configured, is equal to `config.context_length` + 7 (as by default, the largest
                look-back index in `config.lags_sequence` is 7). The property `_past_length` returns the actual length
                of the past.

                The `past_values` is what the Transformer encoder gets as input (with optional additional features,
                such as `static_categorical_features`, `static_real_features`, `past_time_features` and lags).

                Optionally, missing values need to be replaced with zeros and indicated via the `past_observed_mask`.

                For multivariate time series, the `input_size` > 1 dimension is required and corresponds to the number
                of variates in the time series per time step.
            past_time_features (`torch.FloatTensor` of shape `(batch_size, sequence_length, num_features)`):
                Required time features, which the model internally will add to `past_values`. These could be things
                like "month of year", "day of the month", etc. encoded as vectors (for instance as Fourier features).
                These could also be so-called "age" features, which basically help the model know "at which point in
                life" a time-series is. Age features have small values for distant past time steps and increase
                monotonically the more we approach the current time step. Holiday features are also a good example of
                time features.

                These features serve as the "positional encodings" of the inputs. So contrary to a model like BERT,
                where the position encodings are learned from scratch internally as parameters of the model, the Time
                Series Transformer requires to provide additional time features. The Time Series Transformer only
                learns additional embeddings for `static_categorical_features`.

                Additional dynamic real covariates can be concatenated to this tensor, with the caveat that these
                features must but known at prediction time.

                The `num_features` here is equal to `config.`num_time_features` + `config.num_dynamic_real_features`.
            future_time_features (`torch.FloatTensor` of shape `(batch_size, prediction_length, num_features)`):
                Required time features for the prediction window, which the model internally will add to sampled
                predictions. These could be things like "month of year", "day of the month", etc. encoded as vectors
                (for instance as Fourier features). These could also be so-called "age" features, which basically help
                the model know "at which point in life" a time-series is. Age features have small values for distant
                past time steps and increase monotonically the more we approach the current time step. Holiday features
                are also a good example of time features.

                These features serve as the "positional encodings" of the inputs. So contrary to a model like BERT,
                where the position encodings are learned from scratch internally as parameters of the model, the Time
                Series Transformer requires to provide additional time features. The Time Series Transformer only
                learns additional embeddings for `static_categorical_features`.

                Additional dynamic real covariates can be concatenated to this tensor, with the caveat that these
                features must but known at prediction time.

                The `num_features` here is equal to `config.`num_time_features` + `config.num_dynamic_real_features`.
            past_observed_mask (`torch.BoolTensor` of shape `(batch_size, sequence_length)` or `(batch_size, sequence_length, input_size)`, *optional*):
                Boolean mask to indicate which `past_values` were observed and which were missing. Mask values selected
                in `[0, 1]`:

                - 1 for values that are **observed**,
                - 0 for values that are **missing** (i.e. NaNs that were replaced by zeros).

            static_categorical_features (`torch.LongTensor` of shape `(batch_size, number of static categorical features)`, *optional*):
                Optional static categorical features for which the model will learn an embedding, which it will add to
                the values of the time series.

                Static categorical features are features which have the same value for all time steps (static over
                time).

                A typical example of a static categorical feature is a time series ID.
            static_real_features (`torch.FloatTensor` of shape `(batch_size, number of static real features)`, *optional*):
                Optional static real features which the model will add to the values of the time series.

                Static real features are features which have the same value for all time steps (static over time).

                A typical example of a static real feature is promotion information.
            output_attentions (`bool`, *optional*):
                Whether or not to return the attentions tensors of all attention layers.
            output_hidden_states (`bool`, *optional*):
                Whether or not to return the hidden states of all layers.

        Return:
            [`SampleTSPredictionOutput`] where the outputs `sequences` tensor will have shape `(batch_size, number of
            samples, prediction_length)` or `(batch_size, number of samples, prediction_length, input_size)` for
            multivariate predictions.
        """
        outputs = self(
            static_categorical_features=static_categorical_features,
            static_real_features=static_real_features,
            past_time_features=past_time_features,
            past_values=past_values,
            past_observed_mask=past_observed_mask,
            future_time_features=None,
            future_values=None,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            use_cache=False,
        )

        decoder = self.model.get_decoder()
        enc_last_hidden = outputs.encoder_last_hidden_state
        loc = outputs.loc
        scale = outputs.scale
        static_feat = outputs.static_features

        num_parallel_samples = self.config.num_parallel_samples
        repeated_loc = loc.repeat_interleave(repeats=num_parallel_samples, dim=0)
        repeated_scale = scale.repeat_interleave(
            repeats=num_parallel_samples, dim=0)

        repeated_past_values = (
            past_values.repeat_interleave(
                repeats=num_parallel_samples, dim=0) - repeated_loc
        ) / repeated_scale

        time_features = ops.cat((past_time_features, future_time_features), dim=1)

        expanded_static_feat = static_feat.unsqueeze(
            1).broadcast_to((-1, time_features.shape[1], -1))
        features = ops.cat((expanded_static_feat, time_features), dim=-1)
        repeated_features = features.repeat_interleave(
            repeats=num_parallel_samples, dim=0)

        repeated_enc_last_hidden = enc_last_hidden.repeat_interleave(
            repeats=num_parallel_samples, dim=0)

        lagged_sequence = self.model.get_lagged_subsequences(
            sequence=repeated_past_values, subsequences_length=self.config.context_length
        )
        lags_shape = lagged_sequence.shape
        reshaped_lagged_sequence = lagged_sequence.reshape(lags_shape[0], lags_shape[1], -1)
        seasonal_input, trend_input = self.model.decomposition_layer(reshaped_lagged_sequence)

        mean = ops.mean(reshaped_lagged_sequence, dim=1).unsqueeze(
            1).tile((1, self.config.prediction_length, 1))
        zeros = ops.zeros(reshaped_lagged_sequence.shape[0], self.config.prediction_length, reshaped_lagged_sequence.shape[2])

        decoder_input = ops.cat(
            (
                ops.cat(
                    (seasonal_input[:, -self.config.label_length:, ...], zeros), dim=1),
                repeated_features[:, -self.config.prediction_length - self.config.label_length :, ...],
            ),
            dim=-1,
        )
        trend_init = ops.cat(
            (
                ops.cat(
                    (trend_input[:, -self.config.label_length:, ...], mean), dim=1),
                repeated_features[:, -self.config.prediction_length - self.config.label_length :, ...],
            ),
            dim=-1,
        )
        decoder_outputs = decoder(
            trend=trend_init, inputs_embeds=decoder_input, encoder_hidden_states=repeated_enc_last_hidden
        )
        decoder_last_hidden = decoder_outputs.last_hidden_state
        trend = decoder_outputs.trend
        params = self.output_params(decoder_last_hidden + trend)
        distr = self.output_distribution(params, loc=repeated_loc, scale=repeated_scale)
        future_samples = distr.sample()

        return SampleTSPredictionOutput(
            sequences=future_samples.reshape(
                (-1, num_parallel_samples, self.config.prediction_length) + self.target_shape,
            )
        )

mindnlp.transformers.models.autoformer.modeling_autoformer.AutoformerForPrediction.__init__(config)

Initializes an instance of AutoformerForPrediction.

PARAMETER	DESCRIPTION
self	The instance of the class.
config	An object containing the configuration settings for AutoformerForPrediction. TYPE: AutoformerConfig

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If the 'config.distribution_output' is not 'student_t' or 'normal'.
ValueError	If the 'config.loss' is not 'nll'.

Source code in mindnlp/transformers/models/autoformer/modeling_autoformer.py

def __init__(self, config: AutoformerConfig):
    """
    Initializes an instance of AutoformerForPrediction.

    Args:
        self: The instance of the class.
        config (AutoformerConfig): An object containing the configuration settings for AutoformerForPrediction.

    Returns:
        None.

    Raises:
        ValueError: If the 'config.distribution_output' is not 'student_t' or 'normal'.
        ValueError: If the 'config.loss' is not 'nll'.
    """
    super().__init__(config)
    self.model = AutoformerModel(config)
    if config.distribution_output == "student_t":
        self.distribution_output = StudentTOutput(dim=config.input_size)
    elif config.distribution_output == "normal":
        self.distribution_output = NormalOutput(dim=config.input_size)
    #elif config.distribution_output == "negative_binomial":
    #    self.distribution_output = NegativeBinomialOutput(dim=config.input_size)
    else:
        raise ValueError(f"Unknown distribution output {config.distribution_output}")

    self.parameter_projection = self.distribution_output.get_parameter_projection(self.model.config.feature_size)
    self.target_shape = self.distribution_output.event_shape

    if config.loss == "nll":
        self.loss = nll
    else:
        raise ValueError(f"Unknown loss function {config.loss}")

    # Initialize weights of distribution_output and apply final processing
    self.post_init()

mindnlp.transformers.models.autoformer.modeling_autoformer.AutoformerForPrediction.forward(past_values, past_time_features, past_observed_mask, static_categorical_features=None, static_real_features=None, future_values=None, future_time_features=None, future_observed_mask=None, decoder_attention_mask=None, head_mask=None, decoder_head_mask=None, cross_attn_head_mask=None, encoder_outputs=None, past_key_values=None, output_hidden_states=None, output_attentions=None, use_cache=None, return_dict=None)

RETURNS	DESCRIPTION
Union[Seq2SeqTSPredictionOutput, Tuple]	Union[Seq2SeqTSPredictionOutput, Tuple]

Example

>>> from huggingface_hub import hf_hub_download
...
>>> from transformers import AutoformerForPrediction
...
>>> file = hf_hub_download(
...     repo_id="hf-internal-testing/tourism-monthly-batch", filename="train-batch.pt", repo_type="dataset"
... )
>>> batch = torch.load(file)
...
>>> model = AutoformerForPrediction.from_pretrained("huggingface/autoformer-tourism-monthly")
...
>>> # during training, one provides both past and future values
>>> # as well as possible additional features
>>> outputs = model(
...     past_values=batch["past_values"],
...     past_time_features=batch["past_time_features"],
...     past_observed_mask=batch["past_observed_mask"],
...     static_categorical_features=batch["static_categorical_features"],
...     static_real_features=batch["static_real_features"],
...     future_values=batch["future_values"],
...     future_time_features=batch["future_time_features"],
... )
...
>>> loss = outputs.loss
>>> loss.backward()
...
>>> # during inference, one only provides past values
>>> # as well as possible additional features
>>> # the model autoregressively generates future values
>>> outputs = model.generate(
...     past_values=batch["past_values"],
...     past_time_features=batch["past_time_features"],
...     past_observed_mask=batch["past_observed_mask"],
...     static_categorical_features=batch["static_categorical_features"],
...     static_real_features=batch["static_real_features"],
...     future_time_features=batch["future_time_features"],
... )
...
>>> mean_prediction = outputs.sequences.mean(dim=1)

Source code in mindnlp/transformers/models/autoformer/modeling_autoformer.py

def forward(
    self,
    past_values: mindspore.Tensor,
    past_time_features: mindspore.Tensor,
    past_observed_mask: mindspore.Tensor,
    static_categorical_features: Optional[mindspore.Tensor] = None,
    static_real_features: Optional[mindspore.Tensor] = None,
    future_values: Optional[mindspore.Tensor] = None,
    future_time_features: Optional[mindspore.Tensor] = None,
    future_observed_mask: Optional[mindspore.Tensor] = None,
    decoder_attention_mask: Optional[mindspore.Tensor] = None,
    head_mask: Optional[mindspore.Tensor] = None,
    decoder_head_mask: Optional[mindspore.Tensor] = None,
    cross_attn_head_mask: Optional[mindspore.Tensor] = None,
    encoder_outputs: Optional[List[mindspore.Tensor]] = None,
    past_key_values: Optional[List[mindspore.Tensor]] = None,
    output_hidden_states: Optional[bool] = None,
    output_attentions: Optional[bool] = None,
    use_cache: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Seq2SeqTSPredictionOutput, Tuple]:
    r"""
    Returns:
        Union[Seq2SeqTSPredictionOutput, Tuple]

    Example:
        ```python
        >>> from huggingface_hub import hf_hub_download
        ...
        >>> from transformers import AutoformerForPrediction
        ...
        >>> file = hf_hub_download(
        ...     repo_id="hf-internal-testing/tourism-monthly-batch", filename="train-batch.pt", repo_type="dataset"
        ... )
        >>> batch = torch.load(file)
        ...
        >>> model = AutoformerForPrediction.from_pretrained("huggingface/autoformer-tourism-monthly")
        ...
        >>> # during training, one provides both past and future values
        >>> # as well as possible additional features
        >>> outputs = model(
        ...     past_values=batch["past_values"],
        ...     past_time_features=batch["past_time_features"],
        ...     past_observed_mask=batch["past_observed_mask"],
        ...     static_categorical_features=batch["static_categorical_features"],
        ...     static_real_features=batch["static_real_features"],
        ...     future_values=batch["future_values"],
        ...     future_time_features=batch["future_time_features"],
        ... )
        ...
        >>> loss = outputs.loss
        >>> loss.backward()
        ...
        >>> # during inference, one only provides past values
        >>> # as well as possible additional features
        >>> # the model autoregressively generates future values
        >>> outputs = model.generate(
        ...     past_values=batch["past_values"],
        ...     past_time_features=batch["past_time_features"],
        ...     past_observed_mask=batch["past_observed_mask"],
        ...     static_categorical_features=batch["static_categorical_features"],
        ...     static_real_features=batch["static_real_features"],
        ...     future_time_features=batch["future_time_features"],
        ... )
        ...
        >>> mean_prediction = outputs.sequences.mean(dim=1)
        ```
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    if future_values is not None:
        use_cache = False

    outputs = self.model(
        past_values=past_values,
        past_time_features=past_time_features,
        past_observed_mask=past_observed_mask,
        static_categorical_features=static_categorical_features,
        static_real_features=static_real_features,
        future_values=future_values,
        future_time_features=future_time_features,
        decoder_attention_mask=decoder_attention_mask,
        head_mask=head_mask,
        decoder_head_mask=decoder_head_mask,
        cross_attn_head_mask=cross_attn_head_mask,
        encoder_outputs=encoder_outputs,
        past_key_values=past_key_values,
        output_hidden_states=output_hidden_states,
        output_attentions=output_attentions,
        use_cache=use_cache,
    )

    prediction_loss = None
    params = None
    if future_values is not None:
        # outputs.last_hidden_state and trend
        # loc is 4rd last and scale is 3rd last output
        params = self.output_params(outputs[0] + outputs[1])
        distribution = self.output_distribution(params, loc=outputs[-3], scale=outputs[-2])

        loss = self.loss(distribution, future_values)

        if future_observed_mask is None:
            future_observed_mask = ops.ones_like(future_values)

        if len(self.target_shape) == 0:
            loss_weights = future_observed_mask
        else:
            loss_weights, _ = future_observed_mask.min(
                dim=-1, keepdims=False)

        prediction_loss = weighted_average(loss, weights=loss_weights)

    if not return_dict:
        outputs = ((params,) + outputs[2:]) if params is not None else outputs[2:]
        return ((prediction_loss,) + outputs) if prediction_loss is not None else outputs

    return Seq2SeqTSPredictionOutput(
        loss=prediction_loss,
        params=params,
        past_key_values=outputs.past_key_values,
        decoder_hidden_states=outputs.decoder_hidden_states,
        decoder_attentions=outputs.decoder_attentions,
        cross_attentions=outputs.cross_attentions,
        encoder_last_hidden_state=outputs.encoder_last_hidden_state,
        encoder_hidden_states=outputs.encoder_hidden_states,
        encoder_attentions=outputs.encoder_attentions,
        loc=outputs.loc,
        scale=outputs.scale,
        static_features=outputs.static_features,
    )

mindnlp.transformers.models.autoformer.modeling_autoformer.AutoformerForPrediction.generate(past_values, past_time_features, future_time_features, past_observed_mask=None, static_categorical_features=None, static_real_features=None, output_attentions=None, output_hidden_states=None)

Greedily generate sequences of sample predictions from a model with a probability distribution head.

PARAMETER	DESCRIPTION
past_values	Past values of the time series, that serve as context in order to predict the future. The sequence size of this tensor must be larger than the context_length of the model, since the model will use the larger size to forward lag features, i.e. additional values from the past which are added in order to serve as "extra context". The sequence_length here is equal to config.context_length + max(config.lags_sequence), which if no lags_sequence is configured, is equal to config.context_length + 7 (as by default, the largest look-back index in config.lags_sequence is 7). The property _past_length returns the actual length of the past. The past_values is what the Transformer encoder gets as input (with optional additional features, such as static_categorical_features, static_real_features, past_time_features and lags). Optionally, missing values need to be replaced with zeros and indicated via the past_observed_mask. For multivariate time series, the input_size > 1 dimension is required and corresponds to the number of variates in the time series per time step. TYPE: `torch.FloatTensor` of shape `(batch_size, sequence_length)` or `(batch_size, sequence_length, input_size)`
past_time_features	Required time features, which the model internally will add to past_values. These could be things like "month of year", "day of the month", etc. encoded as vectors (for instance as Fourier features). These could also be so-called "age" features, which basically help the model know "at which point in life" a time-series is. Age features have small values for distant past time steps and increase monotonically the more we approach the current time step. Holiday features are also a good example of time features. These features serve as the "positional encodings" of the inputs. So contrary to a model like BERT, where the position encodings are learned from scratch internally as parameters of the model, the Time Series Transformer requires to provide additional time features. The Time Series Transformer only learns additional embeddings for static_categorical_features. Additional dynamic real covariates can be concatenated to this tensor, with the caveat that these features must but known at prediction time. The num_features here is equal to config.num_time_features+config.num_dynamic_real_features`. TYPE: `torch.FloatTensor` of shape `(batch_size, sequence_length, num_features)`
future_time_features	Required time features for the prediction window, which the model internally will add to sampled predictions. These could be things like "month of year", "day of the month", etc. encoded as vectors (for instance as Fourier features). These could also be so-called "age" features, which basically help the model know "at which point in life" a time-series is. Age features have small values for distant past time steps and increase monotonically the more we approach the current time step. Holiday features are also a good example of time features. These features serve as the "positional encodings" of the inputs. So contrary to a model like BERT, where the position encodings are learned from scratch internally as parameters of the model, the Time Series Transformer requires to provide additional time features. The Time Series Transformer only learns additional embeddings for static_categorical_features. Additional dynamic real covariates can be concatenated to this tensor, with the caveat that these features must but known at prediction time. The num_features here is equal to config.num_time_features+config.num_dynamic_real_features`. TYPE: `torch.FloatTensor` of shape `(batch_size, prediction_length, num_features)`
past_observed_mask	Boolean mask to indicate which past_values were observed and which were missing. Mask values selected in [0, 1]: 1 for values that are observed, 0 for values that are missing (i.e. NaNs that were replaced by zeros). TYPE: `torch.BoolTensor` of shape `(batch_size, sequence_length)` or `(batch_size, sequence_length, input_size)`, *optional* DEFAULT: None
static_categorical_features	Optional static categorical features for which the model will learn an embedding, which it will add to the values of the time series. Static categorical features are features which have the same value for all time steps (static over time). A typical example of a static categorical feature is a time series ID. TYPE: `torch.LongTensor` of shape `(batch_size, number of static categorical features)`, *optional* DEFAULT: None
static_real_features	Optional static real features which the model will add to the values of the time series. Static real features are features which have the same value for all time steps (static over time). A typical example of a static real feature is promotion information. TYPE: `torch.FloatTensor` of shape `(batch_size, number of static real features)`, *optional* DEFAULT: None
output_attentions	Whether or not to return the attentions tensors of all attention layers. TYPE: `bool`, *optional* DEFAULT: None
output_hidden_states	Whether or not to return the hidden states of all layers. TYPE: `bool`, *optional* DEFAULT: None

Return

[SampleTSPredictionOutput] where the outputs sequences tensor will have shape (batch_size, number of samples, prediction_length) or (batch_size, number of samples, prediction_length, input_size) for multivariate predictions.

Source code in mindnlp/transformers/models/autoformer/modeling_autoformer.py

def generate(
    self,
    past_values: mindspore.Tensor,
    past_time_features: mindspore.Tensor,
    future_time_features: mindspore.Tensor,
    past_observed_mask: Optional[mindspore.Tensor] = None,
    static_categorical_features: Optional[mindspore.Tensor] = None,
    static_real_features: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
) -> SampleTSPredictionOutput:
    r"""
    Greedily generate sequences of sample predictions from a model with a probability distribution head.

    Parameters:
        past_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)` or `(batch_size, sequence_length, input_size)`):
            Past values of the time series, that serve as context in order to predict the future. The sequence size
            of this tensor must be larger than the `context_length` of the model, since the model will use the
            larger size to forward lag features, i.e. additional values from the past which are added in order to
            serve as "extra context".

            The `sequence_length` here is equal to `config.context_length` + `max(config.lags_sequence)`, which if
            no `lags_sequence` is configured, is equal to `config.context_length` + 7 (as by default, the largest
            look-back index in `config.lags_sequence` is 7). The property `_past_length` returns the actual length
            of the past.

            The `past_values` is what the Transformer encoder gets as input (with optional additional features,
            such as `static_categorical_features`, `static_real_features`, `past_time_features` and lags).

            Optionally, missing values need to be replaced with zeros and indicated via the `past_observed_mask`.

            For multivariate time series, the `input_size` > 1 dimension is required and corresponds to the number
            of variates in the time series per time step.
        past_time_features (`torch.FloatTensor` of shape `(batch_size, sequence_length, num_features)`):
            Required time features, which the model internally will add to `past_values`. These could be things
            like "month of year", "day of the month", etc. encoded as vectors (for instance as Fourier features).
            These could also be so-called "age" features, which basically help the model know "at which point in
            life" a time-series is. Age features have small values for distant past time steps and increase
            monotonically the more we approach the current time step. Holiday features are also a good example of
            time features.

            These features serve as the "positional encodings" of the inputs. So contrary to a model like BERT,
            where the position encodings are learned from scratch internally as parameters of the model, the Time
            Series Transformer requires to provide additional time features. The Time Series Transformer only
            learns additional embeddings for `static_categorical_features`.

            Additional dynamic real covariates can be concatenated to this tensor, with the caveat that these
            features must but known at prediction time.

            The `num_features` here is equal to `config.`num_time_features` + `config.num_dynamic_real_features`.
        future_time_features (`torch.FloatTensor` of shape `(batch_size, prediction_length, num_features)`):
            Required time features for the prediction window, which the model internally will add to sampled
            predictions. These could be things like "month of year", "day of the month", etc. encoded as vectors
            (for instance as Fourier features). These could also be so-called "age" features, which basically help
            the model know "at which point in life" a time-series is. Age features have small values for distant
            past time steps and increase monotonically the more we approach the current time step. Holiday features
            are also a good example of time features.

            These features serve as the "positional encodings" of the inputs. So contrary to a model like BERT,
            where the position encodings are learned from scratch internally as parameters of the model, the Time
            Series Transformer requires to provide additional time features. The Time Series Transformer only
            learns additional embeddings for `static_categorical_features`.

            Additional dynamic real covariates can be concatenated to this tensor, with the caveat that these
            features must but known at prediction time.

            The `num_features` here is equal to `config.`num_time_features` + `config.num_dynamic_real_features`.
        past_observed_mask (`torch.BoolTensor` of shape `(batch_size, sequence_length)` or `(batch_size, sequence_length, input_size)`, *optional*):
            Boolean mask to indicate which `past_values` were observed and which were missing. Mask values selected
            in `[0, 1]`:

            - 1 for values that are **observed**,
            - 0 for values that are **missing** (i.e. NaNs that were replaced by zeros).

        static_categorical_features (`torch.LongTensor` of shape `(batch_size, number of static categorical features)`, *optional*):
            Optional static categorical features for which the model will learn an embedding, which it will add to
            the values of the time series.

            Static categorical features are features which have the same value for all time steps (static over
            time).

            A typical example of a static categorical feature is a time series ID.
        static_real_features (`torch.FloatTensor` of shape `(batch_size, number of static real features)`, *optional*):
            Optional static real features which the model will add to the values of the time series.

            Static real features are features which have the same value for all time steps (static over time).

            A typical example of a static real feature is promotion information.
        output_attentions (`bool`, *optional*):
            Whether or not to return the attentions tensors of all attention layers.
        output_hidden_states (`bool`, *optional*):
            Whether or not to return the hidden states of all layers.

    Return:
        [`SampleTSPredictionOutput`] where the outputs `sequences` tensor will have shape `(batch_size, number of
        samples, prediction_length)` or `(batch_size, number of samples, prediction_length, input_size)` for
        multivariate predictions.
    """
    outputs = self(
        static_categorical_features=static_categorical_features,
        static_real_features=static_real_features,
        past_time_features=past_time_features,
        past_values=past_values,
        past_observed_mask=past_observed_mask,
        future_time_features=None,
        future_values=None,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        use_cache=False,
    )

    decoder = self.model.get_decoder()
    enc_last_hidden = outputs.encoder_last_hidden_state
    loc = outputs.loc
    scale = outputs.scale
    static_feat = outputs.static_features

    num_parallel_samples = self.config.num_parallel_samples
    repeated_loc = loc.repeat_interleave(repeats=num_parallel_samples, dim=0)
    repeated_scale = scale.repeat_interleave(
        repeats=num_parallel_samples, dim=0)

    repeated_past_values = (
        past_values.repeat_interleave(
            repeats=num_parallel_samples, dim=0) - repeated_loc
    ) / repeated_scale

    time_features = ops.cat((past_time_features, future_time_features), dim=1)

    expanded_static_feat = static_feat.unsqueeze(
        1).broadcast_to((-1, time_features.shape[1], -1))
    features = ops.cat((expanded_static_feat, time_features), dim=-1)
    repeated_features = features.repeat_interleave(
        repeats=num_parallel_samples, dim=0)

    repeated_enc_last_hidden = enc_last_hidden.repeat_interleave(
        repeats=num_parallel_samples, dim=0)

    lagged_sequence = self.model.get_lagged_subsequences(
        sequence=repeated_past_values, subsequences_length=self.config.context_length
    )
    lags_shape = lagged_sequence.shape
    reshaped_lagged_sequence = lagged_sequence.reshape(lags_shape[0], lags_shape[1], -1)
    seasonal_input, trend_input = self.model.decomposition_layer(reshaped_lagged_sequence)

    mean = ops.mean(reshaped_lagged_sequence, dim=1).unsqueeze(
        1).tile((1, self.config.prediction_length, 1))
    zeros = ops.zeros(reshaped_lagged_sequence.shape[0], self.config.prediction_length, reshaped_lagged_sequence.shape[2])

    decoder_input = ops.cat(
        (
            ops.cat(
                (seasonal_input[:, -self.config.label_length:, ...], zeros), dim=1),
            repeated_features[:, -self.config.prediction_length - self.config.label_length :, ...],
        ),
        dim=-1,
    )
    trend_init = ops.cat(
        (
            ops.cat(
                (trend_input[:, -self.config.label_length:, ...], mean), dim=1),
            repeated_features[:, -self.config.prediction_length - self.config.label_length :, ...],
        ),
        dim=-1,
    )
    decoder_outputs = decoder(
        trend=trend_init, inputs_embeds=decoder_input, encoder_hidden_states=repeated_enc_last_hidden
    )
    decoder_last_hidden = decoder_outputs.last_hidden_state
    trend = decoder_outputs.trend
    params = self.output_params(decoder_last_hidden + trend)
    distr = self.output_distribution(params, loc=repeated_loc, scale=repeated_scale)
    future_samples = distr.sample()

    return SampleTSPredictionOutput(
        sequences=future_samples.reshape(
            (-1, num_parallel_samples, self.config.prediction_length) + self.target_shape,
        )
    )

mindnlp.transformers.models.autoformer.modeling_autoformer.AutoformerForPrediction.get_decoder()

todo add docstring

Source code in mindnlp/transformers/models/autoformer/modeling_autoformer.py

def get_decoder(self):
    r"""
    #todo add docstring
    """
    return self.model.get_decoder()

mindnlp.transformers.models.autoformer.modeling_autoformer.AutoformerForPrediction.get_encoder()

todo add docstring

Source code in mindnlp/transformers/models/autoformer/modeling_autoformer.py

def get_encoder(self):
    r"""
    #todo add docstring
    """
    return self.model.get_encoder()

mindnlp.transformers.models.autoformer.modeling_autoformer.AutoformerForPrediction.output_distribution(params, loc=None, scale=None, trailing_n=None)

todo add docstring

Source code in mindnlp/transformers/models/autoformer/modeling_autoformer.py

def output_distribution(self, params, loc=None, scale=None, trailing_n=None):
    r"""
    #todo add docstring
    """
    sliced_params = params
    if trailing_n is not None:
        sliced_params = [p[:, -trailing_n:] for p in params]
    return self.distribution_output.distribution(sliced_params, loc=loc, scale=scale)

mindnlp.transformers.models.autoformer.modeling_autoformer.AutoformerForPrediction.output_params(decoder_output)

todo add docstring

Source code in mindnlp/transformers/models/autoformer/modeling_autoformer.py

def output_params(self, decoder_output):
    r"""
    #todo add docstring
    """
    return self.parameter_projection(decoder_output[:, -self.config.prediction_length :, :])

mindnlp.transformers.models.autoformer.modeling_autoformer.AutoformerPreTrainedModel

Bases: PreTrainedModel

todo add docstring

Source code in mindnlp/transformers/models/autoformer/modeling_autoformer.py

class AutoformerPreTrainedModel(PreTrainedModel):
    r"""
    #todo add docstring
    """
    config_class = AutoformerConfig
    base_model_prefix = "model"
    main_input_name = "past_values"

    def _init_weights(self, cell):
        """
        Initializes the weights of the given cell.

        Args:
            self (AutoformerPreTrainedModel): The instance of the AutoformerPreTrainedModel class.
            cell: The cell whose weights need to be initialized.

        Returns:
            None: This method initializes the weights of the given cell in-place.

        Raises:
            None.
        """
        std = self.config.init_std
        if isinstance(cell, (nn.Linear, nn.Conv1d)):
            cell.weight.set_data(initializer(Normal(std),
                                             cell.weight.shape,
                                             cell.weight.dtype))
            if cell.bias is not None:
                cell.bias.set_data(initializer(
                    'zeros', cell.bias.shape, cell.bias.dtype))
        elif isinstance(cell, AutoformerSinusoidalPositionalEmbedding):
            pass
        elif isinstance(cell, nn.Embedding):
            cell.weight.set_data(initializer(Normal(std),
                                                      cell.weight.shape,
                                                      cell.weight.dtype))