should
refer to this superclass for more information regarding those methods.

Args:
    vocab_file (`str`):
        [SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
        contains the vocabulary necessary to instantiate a tokenizer.
    pad_token (`str`, *optional*, defaults to `"<pad>"`):
        The token used for padding, for example when batching sequences of different lengths.
    eos_token (`str`, *optional*, defaults to `"</s>"`):
        The end of sequence token.

        <Tip>

        When building a sequence using special tokens, this is not the token that is used for the end of sequence.
        The token used is the `sep_token`.

        </Tip>

    unk_token (`str`, *optional*, defaults to `"<unk>"`):
        The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
        token instead.
    mask_token (`str`, *optional*, defaults to `"<mask_2>"`):
        The token used for masking single token values. This is the token used when training this model with masked
        language modeling (MLM). This is the token that the PEGASUS encoder will try to predict during pretraining.
        It corresponds to *[MASK2]* in [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive
        Summarization](https://huggingface.co/papers/1912.08777).
    mask_token_sent (`str`, *optional*, defaults to `"<mask_1>"`):
        The token used for masking whole target sentences. This is the token used when training this model with gap
        sentences generation (GSG). This is the sentence that the PEGASUS decoder will try to predict during
        pretraining. It corresponds to *[MASK1]* in [PEGASUS: Pre-training with Extracted Gap-sentences for
        Abstractive Summarization](https://huggingface.co/papers/1912.08777).
    additional_special_tokens (`List[str]`, *optional*):
        Additional special tokens used by the tokenizer. If no additional_special_tokens are provided <mask_2> and
        <unk_2, ..., unk_102> are used as additional special tokens corresponding to the [original PEGASUS
        tokenizer](https://github.com/google-research/pegasus/blob/939830367bcf411193d2b5eca2f2f90f3f9260ca/pegasus/ops/pretrain_parsing_ops.cc#L66)
        that uses the tokens 2 - 104 only for pretraining
Ú	input_idsÚattention_maskc