cCiD;SrSSKJrJr SSKrSSKJr SSKJ r J r SSK J r J r JrJrJrJr SSKJrJrJr SS KJr \R0"\5rS r\"S 5Vs/sH nS US S3PM sn\"S5Vs/sH nSUSS3PM sn-r"SS\5r"SS\ 5r"SS\SS9r S\!4Sjr"Sr#Sr$Sr%"SS \5r&S /r'gs snfs snf)!z Processor class for PaliGemma. )OptionalUnionN) BatchFeature) ImageInputis_valid_image) ImagesKwargsMultiModalDataProcessingKwargsProcessorMixin TextKwargsUnpack) AddedTokenPreTokenizedInput TextInput)loggingziz4>z3c@\rSrSr%\\\\\\\\4\ S'Sr g)PaliGemmaTextKwargs+suffixN) __name__ __module__ __qualname____firstlineno__rrrrlist__annotations____static_attributes__rl/home/james-whalen/.local/lib/python3.13/site-packages/transformers/models/paligemma/processing_paligemma.pyrr+s& U9&7i$O`Jaab ccr!rc&\rSrSr%\\\S'Srg)PaliGemmaImagesKwargs/do_convert_rgbrN)rrrrrboolrr rr!r"r$r$/s TN"r!r$c>\rSrSr%\\S'\\S'SSS.SS0S.rS rg ) PaliGemmaProcessorKwargs3 text_kwargs images_kwargsF)paddingreturn_mm_token_type_ids data_formatchannels_first)r+r,rN) rrrrrrr$ _defaultsr rr!r"r)r)3s/$$(((- + Ir!r)F)totalreturncR[U[5=(a URS5$)Nhttp) isinstancestr startswith)vals r"is_urlr:Bs c3  :CNN6$::r!c<[U5=(d [U5$N)r:relems r"is_image_or_image_urlr?Gs $< />$//r!cF[U[5=(d [U5$r<)r6r7r?r=s r"_is_str_or_imagerAKs dS " A&;D&AAr!cX2-U-UUS3$)a Builds a string from the input prompt and image tokens. For example, for the call: build_string_from_input( prompt="Prefix str" bos_token="", image_seq_len=3, image_token="", ) The output will be: "Initial str" Args: prompt (`list[Union[str, ImageInput]]`): The input prompt. bos_token (`str`): The beginning of sentence token. image_seq_len (`int`): The length of the image sequence. image_token (`str`): The image token. num_images (`int`): Number of images in the prompt.  rprompt bos_token image_seq_len image_token num_imagess r"build_string_from_inputrJOs"&)J6 7 {6(" MMr!c ^\rSrSrSrSS/rSrSrSU4SjjrSS\ \ S \ \ \ \\ \\ 4S \\S \4S jjrSS jrSrU=r$)PaliGemmaProcessorea Constructs a PaliGemma processor which wraps a PaliGemma image processor and a PaliGemma tokenizer into a single processor. [`PaliGemmaProcessor`] offers all the functionalities of [`SiglipImageProcessor`] and [`GemmaTokenizerFast`]. See the [`~PaliGemmaProcessor.__call__`] and [`~PaliGemmaProcessor.decode`] for more information. Args: image_processor ([`SiglipImageProcessor`], *optional*): The image processor is a required input. tokenizer ([`GemmaTokenizerFast`], *optional*): The tokenizer is a required input. chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string. image_processor tokenizer)SiglipImageProcessorSiglipImageProcessorFast)GemmaTokenizerGemmaTokenizerFastc >[US5(d [S5eURUl[US5(dK[[SSS9nSU/0nUR U5 UR [5Ul[UlO"URUlURUlUR[5 SUl SUl [TU]9XUS9 g) Nimage_seq_lengthz;Image processor is missing an `image_seq_length` attribute.rHFT) normalizedspecialadditional_special_tokens) chat_template)hasattr ValueErrorrUr IMAGE_TOKENadd_special_tokensconvert_tokens_to_idsimage_token_idrH add_tokens EXTRA_TOKENS add_bos_token add_eos_tokensuper__init__)selfrNrOrYkwargsrH tokens_to_add __class__s r"rePaliGemmaProcessor.__init__ys(:;;Z[ [ / @ @y-00$[UDQK8;-HM  ( ( 7"+"A"A+"ND *D "+":":D (44D \*"' "'  =Qr!imagestextrgr3c & UR"[4SURR0UD6nUSR SS5nUSLnUc [ S5eUc[ RS5 Sn[U5(aU/nO)[U[5(a[US5(aUGb\UGbX[S U55(Gd[ RS 5 [U[5(aQ[U[5(a<[U5[U5:wa$[ S [U5S [U5S 35e[U5(aU//nO[U[[45(a&[US5(aUV s/sHo/PM nn OZ[U[[45(a4[US[[45(a[USS5(d [ S5e[!X!5V V s/sHTup[#U URR$UR&[([U [5(a [U 5OSS9PMV n n n O/n UHnUR+[([(UR&-5nUR-[(5nUS:waU[[(5-OSnUSUURR$-UUS-nU R/U5 M U Vs/sHoS3PM n nUb[U5(aU/nUb)UVs/sHnUURR0-PM nnUR2"U40USD6SnUSR SS5nUSR SS5nUR"W 4UUS.USD6nUR5U US/S9 0UESU0EnU(aK[6R8"US5nSU[6R8"US5S:H'UR;SU05 U(aW[6R8"US5n[6R<"US5nSUUUR>:H'URA5US'[CUUS9$s sn fs sn n fs snfs snf) a Main method to prepare for the model one or several sequences(s) and image(s). This method forwards the `text` and `kwargs` arguments to GemmaTokenizerFast's [`~GemmaTokenizerFast.__call__`] if `text` is not `None` to encode the text. To prepare the image(s), this method forwards the `images` and `kwargs` arguments to SiglipImageProcessor's [`~SiglipImageProcessor.__call__`] if `images` is not `None`. Please refer to the docstring of the above two methods for more information. The usage for PaliGemma fine-tuning preparation is slightly different than usual. suffix passed are suffixes to the prompt in `text`, and will be placed after the prompt. This is because attention is handled differently for the prefix and the suffix. For instance, ```python image = PIL_cow_image prompt = "answer en Where is the cow standing?" suffix = "on the beach" inputs = processor(text=prompt, images=image, suffix=suffix) ``` Here `inputs` will contain the `input_ids` and `token_type_ids` that follow ```python inputs["input_ids"][:, 256:] # tensor([[ 2, 6006, 603, 573, 13910, 9980, 235336, 108, 477, 573, 8318]]) inputs["token_type_ids"][:, 256:] tensor([[0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1]]) ``` Meaning the last three tokens are of "label" ("suffix") type while the other ones are of "prefix" type. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. In case of a NumPy array/PyTorch tensor, each image should be of shape (C, H, W), where C is a number of channels, H and W are image height and width. text (`str`, `list[str]`, `list[list[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. suffix (`str`, `list[str]`, `list[list[str]]`): The suffixes or batch of suffixes to be encoded. Only necessary for finetuning. See https://github.com/google-research/big_vision/blob/main/big_vision/configs/proj/paligemma/README.md for more information. If your prompt is " What is on the image", the suffix corresponds to the expected prediction "a cow sitting on a bench". Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. If `suffix` is provided, the `input_ids` will also contain the suffix input ids. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. - **labels** -- Labels compatible with training if `suffix` is not None tokenizer_init_kwargsr+rNzF`images` are expected as arguments to a `PaliGemmaProcessor` instance.z]You are using PaliGemma without a text prefix. It will perform as a picture-captioning model.rc34# UHn[U;v M g7fr<)r\).0samples r" .PaliGemmaProcessor.__call__..s@4{f,4saLYou are passing both `text` and `images` to `PaliGemmaProcessor`. The processor expects special image tokens in the text, as many tokens as there are images per each text. It is recommended to add `` tokens in the very beginning of your text. For this call, we will infer how many images each text has and add special tokens.z Received z images for zK prompts. Each prompt should be associated with an image or list of images.zAimages must be an image, list of images or list of list of imagesrDrCr, pixel_valuesreturn_tensorsr.) text_pairreturn_token_type_idsimage) modalities input_idsitoken_type_idslabelsmm_token_type_ids)data tensor_type)" _merge_kwargsr)rO init_kwargspopr[logger warning_oncerAr6ranywarninglenrtupleziprJrFrUr\replacerfindappend eos_tokenrN_check_special_mm_tokensnparrayupdate zeros_liker_tolistr)rfrkrlaudiovideosrg output_kwargsrrzr{rE image_list input_stringsexpanded_samplesrrexpanded_samplebos_rfind_index bos_indexsfxrwrxr.inputs return_datar array_idsrs r"__call__PaliGemmaProcessor.__call__sD** $ "&.."<"<  }-11(DA &d 2 >ef f <   o D D ! !6D d # #(8a(A(A    2@4@@@<dD))j.F.F6{c$i/('F }LT LWX "&))%hZFu 66>&QR);T;T39:6%g6F:Fve}55"6!9tUm<<&vay|44$%hii/2$.? !/@*,%"&..":":&*&;&;$/6@T6R6R3z?XY /@ ! $& "F&,nn[+PTPePeBe&fO&5&;&;K&HOFUY[F[#k2B BabI' 3dnn6N6NNQ`ajakQll$%++O<#>N N=M682=M N  "26":":XF  @FGcDNN444FG++FUmO6TUVde &}599:JDQ#0#?#C#CD^`d#e   "7 M*   %%mV %R>>> XXf[12F>BF288F#345: ;   &1 2 #[!9:I " k+.F G BC i4+>+>> ?/@/G/G/IK+ ,.IIw; !(!O Hs Q>AR<R )#Rc 0nUb;UR/[U5-nS/[U5-nURXES.5 [S0UD6$)ax Computes the number of placeholder tokens needed for multimodal inputs with the given sizes. Args: image_sizes (list[list[str]], *optional*): The input sizes formatted as (height, width) per each image. Returns: `MultiModalData`: A `MultiModalData` object holding number of tokens per each of the provided input modalities, along with other useful data. ru)num_image_tokensnum_image_patchesr)rUrrr )rf image_sizesrg vision_datarrs r"_get_num_multimodal_tokens-PaliGemmaProcessor._get_num_multimodal_tokens=sZ  " $ 5 56[9II !"c+&6 6    4Dm n, ,,r!)rUrHr_)NNN)NNNNr<)rrrr__doc__ attributesimage_processor_classtokenizer_classrerrrrrrrr)rrrr __classcell__)ris@r"rLrLes $[1JP>O R<(,^b fJ$fJI0$y/4HYCZZ[fJ 12 fJ fJP--r!rL)(rtypingrrnumpyrfeature_extraction_utilsr image_utilsrrprocessing_utilsr r r r r rtokenization_utils_baserrrutilsr get_loggerrrr\rangerarr$r)r'r:r?rArJrL__all__)is0r"rs#45PO   H % ).t5A$qgQ5RWX[R\8]R\Q4#waR\8]] d*d#L# /u ;4; 0BN,h-h-V Q 68]s C 8C