+hbSSKJr SSKJrJrJrJrJrJr SSK r SSK r SSK J r SSKJr SSKJrJr SSKJr SS KJrJr SS KJrJr SS KJrJrJrJr SS K J!r! S SK"J#r# SSK$J%r% \"5(a SSK&J's J(r) Sr*OSr*\RV"\,5r-Sr.\"SS\55r/"SS\#5r0g)) dataclass)AnyDictListOptionalTupleUnionN)Image)tqdm) CLIPTextModel CLIPTokenizer)PipelineImageInput) AutoencoderKLUNet2DConditionModel) DDIMScheduler LCMScheduler) BaseOutputis_torch_xla_availableloggingreplace_example_docstring) randn_tensor)DiffusionPipeline)MarigoldImageProcessorTFa Examples: ```py >>> import diffusers >>> import torch >>> pipe = diffusers.MarigoldIntrinsicsPipeline.from_pretrained( ... "prs-eth/marigold-iid-appearance-v1-1", variant="fp16", torch_dtype=torch.float16 ... ).to("cuda") >>> image = diffusers.utils.load_image("https://marigoldmonodepth.github.io/images/einstein.jpg") >>> intrinsics = pipe(image) >>> vis = pipe.image_processor.visualize_intrinsics(intrinsics.prediction, pipe.target_properties) >>> vis[0]["albedo"].save("einstein_albedo.png") >>> vis[0]["roughness"].save("einstein_roughness.png") >>> vis[0]["metallicity"].save("einstein_metallicity.png") ``` ```py >>> import diffusers >>> import torch >>> pipe = diffusers.MarigoldIntrinsicsPipeline.from_pretrained( ... "prs-eth/marigold-iid-lighting-v1-1", variant="fp16", torch_dtype=torch.float16 ... ).to("cuda") >>> image = diffusers.utils.load_image("https://marigoldmonodepth.github.io/images/einstein.jpg") >>> intrinsics = pipe(image) >>> vis = pipe.image_processor.visualize_intrinsics(intrinsics.prediction, pipe.target_properties) >>> vis[0]["albedo"].save("einstein_albedo.png") >>> vis[0]["shading"].save("einstein_shading.png") >>> vis[0]["residual"].save("einstein_residual.png") ``` c\rSrSr%Sr\\R\R4\ S'\S\R\R4\ S'\S\R4\ S'Sr g)MarigoldIntrinsicsOutput_aq Output class for Marigold Intrinsic Image Decomposition pipeline. Args: prediction (`np.ndarray`, `torch.Tensor`): Predicted image intrinsics with values in the range [0, 1]. The shape is $(numimages * numtargets) imes 3 imes height imes width$ for `torch.Tensor` or $(numimages * numtargets) imes height imes width imes 3$ for `np.ndarray`, where `numtargets` corresponds to the number of predicted target modalities of the intrinsic image decomposition. uncertainty (`None`, `np.ndarray`, `torch.Tensor`): Uncertainty maps computed from the ensemble, with values in the range [0, 1]. The shape is $(numimages * numtargets) imes 3 imes height imes width$ for `torch.Tensor` or $(numimages * numtargets) imes height imes width imes 3$ for `np.ndarray`. latent (`None`, `torch.Tensor`): Latent features corresponding to the predictions, compatible with the `latents` argument of the pipeline. The shape is $(numimages * numensemble) imes (numtargets * 4) imes latentheight imes latentwidth$. predictionN uncertaintylatent) __name__ __module__ __qualname____firstlineno____doc__r npndarraytorchTensor__annotations____static_attributes__r#s/home/james-whalen/.local/lib/python3.13/site-packages/diffusers/pipelines/marigold/pipeline_marigold_intrinsics.pyrr_sO$bjj%,,.//tRZZ566 $ $ %%r/rc!^\rSrSrSrSrSrS*S\S\S\ \ \ 4S\ S \ S \\S \\\\4S \\S \\4U4Sjjjr\S5rS\S\S\S\S\S\S\S\\\\4S\\R0S\\ \R2\\R24S\S\S\4Sjr\R:R<S+Sj5r\R@"5\!"\"5S,S\S\\S\S\\S\S\S\S\S\\\\4S\\ \R0\\R04S\\ \R2\\R24S\S\S \S!\4S"jj55r#S\R0S\\R0S\\R2S\S\S\$\R0\R044 S#jr%S$\R0S\R04S%jr&\'S-S&\R0S\S'\S\$\R0\\R044S(jj5r(S)r)U=r*$).MarigoldIntrinsicsPipelinexa= Pipeline for Intrinsic Image Decomposition (IID) using the Marigold method: https://marigoldcomputervision.github.io. This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.) Args: unet (`UNet2DConditionModel`): Conditional U-Net to denoise the targets latent, conditioned on image latent. vae (`AutoencoderKL`): Variational Auto-Encoder (VAE) Model to encode and decode images and predictions to and from latent representations. scheduler (`DDIMScheduler` or `LCMScheduler`): A scheduler to be used in combination with `unet` to denoise the encoded image latents. text_encoder (`CLIPTextModel`): Text-encoder, for empty text embedding. tokenizer (`CLIPTokenizer`): CLIP tokenizer. prediction_type (`str`, *optional*): Type of predictions made by the model. target_properties (`Dict[str, Any]`, *optional*): Properties of the predicted modalities, such as `target_names`, a `List[str]` used to define the number, order and names of the predicted modalities, and any other metadata that may be required to interpret the predictions. default_denoising_steps (`int`, *optional*): The minimum number of denoising diffusion steps that are required to produce a prediction of reasonable quality with the given model. This value must be set in the model config. When the pipeline is called without explicitly setting `num_inference_steps`, the default value is used. This is required to ensure reasonable results with various model flavors compatible with the pipeline, such as those relying on very short denoising schedules (`LCMScheduler`) and those with full diffusion schedules (`DDIMScheduler`). default_processing_resolution (`int`, *optional*): The recommended value of the `processing_resolution` parameter of the pipeline. This value must be set in the model config. When the pipeline is called without explicitly setting `processing_resolution`, the default value is used. This is required to ensure reasonable results with various model flavors trained with varying optimal processing resolution values. ztext_encoder->unet->vae) intrinsicsunetvae scheduler text_encoder tokenizerprediction_typetarget_propertiesdefault_denoising_stepsdefault_processing_resolutionc >[T U]5 X`R;a&[R SUSURS35 UR UUUUUS9 UR UUUU S9 [USS5(a/S[URRR5S- -OS Ul Xpl XlXlSUl[#URS 9Ulg) Nz*Potentially unsupported `prediction_type='z&'`; values supported by the pipeline: .)r5r6r7r8r9)r:r;r<r=r6rr)vae_scale_factor)super__init__supported_prediction_typesloggerwarningregister_modulesregister_to_configgetattrlenr6configblock_out_channelsrAr;r<r=empty_text_embeddingrimage_processor) selfr5r6r7r8r9r:r;r<r= __class__s r0rC#MarigoldIntrinsicsPipeline.__init__s  "A"A A NN<_$-J*$(!5tG\G\]r/cURRRURRR-$)N)r5rK out_channelsr6latent_channels)rOs r0 n_targets$MarigoldIntrinsicsPipeline.n_targetss+yy,,0O0OOOr/imagenum_inference_steps ensemble_sizeprocessing_resolutionresample_method_inputresample_method_output batch_sizeensembling_kwargslatents generator output_typeoutput_uncertaintyreturnc H ^ S[URRR5S- -n XR:wa[ SURSU S35eUc [ S5eUS:a [ S5eUS:a [ S5eUS:Xa[ RS 5 US:XaU (a [ S 5eUc [ S 5eUS :a [ S 5eX@R-S :wa[ SURS35eUS;a [ S5eUS;a [ S5eUS:a [ S5eU S;a [ S5eU bT b [ S5eUb:[U[5(d [ S5eSU;aUSS;a [ S5eS nSunn[U[5(dU/n[U5GHunn[U[R5(d[R"U5(a`UR S;a[ SUSUR"S35eUR"SSunnSnUR S :XaUR"S nOL[U[$R$5(aUR&unnSnO[ S!US"[)U5S35eUcUUnnO"UU4UU4:wa[ S#US$UU4S%UU435eUU- nGM U Gb%[R"U 5(d [ S&5eU R+5S :wa[ S'U R"S35eUS :a=[-UU5nUU-U-nX-U-nUS :XdUS :Xa[ S(US)US*35eUUnnXR-S- UR-nUUR-S- UR-nX-UR.RR0UU4nU R"U:wa[ S+U R"S,US35eT b[T [5(aD[T 5X-:wa [ S-5e[3U 4S.jT 55(d [ S/5eU$[T [R45(d[ S0[)T 5S35eU$)1Nrrz/`vae_scale_factor` computed at initialization (z) differs from the actual one (z).zW`num_inference_steps` is not specified and could not be resolved from the model config.z'`num_inference_steps` must be positive.z!`ensemble_size` must be positive.zk`ensemble_size` == 2 results are similar to no ensembling (1); consider increasing the value to at least 3.zpComputing uncertainty by setting `output_uncertainty=True` also requires setting `ensemble_size` greater than 1.zY`processing_resolution` is not specified and could not be resolved from the model config.rzx`processing_resolution` must be non-negative: 0 for native resolution, or any positive value for downsampled processing.z.`processing_resolution` must be a multiple of r?)nearestz nearest-exactbilinearbicubicareazy`resample_method_input` takes string values compatible with PIL library: nearest, nearest-exact, bilinear, bicubic, area.zz`resample_method_output` takes string values compatible with PIL library: nearest, nearest-exact, bilinear, bicubic, area.z`batch_size` must be positive.)ptr)z*`output_type` must be one of `pt` or `np`.z2`latents` and `generator` cannot be used together.z)`ensembling_kwargs` must be a dictionary. reductionmedianmeanzF`ensembling_kwargs['reduction']` can be either `'median'` or `'mean'`.)NN)rrz`image[z(]` has unsupported dimensions or shape: rnzUnsupported `image[z ]` type: z Input `image[z]` has incompatible dimensions z with the previous images z!`latents` must be a torch.Tensor.z/`latents` has unsupported dimensions or shape: z*Extreme aspect ratio of the input image: [z x ]z`latents` has unexpected shape=z expected=z^The number of generators must match the total number of ensemble members for all input images.c3># UH4oRRTSRR:Hv M6 g7f)rN)devicetype).0gr`s r0 :MarigoldIntrinsicsPipeline.check_inputs..Ms,Xi88==IaL,?,?,D,DDis=> > l *IJ J  9#8QR R  (/66 !LMM//4Ek4RZl4l !ijj 1%&&GE&FAs#rzz**eooc.B.B889,$wqc1YZ]ZcZcYdde%fgg99RS>S88q=))A,CC--88S #6qc49+Q!OPPyC11QC:% #A3&EsCj\Qkmnpqlrkst # J'',  ??7++ !DEE{{}! #RSZS`S`Raab!cdd$q(q!911X=1X=A:!$'QRSQTTWXYWZZ[%\]]e1***Q.43H3HHAT***Q.43H3HHA(8$)):J:J:W:WYZ\]^N}}. #B7==/Q[\j[kkl!mnn  )T**y>Z%??$xXiXXX$%bccY  5??;; #?Y?PPQ!RSSr/c[US5(d0UlOA[UR[5(d"[ S[ UR5S35e[S0URD6nUR SU5US'UR SU5US'Ub [U40UD6$Ub [SSU0UD6$[ S5e) N_progress_bar_configz=`self._progress_bar_config` should be of type `dict`, but is r?descleavetotalz/Either `total` or `iterable` has to be defined.r#)hasattrrryrzrxrsgetr )rOiterablerrrprogress_bar_configs r0 progress_bar'MarigoldIntrinsicsPipeline.progress_barTst344(*D %D55t<<OPTUYUnUnPoOppqr #?T%>%>?&9&=&=fd&KF#':'>'>w'NG$  8$78 8  ;e;':; ;NO Or/match_input_resolution output_latent return_dictc URnURnUc URnUc URnUR UUUUUUUU U U U U 5 nUR c]SnUR USUR RSSS9nURRU5nURU5SUlURRXUUU5unnnURXXU5unnAUR RUUS9RUS S 5n/nUR[!SUU-U5SS S 9HnUUUU-nUUUU-nUR"SnUSUn UR$R'UUS 9 URUR$R(S SS 9Hwn![*R,"UU/S S9n"UR/U"U!U S S9Sn#UR$R1U#U!UU S9R2n[4(dMb[6R8"5 My UR;U5 M [*R,"USS9nAAAAAA A"A#UR<"UU-UR>-UR@RBRD/UR"SSQ76n$[*R,"[!SU$R"SU5Vs/sHnURGU$UUU-5PM snSS9n%A$U(dSnURRIU%U5n%Sn&US :aU%R<"UX0R>/U%R"S SQ76n%[!U5Vs/sH#nURJ"U%UU 40U =(d 0D6PM% n%n[MU%6un%n&[*R,"U%SS9n%U (a[*R,"U&SS9n&OSn&U(aBURROU%UUS S9n%U&b#U (aURROU&UUS S9n&U S:Xa@URRQU%5n%U&b"U (aURRQU&5n&URS5 U(dU%U&U4$[UU%U&US9$s snfs snf)a Function invoked when calling the pipeline. Args: image (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `List[PIL.Image.Image]`, `List[np.ndarray]`), `List[torch.Tensor]`: An input image or images used as an input for the intrinsic decomposition task. For arrays and tensors, the expected value range is between `[0, 1]`. Passing a batch of images is possible by providing a four-dimensional array or a tensor. Additionally, a list of images of two- or three-dimensional arrays or tensors can be passed. In the latter case, all list elements must have the same width and height. num_inference_steps (`int`, *optional*, defaults to `None`): Number of denoising diffusion steps during inference. The default value `None` results in automatic selection. ensemble_size (`int`, defaults to `1`): Number of ensemble predictions. Higher values result in measurable improvements and visual degradation. processing_resolution (`int`, *optional*, defaults to `None`): Effective processing resolution. When set to `0`, matches the larger input image dimension. This produces crisper predictions, but may also lead to the overall loss of global context. The default value `None` resolves to the optimal value from the model config. match_input_resolution (`bool`, *optional*, defaults to `True`): When enabled, the output prediction is resized to match the input dimensions. When disabled, the longer side of the output will equal to `processing_resolution`. resample_method_input (`str`, *optional*, defaults to `"bilinear"`): Resampling method used to resize input images to `processing_resolution`. The accepted values are: `"nearest"`, `"nearest-exact"`, `"bilinear"`, `"bicubic"`, or `"area"`. resample_method_output (`str`, *optional*, defaults to `"bilinear"`): Resampling method used to resize output predictions to match the input resolution. The accepted values are `"nearest"`, `"nearest-exact"`, `"bilinear"`, `"bicubic"`, or `"area"`. batch_size (`int`, *optional*, defaults to `1`): Batch size; only matters when setting `ensemble_size` or passing a tensor of images. ensembling_kwargs (`dict`, *optional*, defaults to `None`) Extra dictionary with arguments for precise ensembling control. The following options are available: - reduction (`str`, *optional*, defaults to `"median"`): Defines the ensembling function applied in every pixel location, can be either `"median"` or `"mean"`. latents (`torch.Tensor`, *optional*, defaults to `None`): Latent noise tensors to replace the random initialization. These can be taken from the previous function call's output. generator (`torch.Generator`, or `List[torch.Generator]`, *optional*, defaults to `None`): Random number generator object to ensure reproducibility. output_type (`str`, *optional*, defaults to `"np"`): Preferred format of the output's `prediction` and the optional `uncertainty` fields. The accepted values are: `"np"` (numpy array) or `"pt"` (torch tensor). output_uncertainty (`bool`, *optional*, defaults to `False`): When enabled, the output's `uncertainty` field contains the predictive uncertainty map, provided that the `ensemble_size` argument is set to a value above 2. output_latent (`bool`, *optional*, defaults to `False`): When enabled, the output's `latent` field contains the latent codes corresponding to the predictions within the ensemble. These codes can be saved, modified, and used for subsequent calls with the `latents` argument. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.marigold.MarigoldIntrinsicsOutput`] instead of a plain tuple. Examples: Returns: [`~pipelines.marigold.MarigoldIntrinsicsOutput`] or `tuple`: If `return_dict` is `True`, [`~pipelines.marigold.MarigoldIntrinsicsOutput`] is returned, otherwise a `tuple` is returned where the first element is the prediction, the second element is the uncertainty (or `None`), and the third is the latent (or `None`). N do_not_padTri)padding max_length truncationreturn_tensorsr)rrdtyperzMarigold predictions...)rr)rrFzDiffusion steps...r)encoder_hidden_statesr)r`r)is_aar))r r!r")+_execution_devicerr<r=rrMr9model_max_length input_idstor8rN preprocessprepare_latentsrepeatrrangerr7 set_timesteps timestepsr+catr5step prev_sample XLA_AVAILABLExm mark_stepappendreshaperUr6rKrTdecode_prediction unpad_imageensemble_intrinsicszipresize_antialias pt_to_numpymaybe_free_model_hooksr)'rOrWrXrYrZrr[r\r]r^r_r`rarbrrrrrrprompt text_inputstext_input_idsroriginal_resolution image_latent pred_latentbatch_empty_text_embedding pred_latentsrbatch_image_latentbatch_pred_latenteffective_batch_sizetextt batch_latentnoisepred_latent_for_decodingr r!s' r0__call__#MarigoldIntrinsicsPipeline.__call__gs.d''   &"&">">  ($($F$F !&&    ! ! "        "  $ $ ,F..$>>::# )K)2255f=N(,(9(9.(I!(LD %/3.B.B.M.M *?/ +w+%)$8$8 Ij% ! k %)%>%>%A%AW\%A%]%d%d 1& " "" !Z-/ -.C/CDD NN ( ()O)PVWX  ,[` abcd$(NN$7$71/9%8%+"!=LLNi    1 2' *ii !4    &      $/#6#6  & 79X9X$ [f[l[lmnmo[p$  YYq":"@"@"CZP PA&&'?A N'STP    %K))55j'J  1 #++M>>>/1Gu?J&+="22CC!46LTYD $ --99*EJ&+="22>>{K  ##% [9 9'!#  y 0s !Q>*Rc Sn[R"[SURSU5Vs/sH(nU"URR XXu-55PM* snSS9nXRR R-nURUSS9nURuppUn U c/[XRU -X4UURURS9n X4$s snf)Nc[US5(aURR5$[US5(a UR$[ S5e)N latent_distr_z3Could not access latents of provided encoder_output)rrmoder_AttributeError)encoder_outputs r0retrieve_latentsDMarigoldIntrinsicsPipeline.prepare_latents..retrieve_latents{sI~}55%11668833%---$%Z[[r/rr)r`rrr) r+rrrr6encoderKscaling_factorrepeat_interleaverrUrrr)rOrWr_r`rYr]rrrN_ECrrrs r0r*MarigoldIntrinsicsPipeline.prepare_latentsss \yyq%++a.*= =A!1>1J!KL=   $hhoo&D&DD #55m5K #))   &nnq(!/##**"(( K((' s/CrcUR5S:wd1URSURRR:wa:[ SURRRSURS35eURR XRRR- SS9Sn[R"US S 5nUS -S - nU$) Nrnrz Expecting 4D tensor of shape [B,z ,H,W]; got r?F)rrgg?g@) rrr6rKrTrxdecoderr+clip)rOrr s r0r,MarigoldIntrinsicsPipeline.decode_predictions ??  ![%6%6q%9TXX__=\=\%\2488??3R3R2SS^_j_p_p^qqrs XX__[88??3Q3Q%Q_d_efgh ZZ D#6  3&#- r/targetsrjcRUR5S:wdURSS:wa[SURS35eUS;a[SUS35eURup4pVnSnUS :Xa4[R"US S 9n U(a[R "US S 9nX4$US :Xar[R "US S S9Rn U(a7[R"X - 5n[R "US S 9RnU RS 5n X4$[SUS35e)a Ensembles the intrinsic decomposition represented by the `targets` tensor with expected shape `(B, T, 3, H, W)`, where B is the number of ensemble members for a given prediction of size `(H x W)`, and T is the number of predicted targets. Args: targets (`torch.Tensor`): Input ensemble of intrinsic image decomposition maps. output_uncertainty (`bool`, *optional*, defaults to `False`): Whether to output uncertainty map. reduction (`str`, *optional*, defaults to `"mean"`): Reduction method used to ensemble aligned predictions. The accepted values are: `"median"` and `"mean"`. Returns: A tensor of aligned and ensembled intrinsic decomposition maps with shape `(T, 3, H, W)` and optionally a tensor of uncertainties of shape `(T, 3, H, W)`. rrz.Expecting 4D tensor of shape [B,T,3,H,W]; got r?rkzUnrecognized reduction method: NrmrrrlT)rkeepdim) rrrxr+rmstdrlvaluesabssqueeze) rrbrjBT_rrr!r s r0r.MarigoldIntrinsicsPipeline.ensemble_intrinsicss"0 ;;=A q!1Q!6Mgmm_\]^_ _ . .>ykKL L aA  G3J!#iiQ7 &&( "g1dCJJJ!#ii(<= #ll;A>EE #++A.J&&>ykKL Lr/)rr<r=rMrNr;rA)NNNN)NNNT)NrNTrfrfrNNNr)FFT)Frl)+r$r%r&r'r(model_cpu_offload_seqrDrrr rrr r rstrrrintrCpropertyrUrr+r,rrboolrcompilerdisablerno_gradrEXAMPLE_DOC_STRINGrrrr staticmethodrr. __classcell__)rPs@r0r2r2xs$L6!0*.6:157;*^"*^*^ 45 *^ $ *^ ! *^"#*^$DcN3*^"*#*^(0}*^*^XPP@!@!@ @ # @ # @!$@@$DcN3@%,,'@E%//43H"HIJ@@!@ @D ^^PP$ ]]_12.2/3'+%/&06:EIMQ#(# !H !H &c]H  H (} H !% H  #H !$H H $DcN3H % d5<<.@ @ABH E%//43H"HIJH H !H H !H 3H T$)||$)%,,'$)EOO, $)  $)  $) u||U\\) *$)L U\\ ell $)!*'*' *'*' u||Xell33 4 *'*'r/r2)1 dataclassesrtypingrrrrrr numpyr)r+PILr tqdm.autor transformersr r rNrmodelsrr schedulersrrutilsrrrrutils.torch_utilsrpipeline_utilsrmarigold_image_processingrtorch_xla.core.xla_modelcore xla_modelrr get_loggerr$rErrr2r#r/r0rs&":: 51 ..=))MM   H %"J &z& &0Y '!2Y 'r/