+h: SSKrSSKJrJrJrJrJrJrJr SSK r SSK r SSK r SSK Js Jr SSKJrJrJrJr SSKJrJr SSKJrJrJrJr SSKJrJ r J!r!J"r" SSK#J$r$ SS K%J&r& SS K'J(r( SS K)J*r* SS K+J,r,J-r-J.r.J/r/J0r0J1r1 SS K2J3r3J4r4 SSK5J6r6 SSK7J8r8 SSK9J:r:J;r; SSKJ?s J@rA SrBOSrB\.R"\D5rESrFSS\ RS\\ RS\I4SjjrJ"SS\:\;\\\\8\5 rKg)N)AnyCallableDictListOptionalTupleUnion)CLIPImageProcessor CLIPTextModel CLIPTokenizerCLIPVisionModelWithProjection)PipelineImageInputVaeImageProcessor)FromSingleFileMixinIPAdapterMixinStableDiffusionLoraLoaderMixinTextualInversionLoaderMixin) AutoencoderKLImageProjectionUNet2DConditionModelUNetMotionModel)SparseControlNetModel)adjust_lora_scale_text_encoder) MotionAdapter)KarrasDiffusionSchedulers)USE_PEFT_BACKENDis_torch_xla_availableloggingreplace_example_docstringscale_lora_layersunscale_lora_layers)is_compiled_module randn_tensor)VideoProcessor) FreeInitMixin)DiffusionPipelineStableDiffusionMixin)AnimateDiffPipelineOutputTFa Examples: ```python >>> import torch >>> from diffusers import AnimateDiffSparseControlNetPipeline >>> from diffusers.models import AutoencoderKL, MotionAdapter, SparseControlNetModel >>> from diffusers.schedulers import DPMSolverMultistepScheduler >>> from diffusers.utils import export_to_gif, load_image >>> model_id = "SG161222/Realistic_Vision_V5.1_noVAE" >>> motion_adapter_id = "guoyww/animatediff-motion-adapter-v1-5-3" >>> controlnet_id = "guoyww/animatediff-sparsectrl-scribble" >>> lora_adapter_id = "guoyww/animatediff-motion-lora-v1-5-3" >>> vae_id = "stabilityai/sd-vae-ft-mse" >>> device = "cuda" >>> motion_adapter = MotionAdapter.from_pretrained(motion_adapter_id, torch_dtype=torch.float16).to(device) >>> controlnet = SparseControlNetModel.from_pretrained(controlnet_id, torch_dtype=torch.float16).to(device) >>> vae = AutoencoderKL.from_pretrained(vae_id, torch_dtype=torch.float16).to(device) >>> scheduler = DPMSolverMultistepScheduler.from_pretrained( ... model_id, ... subfolder="scheduler", ... beta_schedule="linear", ... algorithm_type="dpmsolver++", ... use_karras_sigmas=True, ... ) >>> pipe = AnimateDiffSparseControlNetPipeline.from_pretrained( ... model_id, ... motion_adapter=motion_adapter, ... controlnet=controlnet, ... vae=vae, ... scheduler=scheduler, ... torch_dtype=torch.float16, ... ).to(device) >>> pipe.load_lora_weights(lora_adapter_id, adapter_name="motion_lora") >>> pipe.fuse_lora(lora_scale=1.0) >>> prompt = "an aerial view of a cyberpunk city, night time, neon lights, masterpiece, high quality" >>> negative_prompt = "low quality, worst quality, letterboxed" >>> image_files = [ ... "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/animatediff-scribble-1.png", ... "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/animatediff-scribble-2.png", ... "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/animatediff-scribble-3.png", ... ] >>> condition_frame_indices = [0, 8, 15] >>> conditioning_frames = [load_image(img_file) for img_file in image_files] >>> video = pipe( ... prompt=prompt, ... negative_prompt=negative_prompt, ... num_inference_steps=25, ... conditioning_frames=conditioning_frames, ... controlnet_conditioning_scale=1.0, ... controlnet_frame_indices=condition_frame_indices, ... generator=torch.Generator().manual_seed(1337), ... ).frames[0] >>> export_to_gif(video, "output.gif") ``` encoder_output generator sample_modec[US5(a!US:XaURRU5$[US5(a US:XaURR5$[US5(a UR$[ S5e)N latent_distsampleargmaxlatentsz3Could not access latents of provided encoder_output)hasattrr0r1moder3AttributeError)r,r-r.s y/home/james-whalen/.local/lib/python3.13/site-packages/diffusers/pipelines/animatediff/pipeline_animatediff_sparsectrl.pyretrieve_latentsr8ws}~}--+2I))00;;  / /K84K))..00  + +%%%RSSc7<^\rSrSrSrSr/SQr/SQrSIS\S\ S \ S \ \ \ 4S \S \S \S\S\4U4SjjjrSJS\\R,S\\R,S\\S\\4SjjrSKSjrSrSrSrSLS\4SjjrSrSKSjr Sr!S \R,S!\S"\S#\RDS$\RFS%\$\R,\R,44 S&jr%\&S'5r'\&S(5r(\&S)5r)\&S*5r*\&S+5r+\RX"5\-"\.5SSSS,S-S.SS/S0SSSSSSSS1S2SSS3/S4SSS5/4S6\\ \/\0\/4S7\\S8\\S!\S9\S:\S;\\ \/\0\/4S<\S=\S>\\ \Rb\0\Rb4S5\\R,S\\R,S\\R,S?\\2S@\\0\R,S \\0\2SA\/SB\3SC\\4\/\54S\ \\0\4S"\0\SD\3S\\SE\\6\\\4/S4SF\0\/42SGjj55r7SHr8U=r9$)M#AnimateDiffSparseControlNetPipelinea Pipeline for controlled text-to-video generation using the method described in [SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models](https://huggingface.co/papers/2311.16933). This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods implemented for all pipelines (downloading, saving, running on a particular device, etc.). The pipeline also inherits the following loading methods: - [`~loaders.TextualInversionLoaderMixin.load_textual_inversion`] for loading textual inversion embeddings - [`~loaders.StableDiffusionLoraLoaderMixin.load_lora_weights`] for loading LoRA weights - [`~loaders.StableDiffusionLoraLoaderMixin.save_lora_weights`] for saving LoRA weights - [`~loaders.IPAdapterMixin.load_ip_adapter`] for loading IP Adapters Args: vae ([`AutoencoderKL`]): Variational Auto-Encoder (VAE) Model to encode and decode images to and from latent representations. text_encoder ([`CLIPTextModel`]): Frozen text-encoder ([clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14)). tokenizer (`CLIPTokenizer`): A [`~transformers.CLIPTokenizer`] to tokenize text. unet ([`UNet2DConditionModel`]): A [`UNet2DConditionModel`] used to create a UNetMotionModel to denoise the encoded video latents. motion_adapter ([`MotionAdapter`]): A [`MotionAdapter`] to be used in combination with `unet` to denoise the encoded video latents. scheduler ([`SchedulerMixin`]): A scheduler to be used in combination with `unet` to denoise the encoded image latents. Can be one of [`DDIMScheduler`], [`LMSDiscreteScheduler`], or [`PNDMScheduler`]. z&text_encoder->image_encoder->unet->vae)feature_extractor image_encodermotion_adapter)r3 prompt_embedsnegative_prompt_embedsNvae text_encoder tokenizerunetr? controlnet schedulerr=r>c >[T U]5 [U[5(a[R "XE5nUR UUUUUUUUU S9 [USS5(a/S[URRR5S- -OSUl [SURS9Ul[URSSS 9Ulg) N) rBrCrDrEr?rFrGr=r>rBr&r*F) do_resizevae_scale_factorT)rKdo_convert_rgb do_normalize)super__init__ isinstancerr from_unet2dregister_modulesgetattrlenrBconfigblock_out_channelsrKr%video_processorrcontrol_image_processor) selfrBrCrDrEr?rFrGr=r> __class__s r7rO,AnimateDiffSparseControlNetPipeline.__init__s  d0 1 1"..tDD %)!/'  W^^bdikoVpVpc$((//*L*L&MPQ&Q Rvw-PTPePef'8!224V[( $r9r@rA lora_scale clip_skipc  UbS[U[5(a>Xl[(d[ UR U5 O[ UR U5 Ub[U[5(aSn O3Ub![U[5(a [U5n OURSn UGc[U[5(aURXR5nURUSURRSSS9n U Rn URUSSS 9Rn U RS U RS :a[ R""X5(djURR%U SS2URRS- S 245n[&R)S URRS U35 [+UR R,S 5(aAUR R,R.(aU R0R3U5nOSnU c%UR U R3U5US9nUSnOQUR U R3U5USS9nUS U S-*nUR R4R7U5nUR bUR R8nO0UR:bUR:R8nO UR8nUR3UUS9nURunnnUR=SUS5nUR?UU-US 5nU(GaUGc|UcS/U -nOUb;[AU5[AU5La$[CS[AU5S[AU5S35e[U[5(aU/nO2U [U5:wa![ESUS[U5SUSU S3 5eUn[U[5(aURUUR5nURSnURUSUSSS9n[+UR R,S 5(aAUR R,R.(aUR0R3U5nOSnUR URR3U5US9nUSnU(aGURSnUR3UUS9nUR=SUS5nUR?X-US 5nUR b6[U[5(a![(a[GUR U5 Xg4$)a, Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `List[str]`, *optional*): prompt to be encoded device: (`torch.device`): torch device num_images_per_prompt (`int`): number of images that should be generated per prompt do_classifier_free_guidance (`bool`): whether to use classifier free guidance or not negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. lora_scale (`float`, *optional*): A LoRA scale that will be applied to all LoRA layers of the text encoder if LoRA layers are loaded. clip_skip (`int`, *optional*): Number of layers to be skipped from CLIP while computing the prompt embeddings. A value of 1 means that the output of the pre-final layer will be used for computing the prompt embeddings. Nr*r max_lengthTpt)paddingr_ truncationreturn_tensorslongest)rarcz\The following part of your input was truncated because CLIP can only handle sequences up to z tokens: use_attention_mask)attention_mask)rgoutput_hidden_statesdtypedevicez?`negative_prompt` should be the same type to `prompt`, but got z != .z`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.)$rPr _lora_scalerrrCr!strlistrTshapermaybe_convert_promptrDmodel_max_length input_idstorchequal batch_decodeloggerwarningr4rUrfrgto text_modelfinal_layer_normrjrErepeatviewtype TypeError ValueErrorr")rYpromptrknum_images_per_promptdo_classifier_free_guidancenegative_promptr@rAr\r] batch_size text_inputstext_input_idsuntruncated_ids removed_textrgprompt_embeds_dtypebs_embedseq_len_ uncond_tokensr_ uncond_inputs r7 encode_prompt1AnimateDiffSparseControlNetPipeline.encode_promptsQV  !j7U&V&V) $#.t/@/@*M!$"3"3Z@  *VS"9"9J  Jvt$<$<VJ&,,Q/J  $ ;<<226>>J..$>>::# )K)22N"nnVYW[n\ffO$$R(N,@,@,DDU[[NN $~~::#At~~'F'F'JR'O$OP  778 ,Q t((//1EFF4K\K\KcKcKvKv!,!;!;!>!>v!F!% $ 1 1.2C2CF2K\j 1 k -a 0 $ 1 1"%%f-ncg!2! !.b 1IM2B C !% 1 1 < < M Mm \    ("&"3"3"9"9  YY ""&))// "/"5"5 %((/B6(R ,22'1%,,Q0EqI %**86K+KWVXY  '+A+I&!#z 1 #V D>$%# *Lt((//1EFF4K\K\KcKcKvKv!-! " &,2215G%;%>%>EXag%>%h "%;%B%B1F[]^%_ "%;%@%@Acelnp%q "    ($ >??DTDT#D$5$5zB44r9cd[URR55Rn[ U[ R 5(dURUSS9RnURX%S9nU(aqURUSS9RSnURUSS9nUR[ R"U5SS9RSnURUSS9nXg4$URU5RnURUSS9n[ R"U5n X4$) Nr`)rc)rkrjT)rhrdim)nextr> parametersrjrPruTensorr= pixel_valuesrz hidden_statesrepeat_interleave zeros_like image_embeds) rYimagerkrrhrjimage_enc_hidden_statesuncond_image_enc_hidden_statesruncond_image_embedss r7 encode_image0AnimateDiffSparseControlNetPipeline.encode_images?T''2245;;%..**5*FSSE4 &*&8&8UY&8&Z&h&hik&l #&=&O&OPekl&O&m #-1-?-?  'd.@.mB. *.L-]-]%1.^. *+J J--e4AAL'99:OUV9WL"'"2"2<"@ 4 4r9c /nU(a/nUGc&[U[5(dU/n[U5[URRR 5:waB[ S[U5S[URRR 5S35e[XRRR 5Hhup[U [5(+n URXSU 5upURU SSS245 U(dMPWRU SSS245 Mj OEUH?n U(a$U RS5upWRU 5 URU 5 MA /n[U5Hwup[R"U /U-SS9n U(a2[R"WU /U-SS9n [R"X/SS9n U RUS9n URU 5 My U$) NzK`ip_adapter_image` must have same length as the number of IP Adapters. Got z images and z IP Adapters.r*r&rrrk)rPrprTrEencoder_hid_projimage_projection_layersrziprrappendchunk enumeraterucatrz)rYip_adapter_imageip_adapter_image_embedsrkrrrnegative_image_embedssingle_ip_adapter_imageimage_proj_layeroutput_hidden_statesingle_image_embedssingle_negative_image_embedsis r7prepare_ip_adapter_image_embedsCAnimateDiffSparseControlNetPipeline.prepare_ip_adapter_image_embedss9 &$& ! " *.55$4#5 #$DII,F,F,^,^(__ abefvbwaxyEFIJNJSJSJdJdJ|J|F}E~~KL>A ))"<"<"T"T>9'+55E*W&W#DHDUDU+Q8KEA###$7a$@A..)001MdTUg1VW>(?#.H[HaHabcHdE0)001MN##$78 (? #%&/ &= "A"'))-@,ADY,Y_`"a */4yy:OPQ:R9SVk9kqr/s,&+ii1M0cij&k#"5"8"8"8"G  # * *+> ?'>'&r9cSURRR- U-nURup#pEnUR SSSSS5R X$-X5U5nURR U5RnUSSS24R X$S4URSS-5R SSSSS5nUR5nU$)Nr*rr&rre) rBrUscaling_factorrqpermutereshapedecoder1float) rYr3rchannels num_framesheightwidthrvideos r7decode_latents2AnimateDiffSparseControlNetPipeline.decode_latentssdhhoo444w>:A--7 j%//!Q1a0889PRZdij(//dAg&& 'CekkRSRTo'UV^^_`bcefhiklm  r9cnS[[R"URR5R R 55;n0nU(aX$S'S[[R"URR5R R 55;nU(aXS'U$)Netar-)setinspect signaturerGsteprkeys)rYr-r accepts_etaextra_step_kwargsaccepts_generators r7prepare_extra_step_kwargs=AnimateDiffSparseControlNetPipeline.prepare_extra_step_kwargss s7#4#4T^^5H5H#I#T#T#Y#Y#[\\  '*e $(3w/@/@ATAT/U/`/`/e/e/g+hh -6k *  r9?controlnet_conditioning_scalec *^US-S:wd US-S:wa[SUSUS35eU bW[U4SjU 55(d=[STRSU V s/sHoTR;dMU PM sn 35eUbUb[S US US 35eUcUc [S 5eUbA[U[5(d,[U[ 5(d[S [ U535eUbUb[SUSUS 35eUbCUb@URUR:wa&[SURSURS35eUbUb [S5eUb[[U[ 5(d[S[ U535eUSRS;a[SUSRS35e[[S5=(a8 [TR[RRR5n [TR[ 5(d0U (am[TRR"[ 5(aD[U [ 5(aU HnTR%XU5 M OTR%XU5 Oe[TR[ 5(d0U (aJ[TRR"[ 5(a![U [&5(d [)S5eges sn f)NrIrz7`height` and `width` have to be divisible by 8 but are z and rmc3@># UHoTR;v M g7fN)_callback_tensor_inputs).0krYs r7 CAnimateDiffSparseControlNetPipeline.check_inputs..sF 7Y!-- -7Ysz2`callback_on_step_end_tensor_inputs` has to be in z , but found zCannot forward both `prompt`: z and `prompt_embeds`: z2. Please make sure to only forward one of the two.zeProvide either `prompt` or `prompt_embeds`. Cannot leave both `prompt` and `prompt_embeds` undefined.z2`prompt` has to be of type `str` or `list` but is z'Cannot forward both `negative_prompt`: z and `negative_prompt_embeds`: zu`prompt_embeds` and `negative_prompt_embeds` must have the same shape when passed directly, but got: `prompt_embeds` z != `negative_prompt_embeds` zProvide either `ip_adapter_image` or `ip_adapter_image_embeds`. Cannot leave both `ip_adapter_image` and `ip_adapter_image_embeds` defined.z:`ip_adapter_image_embeds` has to be of type `list` but is )rrzF`ip_adapter_image_embeds` has to be a list of 3D or 4D tensors but is Dscaled_dot_product_attentionzLFor single controlnet: `controlnet_conditioning_scale` must be type `float`.)rallrrProrprrqndimr4FrFru_dynamo eval_frameOptimizedModuler _orig_mod check_imagerr)rYrrrrr@rArr"callback_on_step_end_tensor_inputsrrr is_compiledimage_s` r7 check_inputs0AnimateDiffSparseControlNetPipeline.check_inputss A:?eai1nVW]V^^cdicjjklm m - 9#F 7YF C C DTEaEaDbbn|^pH|^vwkokGkGbGpq|^pHoIJ   -";08N}o^00 ^ 5w  FC)@)@TZ\`IaIaQRVW]R^Q_`a a  &+A+M9/9JK*++]_   $)?)K""&<&B&BB --:-@-@,AB.445Q8  ',C,O^  # .5t<< PQUVmQnPop)+00> \]tuv]w]|]|\}}~a!?@ Z OOU]]55EEF t(= > >4??446KLL%&&#F$$V]C$   > 5 t(= > >4??446KLL;UCC nooD 5OpHs L1Lc[U[RR5n[U[R5n[U[ R 5n[U[5=(a' [US[RR5n[U[5=(a [US[R5n[U[5=(a [US[ R 5n U(d:U(d3U(d,U(d%U(dU (d[S[U535eU(aSn O [U5n Ub[U[5(aSn O6Ub![U[5(a [U5n OUbURSn U S:waU W :wa[SU SU 35egg)Nrzimage must be passed and be one of PIL image, numpy array, torch tensor, list of PIL images, list of numpy arrays or list of torch tensors, but is r*zdIf image batch size is not 1, image batch size must be same as prompt batch size. image batch size: z, prompt batch size: )rPPILImagerurnpndarrayrprrrTrorqr) rYrrr@ image_is_pilimage_is_tensor image_is_npimage_is_pil_listimage_is_tensor_listimage_is_np_listimage_batch_sizeprompt_batch_sizes r7r/AnimateDiffSparseControlNetPipeline.check_imageKs!%9 $UELL9  3 &ud3] 58SYY__8])%6]:eAhPUP\P\;]%eT2Wz%(BJJ7W#%($fgklqgrfst   "5z   *VS"9"9 !   Jvt$<$< #F   & - 3 3A 6  q %59J%JvxHwII^_p^qr &K r9c 0UUUX@R-XPR-4n [U[5(a*[U5U:wa[ S[U5SUS35eU c [ XXvS9n OU R U5n XRR-n U $)Nz/You have passed a list of generators of length z+, but requested an effective batch size of z@. Make sure the batch size matches the length of the generators.)r-rkrj) rKrPrprTrr$rzrGinit_noise_sigma) rYrnum_channels_latentsrrrrjrkr-r3rqs r7prepare_latents3AnimateDiffSparseControlNetPipeline.prepare_latentsqs   ++ + ** *   i & &3y>Z+GA#i.AQR&<'gi  ?"5fZGjj(GNN;;;r9cpURRXUS9nURS5RXE5nURupxpnUR 5S:aUR 5S::deURR(aURXx-XU5nSU-S- n[URRU55URRR-n U RXxSX0R-X R-5n OUn U R!SSSSS5n U $)N)rrrr*r&rr)rX preprocess unsqueezerzrqminmaxrF"use_simplified_condition_embeddingrr8rBencoderUrrKr) rYrrrrkrjcontrolnet_imagesrrrconditioning_framess r7 prepare_image1AnimateDiffSparseControlNetPipeline.prepare_images0,,77TY7Z!OOA.11&@:K:Q:Q7 %!$$&!+0A0E0E0G10LLL ?? = = 1 9 9*:QS[ej k  !$5 5 9 "2488??CT3U"VY]YaYaYhYhYwYw"w "5"="=65J5J+JEUjUjLj# #4 199!Q1aH""r9rrcontrolnet_frame_indicesrkrjreturnc URS[U5:deURupgpn [R"XgX)U 4XTS9n [R"USX)U 4XTS9n USS2SS2S[U524U SS2SS2U4'SU SS2SS2U4'X4$)Nr&rir*)rqrTruzeros) rYrrr rkrjrrrrrcontrolnet_condcontrolnet_cond_masks r7#prepare_sparse_control_conditioningGAnimateDiffSparseControlNetPipeline.prepare_sparse_control_conditionings#((+s3K/LLLL1D1J1J. a++zZQV&W_dt${{J:u+U]br:MaQRTsVYZrVsTsNs:t1667?@Q#;;<44r9cUR$r_guidance_scalerYs r7guidance_scale2AnimateDiffSparseControlNetPipeline.guidance_scales###r9cUR$r) _clip_skiprs r7r]-AnimateDiffSparseControlNetPipeline.clip_skips r9c URS:$)Nr*rrs r7r?AnimateDiffSparseControlNetPipeline.do_classifier_free_guidances##a''r9cUR$r)_cross_attention_kwargsrs r7cross_attention_kwargs:AnimateDiffSparseControlNetPipeline.cross_attention_kwargss+++r9cUR$r)_num_timestepsrs r7 num_timesteps1AnimateDiffSparseControlNetPipeline.num_timestepss"""r92g@r*gpilTrFr3rrrnum_inference_stepsrrnum_videos_per_promptrr-rr output_type return_dictr guess_modecallback_on_step_endrc [UR5(aURRO URnU=(d- URRR UR -nU=(d- URRR UR -nSnURUUUUU U UUUUUS9 X`lUUl UUl Ub[U[5(aSnO3Ub![U[5(a [U5nOU RSnUR n[U["5(aURR$O"UR&SRR$nU=(d UnUR(bUR(R+SS5OSnUR-UUUUR.UU U UUR0S9 upUR.(a[2R4"X/5n U R7USS9n UcUb"UR9UUUUU-UR.5nUR;UX2UUR<5nUR?UUUUUR<5un n!UR@RCUUS9 UR@RDn"URRRFn#URIUU-U#UUUU R<UU U 5 n URKX5n$UcUbS W0OSn%URL(a URNOSn&[QU&5GHn'URL(a#URSU U'UUU R<U 5un n"[U"5Ul*[U"5XPR@RV-- n(URYURTS 9n)[[U"5GHun*n+UR.(a[2R4"U /S -5OU n,UR@R]U,U+5n,U(aDUR.(a3U n-UR@R]U-U+5n-U R_S 5Sn.OU,n-U n.URU-U+U.U U!UUS S 9un/n0URU,U+U UU%U/U0S9R`n1UR.(aU1R_S 5un2n3U2UU3U2- --n1UR@Rb"U1U+U 40U$D6Rdn Ub\0n4UHn5[g5U5U4U5'M U"UU*U+U45n6U6RiSU 5n U6RiSU 5n U6RiSU 5n U*[U"5S- :Xd)U*S-U(:a0U*S-UR@RV-S:XaU)Rk5 [l(dGM[nRp"5 GM SSS5 GM US:XaU n7O+URsU 5n8URtRwU8US9n7URy5 U(dU74$[{U7S9$!,(df  GM=f)u The call function to the pipeline for generation. Args: prompt (`str` or `List[str]`, *optional*): The prompt or prompts to guide image generation. If not defined, you need to pass `prompt_embeds`. height (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`): The height in pixels of the generated video. width (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`): The width in pixels of the generated video. num_frames (`int`, *optional*, defaults to 16): The number of video frames that are generated. Defaults to 16 frames which at 8 frames per seconds amounts to 2 seconds of video. num_inference_steps (`int`, *optional*, defaults to 50): The number of denoising steps. More denoising steps usually lead to a higher quality videos at the expense of slower inference. guidance_scale (`float`, *optional*, defaults to 7.5): A higher guidance scale value encourages the model to generate images closely linked to the text `prompt` at the expense of lower image quality. Guidance scale is enabled when `guidance_scale > 1`. negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts to guide what to not include in image generation. If not defined, you need to pass `negative_prompt_embeds` instead. Ignored when not using guidance (`guidance_scale < 1`). eta (`float`, *optional*, defaults to 0.0): Corresponds to parameter eta (η) from the [DDIM](https://huggingface.co/papers/2010.02502) paper. Only applies to the [`~schedulers.DDIMScheduler`], and is ignored in other schedulers. generator (`torch.Generator` or `List[torch.Generator]`, *optional*): A [`torch.Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. latents (`torch.Tensor`, *optional*): Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for video generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor is generated by sampling using the supplied random `generator`. Latents should be of shape `(batch_size, num_channel, num_frames, height, width)`. prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs (prompt weighting). If not provided, text embeddings are generated from the `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs (prompt weighting). If not provided, `negative_prompt_embeds` are generated from the `negative_prompt` input argument. ip_adapter_image: (`PipelineImageInput`, *optional*): Optional image input to work with IP Adapters. ip_adapter_image_embeds (`List[torch.Tensor]`, *optional*): Pre-generated image embeddings for IP-Adapter. It should be a list of length same as number of IP-adapters. Each element should be a tensor of shape `(batch_size, num_images, emb_dim)`. It should contain the negative image embedding if `do_classifier_free_guidance` is set to `True`. If not provided, embeddings are computed from the `ip_adapter_image` input argument. conditioning_frames (`List[PipelineImageInput]`, *optional*): The SparseControlNet input to provide guidance to the `unet` for generation. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generated video. Choose between `torch.Tensor`, `PIL.Image` or `np.array`. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.text_to_video_synthesis.TextToVideoSDPipelineOutput`] instead of a plain tuple. cross_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the [`AttentionProcessor`] as defined in [`self.processor`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). controlnet_conditioning_scale (`float` or `List[float]`, *optional*, defaults to 1.0): The outputs of the ControlNet are multiplied by `controlnet_conditioning_scale` before they are added to the residual in the original `unet`. If multiple ControlNets are specified in `init`, you can set the corresponding scale as a list. controlnet_frame_indices (`List[int]`): The indices where the conditioning frames must be applied for generation. Multiple frames can be provided to guide the model to generate similar structure outputs, where the `unet` can "fill-in-the-gaps" for interpolation videos, or a single frame could be provided for general expected structure. Must have the same length as `conditioning_frames`. clip_skip (`int`, *optional*): Number of layers to be skipped from CLIP while computing the prompt embeddings. A value of 1 means that the output of the pre-final layer will be used for computing the prompt embeddings. callback_on_step_end (`Callable`, *optional*): A function that calls at the end of each denoising steps during the inference. The function is called with the following arguments: `callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict)`. `callback_kwargs` will include a list of all tensors as specified by `callback_on_step_end_tensor_inputs`. callback_on_step_end_tensor_inputs (`List`, *optional*): The list of tensor inputs for the `callback_on_step_end` function. The tensors specified in the list will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the `._callback_tensor_inputs` attribute of your pipeline class. Examples: Returns: [`~pipelines.animatediff.pipeline_output.AnimateDiffPipelineOutput`] or `tuple`: If `return_dict` is `True`, [`~pipelines.animatediff.pipeline_output.AnimateDiffPipelineOutput`] is returned, otherwise a `tuple` is returned where the first element is a list with the generated frames. r*) rrrrr@rArrrrrNrscale)r@rAr\r])repeatsrrr)totalr&F)encoder_hidden_statesrconditioning_maskconditioning_scaler,r+)r2radded_cond_kwargsdown_block_additional_residualsmid_block_additional_residualr3r@rAlatent)rr*)frames)>r#rFrrErU sample_sizerKrrrrrProrprTrq_execution_devicerglobal_pool_conditionsnetsrgetrrr]rurrrrrjrrG set_timesteps timesteps in_channelsrrfree_init_enabled_free_init_num_itersrange_apply_free_initr"order progress_barrscale_model_inputrr1r prev_samplelocalspopupdate XLA_AVAILABLExm mark_steprrWpostprocess_videomaybe_free_model_hooksr+)9rYrrrrr(rrr)rr-r3r@rArrrr*r+rrr r,r]r-rrFrrkr<text_encoder_lora_scalerrrr@rrr5num_free_init_itersfree_init_iternum_warmup_stepsrGrtlatent_model_inputcontrol_model_inputcontrolnet_prompt_embedsdown_block_res_samplesmid_block_res_sample noise_prednoise_pred_uncondnoise_pred_textcallback_kwargsrcallback_outputsr video_tensors9 r7__call__,AnimateDiffSparseControlNetPipeline.__call__slf3ET__2U2UT__..[_[j[j O499++77$:O:OOM))558M8MM ! +'#9-$;/Q%*G   .#'=$  *VS"9"9J  Jvt$<$<VJ&,,Q/J''*&;<<    4 4#**AA   9#9 ?C>Y>Y>eD ' ' + +GT :ko 150B0B   !  , , '#9.nn1C 1 -   + +!II'=&MNM%77 PQ7R   '+B+N?? '2200 L#001DeU[]g]m]mn040X0X -EvzO_O_1 -- $$%8$HNN,,  $yy//;;&& . .           !::9J  +/F/R\ *  <@;Q;Qd77WX#$78N%%%)%:%:^-@&'--Yb&"#&i.D "9~0CnnFZFZ0ZZ "")<)<"=%i0DAqEIEeEeG9q=)Akr&)-)I)IJ\^_)`&!d&F&F.5+.2nn.N.NObde.f+3@3F3Fq3I!3L0.@+3@0CG??+.F(7*>+H#-$)DS D@*,@"&*.;/E*;8N6J"+"f77=G=M=Ma=P:)?%6?]nKn9o%o #nn11*a^L]^jjG+7*,!CA17!OA."D+?aO+\("2"6"6y'"J(8(<(<_m(\ 1A1E1EF^`v1w.C NQ..AE=M3MSTWXSX\`\j\j\p\pRptuRu$++-$} w1>=9R ( "E..w7L((::[f:gE ##%8O(66W>=sG/X79X77 Y )rrrrnr"rXrKrW)NN)NNNNNr)NNNNNNNr):__name__ __module__ __qualname____firstlineno____doc__model_cpu_offload_seq_optional_componentsrrr r r rrrrrr r rOrrurrintrrrrrrrrrrkrjrrpropertyrr]rrr#no_gradr EXAMPLE_DOC_STRINGror Generatorrboolrrrrb__static_attributes__ __classcell__)rZs@r7r;r;sU:EST157;  $ !  (/9:  &  * - . 5  P049=&*#'t5  -t5!) 6t5UOt5C=t5n52+'\ !,# $+//2\(-\~#Nnr2#*5"\\55#& 5  5 {{ 5 u||U\\) *5$$$ ((,,## ]]_1237 $##% #;?%&MQ*.049=9=@DBF  ;?CF/0c #'KO9B 5u7sDI~./u7 u7} u7  u7 ! u7u7"%T#Y"78u7 #u7u7E%//43H"HIJu7%,,'u7  -u7!) 6u7##56u7 "*$u||*?#u7$%u7&'u7(!)c3h 8)u7*(-UDK-?'@+u7,#'s)-u7./u70C=1u72'xc40@$0F'GH3u74-1I5u73u7r9r;)Nr1)Lrtypingrrrrrrr numpyrrrutorch.nn.functionalnn functionalr transformersr r r r image_processorrrloadersrrrrmodelsrrrr(models.controlnets.controlnet_sparsectrlr models.lorarmodels.unets.unet_motion_modelr schedulersrutilsrrrr r!r"utils.torch_utilsr#r$rWr%free_init_utilsr'pipeline_utilsr(r)pipeline_outputr+torch_xla.core.xla_modelcore xla_modelrNrM get_loggerrdrxrnrroror8r;r9r7rsDDD hhDww[[M9;3B-+D6))MM   H %;@ck TLL T-5eoo-F T\_ T{ 7"{ 7r9