+h8SSKJrJrJr SSKrSSKJr SSKJrJ r SSK J r SSK J r SSKJrJr SS KJrJr SS KJr SS KJr SS KJrJrJr \ R8"\5r"S S\R>5r "SS\\5r!g))AnyDictOptionalN) ConfigMixinregister_to_config)logging)LuminaFeedForward) AttentionLuminaAttnProcessor2_0)&LuminaCombinedTimestepCaptionEmbeddingLuminaPatchEmbed)Transformer2DModelOutput) ModelMixin)LuminaLayerNormContinuousLuminaRMSNormZeroRMSNormc^\rSrSrSrSS\S\S\S\S\S\S \S \S \S S 4U4SjjjrSS\ RS\ RS\ RS\ RS\ RS\ RS\ \ \ \44SjjrSrU=r$)LuminaNextDiTBlock$a A LuminaNextDiTBlock for LuminaNextDiT2DModel. Parameters: dim (`int`): Embedding dimension of the input features. num_attention_heads (`int`): Number of attention heads. num_kv_heads (`int`): Number of attention heads in key and value features (if using GQA), or set to None for the same as query. multiple_of (`int`): The number of multiple of ffn layer. ffn_dim_multiplier (`float`): The multiplier factor of ffn layer dimension. norm_eps (`float`): The eps for norm layer. qk_norm (`bool`): normalization for query and key. cross_attention_dim (`int`): Cross attention embedding dimension of the input text prompt hidden_states. norm_elementwise_affine (`bool`, *optional*, defaults to True), dimnum_attention_heads num_kv_heads multiple_offfn_dim_multipliernorm_epsqk_normcross_attention_dimnorm_elementwise_affinereturnNc \>[T U]5 X-Ul[R"[ R "U/55Ul[USX-U(aSOSUUSSS[5S9 Ul [R"5URl [UUX-U(aSOSUUSSS[5S9 Ul [U[SU-S- 5UUS9Ul[#UUU S9Ul['XU S 9Ul['XU S 9Ul['XU S 9Ul['XU S 9Ulg) Nlayer_norm_across_headsh㈵>F) query_dimrdim_headrheadskv_headsepsbiasout_bias processorr)r inner_dimrr) embedding_dimrr )r)elementwise_affine)super__init__head_dimnn Parametertorchzerosgater r attn1Identityto_outattn2r int feed_forwardrnorm1r ffn_norm1norm2 ffn_norm2 norm1_context) selfrrrrrrrrr __class__s h/home/james-whalen/.local/lib/python3.13/site-packages/diffusers/models/transformers/lumina_nextdit2d.pyr2LuminaNextDiTBlock.__init__5s8 2 LL.A-B!CD  $/18-d%!,.   KKM  3/18-d%!,.   .%#+/*#1  '$;  !G^_SCZ[  G^_$%8[rs hidden_statesattention_maskimage_rotary_embencoder_hidden_states encoder_masktembcross_attention_kwargsc UnURX5uppUR"SU U UUUS.UD6n URU5nUR"SU UUUSS.UD6nXRR 5R SSSS5-nX-nURS5nURRS"U5nXRS5R 5URU5--nURURU5SU RS5--5nXRS5R 5URU5--nU$)a Perform a forward pass through the LuminaNextDiTBlock. Parameters: hidden_states (`torch.Tensor`): The input of hidden_states for LuminaNextDiTBlock. attention_mask (`torch.Tensor): The input of hidden_states corresponse attention mask. image_rotary_emb (`torch.Tensor`): Precomputed cosine and sine frequencies. encoder_hidden_states: (`torch.Tensor`): The hidden_states of text prompt are processed by Gemma encoder. encoder_mask (`torch.Tensor`): The hidden_states of text prompt attention mask. temb (`torch.Tensor`): Timestep embedding with text prompt embedding. cross_attention_kwargs (`Dict[str, Any]`): kwargs for cross attention. )rIrLrJquery_rotary_embkey_rotary_embNr)r?r9rCr<r8tanhviewflattenr; unsqueezerAr>r@rB)rDrIrJrKrLrMrNrOresidualnorm_hidden_statesgate_msa scale_mlpgate_mlpself_attn_outputnorm_encoder_hidden_statescross_attn_outputmixed_attn_output mlp_outputs rFforwardLuminaNextDiTBlock.forwardvst,!=AJJ}<[9i:: ,"4)-+  %  &*%7%78M%N" JJ ,"<'-  %  . 0@0E0EaBPQ0RR,@-55b9 ))!,->?  #5#5a#8#=#=#?$**]B[#[[ &&t~~m'DIL_L_`aLbHb'cd %(:(:1(=(B(B(Dt~~V`Ga(aa rH) r9r<r>r@rBr8r3r?rCrA)T)N)__name__ __module__ __qualname____firstlineno____doc__r=floatboolr2r6Tensorrrstrrre__static_attributes__ __classcell__rEs@rFrr$s4)-?t ?t!?t ?t  ?t " ?t?t?t!?t"&?t ?t?tR<@9||9 9 ,, 9 %|| 9 ll 9ll9!)c3h 899rHrc ^\rSrSrSr/SQr\SS\S\\S\\S\\S \\S \\S \\S \\S \\ S\\ S\\ S\\ S\\S\\ SS4U4Sjjj5r SS\ RS\ RS\ RS\ RS\ RS\\\4S\ R4SjjrSrU=r$)LuminaNextDiT2DModela LuminaNextDiT: Diffusion model with a Transformer backbone. Inherit ModelMixin and ConfigMixin to be compatible with the sampler StableDiffusionPipeline of diffusers. Parameters: sample_size (`int`): The width of the latent images. This is fixed during training since it is used to learn a number of position embeddings. patch_size (`int`, *optional*, (`int`, *optional*, defaults to 2): The size of each patch in the image. This parameter defines the resolution of patches fed into the model. in_channels (`int`, *optional*, defaults to 4): The number of input channels for the model. Typically, this matches the number of channels in the input images. hidden_size (`int`, *optional*, defaults to 4096): The dimensionality of the hidden layers in the model. This parameter determines the width of the model's hidden representations. num_layers (`int`, *optional*, default to 32): The number of layers in the model. This defines the depth of the neural network. num_attention_heads (`int`, *optional*, defaults to 32): The number of attention heads in each attention layer. This parameter specifies how many separate attention mechanisms are used. num_kv_heads (`int`, *optional*, defaults to 8): The number of key-value heads in the attention mechanism, if different from the number of attention heads. If None, it defaults to num_attention_heads. multiple_of (`int`, *optional*, defaults to 256): A factor that the hidden size should be a multiple of. This can help optimize certain hardware configurations. ffn_dim_multiplier (`float`, *optional*): A multiplier for the dimensionality of the feed-forward network. If None, it uses a default value based on the model configuration. norm_eps (`float`, *optional*, defaults to 1e-5): A small value added to the denominator for numerical stability in normalization layers. learn_sigma (`bool`, *optional*, defaults to True): Whether the model should learn the sigma parameter, which might be related to uncertainty or variance in predictions. qk_norm (`bool`, *optional*, defaults to True): Indicates if the queries and keys in the attention mechanism should be normalized. cross_attention_dim (`int`, *optional*, defaults to 2048): The dimensionality of the text embeddings. This parameter defines the size of the text representations used in the model. scaling_factor (`float`, *optional*, defaults to 1.0): A scaling factor applied to certain parameters or layers in the model. This can be used for adjusting the overall scale of the model's operations. )patch_embeddernormffn_normN sample_size patch_size in_channels hidden_size num_layersrrrrr learn_sigmarrscaling_factorr!cb>[TU]5 XlX lX0lU (aUS-OUUlX@lX`lXF-UlXl [X#USS9Ul [R"[R"U55Ul[#[%US5U S9Ul[R("[+U5Vs/sHn[-UUUUU U U U 5PM sn5Ul[1U[%US5SSSX"-UR -S9UlXF-S -S :XdS 5egs snf) Nr T)rzr{ embed_dimr*i)r|rFgư>)r/conditioning_embedding_dimr0r)r*out_dimrz+2d rope needs head dim to be divisible by 4)r1r2ryrzr{ out_channelsr|rr3rrrvr4r5r6empty pad_tokenrmintime_caption_embed ModuleListrangerlayersrnorm_out)rDryrzr{r|r}rrrrrr~rrr_rEs rFr2LuminaNextDiT2DModel.__init__sD$ &$&/:K!O &#6 #: ,.!kX\ ekk+&>?"HK.DW# mmz* +A#' &' +   2%'*;'=$+d.?.??   2a71<k>kk<1 sD,rItimesteprLrMrKrOc JURX5uppURUR5nURX#U5n UR 5nUR Hn U "UUUUUU US9nM UR X5nUR=pU SupURS5nX-X--nUSS2SU24RUX-X-XUR5nURSSSSSS5RSS5RSS5nU(dU4$[US 9$) a] Forward pass of LuminaNextDiT. Parameters: hidden_states (torch.Tensor): Input tensor of shape (N, C, H, W). timestep (torch.Tensor): Tensor of diffusion timesteps of shape (N,). encoder_hidden_states (torch.Tensor): Tensor of caption features of shape (N, D). encoder_mask (torch.Tensor): Tensor of caption masks of shape (N, L). )rNrOrNrSrr r)sample)rvtodevicerrmrrrzsizerXrpermuterYr)rDrIrrLrMrKrO return_dictmaskimg_sizerNlayer height_tokens width_tokensheightwidth batch_sizesequence_lengthoutputs rFreLuminaNextDiT2DModel.forward#sK&;?:M:Mm:n7 X+..}/C/CD&&x U#((* [[E! %'=M! m: (,6   "''* !2u7LM%a)9/)9&9:?? /1F eievev &&q!Q1a8@@AFNNqRST9 'v66rH) r3r|r{rrrrrrvrzryrr)r ri rNNr$TTig?)NT)rgrhrirjrk _skip_layerwise_casting_patternsrr=rrlrmr2r6rnrrorrerprqrrs@rFrtrts+Z(N$$%%&%)$&-/&*%(.2$(&*"&-1*->l>lSM>lc] >l c] >l SM >l&c]>lsm>lc]>l%UO>l5/>ld^>l$>l&c]>l!>l !>l>lN2637||37,,37 %|| 37 ll 37  ,, 37!%S#X37 3737rHrt)"typingrrrr6torch.nnr4configuration_utilsrrutilsr attentionr attention_processorr r embeddingsrrmodeling_outputsrmodeling_utilsr normalizationrrr get_loggerrgloggerModulerrtrVrHrFrsg'& B)C8'QQ   H %KK\d7:{d7rH