h-&SSKJr SSKrSSKJs Js Jr SSKJr SSKJ r /SQr S\RS\ S\ S \\R\R44S jr \R\R4S \ S \ S \ S \R4SjjrS\RS\RS\ S\ S \\R\R\R44 SjrS\RS\RS\RS\R$S\ S\ S \\R\R44SjrS\RS\ S\ S\ S \\R\R44 SjrS\RS\RS\ S\ S\ S \R4 SjrS\RS\ S\ S\ S \R4 SjrS\RS\ S\ S\ S \R4 SjrS\RS\ S\ S\ S\ S \R4 S jrS\RS\ S\ S\ S\ S \R4 S!jrg)")TupleN)const) mask_creator) inject_24marlin_24_workspacepack_to_marlin_24unpack_from_marlin_24wsize_ksize_nreturncURX4:Xde[UR55R5R5R 5nX0-R 5UR 54$)aInjects 2:4 sparsity into a weight tensor. The sparsity is applied in a 2:4 ratio, where for every group of 4 weights, 2 will be pruned based on their value. The mask will be created based on the ranked weight values. Args: w (torch.Tensor): The weight tensor to inject sparsity into. size_k (int): The number of input features. size_n (int): The number of output features. Returns: Tuple[torch.Tensor, torch.Tensor]: The pruned weight tensor and the mask tensor. )shapertcudabool contiguous)r r r masks Z/home/james-whalen/.local/lib/python3.13/site-packages/torchao/sparsity/marlin/__init__.pyrrsa 77v& && &   " ' ' ) . . 0D H "DOO$5 55 out_features min_thread_n max_parallelcX-S:Xd SUSU35eX-U-n[R"U[RSS9$)aCreates a workspace for marlin 2:4 quantization. The workspace is used to coordinate the locks during the execution of the kernel. Args: out_features (int): The number of output features. min_thread_n (int, optional): The minimum number of threads per block. Defaults to `MARLIN_24_MIN_THREAD_N`. max_parallel (int, optional): The maximum number of parallel threads. Defaults to `MARLIN_24_MAX_PARALLEL`. Returns: torch.Tensor: The workspace tensor fully initialized with zeros. rzout_features = z, min_thread_n = rdtypedevice)torchzerosint)rrrmax_workspace_sizes rrr$sU  &! + ,'8G +'6,F ;;)6 JJrq_w_24scalesnum_bits group_sizec~URupE[XXR5upgUS-n[XhXR5n [XXSU5n XU4$)aPacks the quantized weights and scales into the marlin 2:4 format. Args: q_w_24 (torch.Tensor): The quantized weight tensor with 2:4 sparsity applied. scales (torch.Tensor): The scale tensor. num_bits (int): The number of bits used for quantization. group_size (int): The group size that was applied during quantization. Returns: Tuple[torch.Tensor, torch.Tensor, torch.Tensor]: The packed quantized weights, the packed scales, and the meta tensor. )r_compress_quantized_24_weight_to_marlin_weights_to_marlin_scales) r"r#r$r% in_featuresr q_w_24_compmetain_features_compmarlin_24_q_w_comp marlin_24_ss rrr:sc !' K6\K#a',|$\xK D 00rr,r-original_shapeclUupg[U/UQUPUP76nUS-n [X Xu5n [XXU5n X4$)aUnpacks the quantized weights and scales from the marlin 2:4 format. Args: q_w_24_comp (torch.Tensor): The packed quantized weights. scales (torch.Tensor): The packed scales. meta (torch.Tensor): The meta tensor. original_shape (torch.Size): The original shape of the weight tensor. group_size (int): The group size that was applied during quantization. num_bits (int): The number of bits used for quantization. Returns: Tuple[torch.Tensor, torch.Tensor]: The unpacked quantized weights and scales. r')_from_marlin_scale_from_marlin_weights_decompress_quantized_24_weight) r,r#r-r1r%r$r+runpacked_scalesr.unpacked_q_w_24_compunpacked_q_w_24s rr r _sc&!/K)W.W*WhWO"a'0| 6$4HO  ++rq_24cjURX4:XdeSU-S- nUS-S-nX- nUR5R5n[R"U5upxUR5R5nXu-n UR URSS-URSS-5nX4$)aCompresses the quantized weights to a 2:4 sparse format. Normalizes the weights over 0 before compressing them. Args: q_24 (torch.Tensor): The quantized weight tensor. size_k (int): The number of input features. size_n (int): The number of output features. num_bits (int): The number of bits used for quantization. Returns: Tuple[torch.Tensor, torch.Tensor]: The compressed quantized weight tensor and the meta tensor. r'r)rrrutils)sparse_semi_structured_from_dense_cutlassresize_) r9r r r$ max_q_valzp q_24_no_zpq_24_no_zp_compr- q_24_comps rr(r(s ::&) )) )h!#I a-A BJ**,J!KKJWO%'')446O $I << 1 *DJJqMA,= >D ?rrCcURX#4:XdeURURSS-URSS-5nSU-S- nUS-S-nX- nUR5R5n[R "Xq5nUR5R5nURURSS-URSS-5nX-n U $)aDecompresses the quantized weights from a 2:4 sparse format and restores the original shape. Args: q_24_comp (torch.Tensor): The compressed quantized weight tensor in 2:4 sparse format. meta (torch.Tensor): The meta tensor. size_k (int): The number of input features. size_n (int): The number of output features. num_bits (int): The number of bits used for quantization. Returns: torch.Tensor: The decompressed quantized weight tensor. r;r'r)rr>rrr<'sparse_semi_structured_to_dense_cutlass) rCr-r r r$r?r@rBrAr9s rr5r5s ??v. .. . << 1 *DJJqMA,= >Dh!#I a-A BnO&'')446O>>UJ**,J << 1 *DJJqMA,= >D ?D Krq_wc[R"U5un n[R"XX$5n[R"U5nUR [ R 5n[ R"URSURSU-4[ R URS9n[U5HnXpSS2USU24X8---nM UR [ RS9nU$)aConverts a quantized and 2:4 sparse format weight tensor to the marlin 2:4 format. Args: q_w (torch.Tensor): The quantized weight tensor in 2:4 sparse format. size_k (int): The number of input features. size_n (int): The number of output features. num_bits (int): The number of bits used for quantization. Returns: torch.Tensor: The weight tensor in the marlin 2:4 format. rr;rNr) r< get_perms_24marlin_permute_weightsget_pack_factortorint64rrrrangeint32) rFr r r$perm_24_ pack_factorq_packedis rr)r)s"&&x0MGQ  & &sF DC''1K && C{{ 1syy|{23kkzzH ; 1>k>)*x|<< {{{-H OrrSc[R"U5un n[R"U5nUR[R 5n[R "URSURSU-4[R URS9n[U5HnXU-- SU-S- -USS2USU24'M UR[RS9n[R"XqX$5n U $)aConverts a weight tensor in the marlin 2:4 format to a regular quantized 2:4 sparse format. Args: q_packed (torch.Tensor): The weight tensor in the marlin 2:4 format. size_k (int): The number of input features. size_n (int): The number of output features. num_bits (int): The number of bits used for quantization. Returns: torch.Tensor: The weight tensor in the quantized 2:4 sparse format. rr;rNrH) r<get_reverse_perms_24rKrLrrMrrrrNrOreverse_marlin_permute_weights) rSr r r$rPrQrR q_w_unpackedrTq_w_comps rr4r4s..x8MGQ''1K {{5;;'H;;  HNN1- ;<kkL ; +31 +E (]a + Q;&'  ???5L33fH Orc[R"U5upVnX1:a*US:wa$URS[U545SS2U4nO#URS[U545SS2U4nURSU45R 5nU$)aConverts a scale tensor to the format necessary for marlin. Args: scales (torch.Tensor): The scale tensor. size_k (int): The number of input features. size_n (int): The number of output features. group_size (int): The group size that was applied during quantization. num_bits (int): The number of bits used for quantization. Returns: torch.Tensor: The scale tensor in the marlin format. N)r<rIreshapelenrr#r r r%r$rQ scale_perm_24scale_perm_single_24s rr*r*s.3-?-?-I*A*zR/S%7 89!]:JKS)=%> ?@ # # ^^RL ) 4 4 6F Mrc"[R"U5upVnX1:a>US:wa8URS[U545SS2U4nURX-U45$URS[U545SS2U4nURS5$)aConverts a scale tensor from the marlin format to their original format. Args: scales (torch.Tensor): The scale tensor in the marlin format. size_k (int): The number of input features. size_n (int): The number of output features. group_size (int): The group size that was applied during quantization. num_bits (int): The number of bits used for quantization. Returns: torch.Tensor: The scale tensor in their original format r[N)r;r[)r<rVr\r]r^s rr3r35s.3-G-G-Q*A*zR/S%7 89!]:JK~~v3V<==S)=%> ?@ # # ~~g&&r)typingrrtorchao.sparsity.marlin.utilssparsitymarlinr<rtorchao.sparsity.utilsr__all__Tensorr r MIN_THREAD_N MAX_PARALLELrrSizer r(r5r)r4r*r3rrrms --// 6 ||6 6*-6 5<< %&6*****KKKK \\ K,"1 LL"1 LL"1"1 "1  5<<u|| 34 "1J$,$, LL$, ,,$,JJ $,  $,  $, 5<< %&$,N  ,,  # -0