oi SSKrSSKrSSKJrJrJrJrJrJ r SSK r SSK r SSK r SSK JrJr SSKJrJrJrJrJrJr /SQrSrSrS r\R6r\"\5\"S 5:aA\R:R<R>r \R:R<RBr"O2\R<R?S S 9r \R<RCS S 9r"S2S \#S\\\$\#4S\\#4Sjjr%S2S \#S\\\$\#4S\\#4Sjjr&S3S\S\\\$\#4S\\'SS4Sjjr("SS\RRRT5r+"SS\RRRT5r,\RZSS.Sj5r.Sr/Sr0Sr1Sr2SS K3J4r4J5r5J6r6J7r7J8r8J9r9J:r:J;r; SS .S4S!jjr<\S";a\R:Rzr>O\S#:Xa\R~Rzr>/q@SqAS5S$jrB"S%S&\RRRT5rCSS'K3JDrDJErEJFrFJGrG \RZS\G\ES(S).S\\'S*\/\ \D\D44S+\$S,\'4S-jj5rHS5S.jrIS/rJS0rK\RZSS.S1j5rLg)6N)UnionOptionalListAnyCallableTuple) _get_dtypeVersion)is_hipget_device_type DEVICE_TYPEDEVICE_TYPE_TORCH DEVICE_COUNTALLOW_PREQUANTIZED_MODELS) calculate_n_gradient_checkpointsprepare_n_gradient_checkpoints'Unsloth_Offloaded_Gradient_Checkpointer%unsloth_offloaded_gradient_checkpoint$patch_unsloth_gradient_checkpointing&unpatch_unsloth_gradient_checkpointingUnsloth_Gradient_Checkpointerunsloth_gradient_checkpointpatch_gradient_checkpointingunpatch_gradient_checkpointing*patch_unsloth_smart_gradient_checkpointing,unpatch_unsloth_smart_gradient_checkpointing,reset_unsloth_gradient_checkpointing_buffersz2.4.0cuda device_typen_layersmethodreturnc[U5[LaUS:deUcSnUS:Xa[US-5nOE[U5[La(US:a"[[R"X- 55nO [ S5eX-n[R "X#[S9nX-n[ U5HnXBS- U- ==S- ss'M [R"S[R"U545nUR5nU$)Nrsqrtg?z3method must be 'sqrt' or an int >0 and <= n_layers.dtyper ) typeintnpceil ValueErrorfullrangehstackcumsumtolist)r%r& n_checkpointssizesizes leftoversk boundariess \/home/james-whalen/.local/lib/python3.13/site-packages/unsloth_zoo/gradient_checkpointing.py!_calculate_n_gradient_checkpointsr=Bs >S X\2 1 ~v HcM* f !BGGH$567 NOO  $D GGM 5E(I 9  Aoa A% Aryy/01J""$J layers_per_checkpointc [U5[LaUS:deUbUS:Xag[X5nUSS:Xa USU:Xde[U5S:Xa[ U5U:Xde[ R "U5R5S:deU$)Nrr )r,r-r=minmaxr.diff)r%r?r;s r<rr]s >S X\2 1$(=(B28SJ a=A *R.H"<= < z?a C Ox$?@ ? 77:  " " $ )* ) r>model use_reentrantc&Sn[US5(aUnO8[US5(a'[URS5(a URnUc [S5eUSLaSn[UR5n[ XA5nXSlX#lg)a Calculates where to place the gradient checkpoints given n_layers. Args: model: Any LlamaModel with layers. layers_per_checkpoint (`Union[str, int]`, *optional*): Can either be `sqrt` or an integer for how many layers per checkpoint you want. The more, the less memory usage, but can be slower. Default is `sqrt`. Choose 1 for Pytorch gradient checkpointing. 2 to wrap 2 layers in 1 module etc. use_reentrant (`bool`, *optional*): https://github.com/pytorch/pytorch/blob/main/torch/utils/checkpoint.py#L354 Optimal gradient checkpointing algorithm `use_reentrant=False` which will be the default in future Pytorch versions doesn't seem to work?? NlayersrEzX`model` or `model.model` does not have attribute `layers`. Are you sure this is a model?FT)hasattrrE TypeErrorlenrHr"_gradient_checkpointing_boundaries%_gradient_checkpointing_use_reentrant)rEr?rF_modelr%r;s r<rros&Fuh  5;; ) )[[F ~rss 6==!H1(RJ3=-3@0r>cN\rSrSrSr\\S55r\\S55r Sr g)rz All Unsloth Zoo code licensed under LGPLv3 Saves VRAM by smartly offloading to RAM. Tiny hit to performance, since we mask the movement via non blocking calls. cURUlURSSS9n[R"5 U"U/UQ76nSSS5 UR U5 XlX0lW$!,(df  N-=f)NcpuT non_blocking)devicetotorchno_gradsave_for_backwardforward_functionargs)ctxrZ hidden_statesr[saved_hidden_statesoutputs r<forward/Unsloth_Offloaded_Gradient_Checkpointer.forwardsk#)) +..uT.J ]]_%m;d;F 12/ _s A(( A6cURunURURSS9R5nUR S5 [ R "5 UR"U/URQ76unSSS5 [ RRWU5 SUR4S[UR5--$!,(df  NU=f)NTrSN) saved_tensorsrVrUdetachrequires_grad_rW enable_gradrZr[autogradbackwardgradrKr\dYr]r_s r<ri0Unsloth_Offloaded_Gradient_Checkpointer.backwards,,%((D(IPPR $$T*    ,,]FSXXFIV! +m((*WS]-BBB! s  C  CN __name__ __module__ __qualname____firstlineno____doc__ staticmethodtorch_amp_custom_fwdr`torch_amp_custom_bwdri__static_attributes__rnr>r<rrsI  CC r>rcN\rSrSrSr\\S55r\\S55r Sr g)rz^ All Unsloth Zoo code licensed under LGPLv3 Same as normal gradient checkpointing but cleaner c[R"5 U"U/UQ76nSSS5 URU5 XlX0lW$!,(df  N-=frc)rWrXrYrZr[)r\rZr]r[r_s r<r`%Unsloth_Gradient_Checkpointer.forwardsG]]_%m;d;F m,/ _s A AcURunUR5nURS5 [R"5 UR "U/UR Q76unSSS5 [RRWU5 SUR4S[UR 5--$!,(df  NU=f)NTrc) rdrerfrWrgrZr[rhrirjrKrks r<ri&Unsloth_Gradient_Checkpointer.backwards,,%,,. $$T*    ,,]FSXXFIV! +m((*WS]-BBB! s  B44 CrnNrornr>r<rrsI CC r>r)rFc0[R"U/UQ76$rc)rapplyfunctionrFr[kwargss r<rrs ( . .x ?$ ??r>c[S5 SSKnURRRRS:XagURRRURRl[ URRlSSKn[ URlS[RS'g)NzDUnsloth: Patched gradient checkpointing for long context finetuning.rr1UNSLOTH_PATCHED) print torch.utilsutils checkpointrp_old_checkpointrtransformers.modeling_utilsmodeling_utilsosenvironrW transformerss r<rrs PQ {{((115\\^d-2[[-C-C-N-NEKK*(MEKK%&-RL*$'BJJ !r>c[S5 SSKnURRRRS:XagURRRURRl[ URRlSSKn[ URlS[RS'g)Nz(Unsloth: Patched gradient checkpointing.rrrr) rrrrrprrrrrrrs r<rrs 45 {{((115RRTZ-2[[-C-C-N-NEKK*(CEKK%&-HL*$'BJJ !r>cSSKn[URRS5(aOURRRURRlURR?gNrrrrIrrrrWs r<rrRu{{%%'899,1KK,B,B,R,R ) KK " " 2r>cSSKn[URRS5(aOURRRURRlURR?grrrs r<rrrr>)check_backward_validity_infer_device_type_get_autocast_kwargs_get_device_moduleget_device_statesdetach_variable contextlibDefaultDeviceTypecUc[R"5nUS:Xag[U5n[X5H0upEUR U5 UR U5 SSS5 M2 g!,(df  MD=f)aSets random number generator states for the specified devices. Args: devices: Device ids to set states for. states: States to set. device_type: ``device_type`` of the devices to set states for. Default is the device returned by a call to ``DefaultDeviceType.get_device_type()``, which is ``cuda`` if not changed by calling ``DefaultDeviceType::set_device_type()``. Nmeta)rr rziprU set_rng_state)devicesstatesr$ device_modulerUstates r<set_device_statesrsi'779 f&{3MW-   ! !& )  ' ' .* ). ) )s A++ A: r"hipxpuc /qSqUcp[S:Xa&[RR 5upUS:nO[S:XaSnO [S:XaSnW(a[R O[Rn[S5H/n[R"SUS SS 9n[RU5 M1 [S ;a[RR5O[RR5n[[U5Vs/sH"n[R"S U[S U3S9PM$ sn5qSq[[U5Vs/sHIn[S:Xa[RR'5O[RR'5PMK sn5q[S ;aZ[[U5Vs/sH9n[RR+[R,"SU355PM; sn5qOc[S:XaY[[U5Vs/sH9n[RR1[R,"SU355PM; sn5q[R2"U5R4S-nSU-qSqSqSqSqgs snf![ a=n[#S5 [#S5 [#S5 [#S5 [#S5 eSnAff=fs snfs snfs snf)Nrr"rTrr!rrRr+rU pin_memoryrr :)r+rUz ========== z?Unsloth: Your setup does not support `PYTORCH_CUDA_ALLOC_CONF` zDPlease set `import os; os.environ['PYTORCH_CUDA_ALLOC_CONF'] = '';` z#Then re-run Unsloth from the start.zcuda:zxpu:i ) CPU_BUFFERS CPU_INDEXrrWr"get_device_capabilitybfloat16float16r2emptyappend device_countrtupler GPU_BUFFERS Exceptionr BACKWARD_PASSStream EXTRA_STREAMSdefault_streamrU MAIN_STREAMScurrent_streamfinfobits MINIMUM_SIZEUSE_UNSLOTH_GC LAST_GC_INDEX FIRST_PASSCURRENT_GC_INDEX) r+ major_version minor_versionSUPPORTS_BFLOAT16ixn_gpusen_bytess r<)initialize_unsloth_gradient_checkpointingr@sKI } & +0::+K+K+M (M!.!!3  E ! $  E ! $ "3 3Z KK%%d S1 +6*HUZZ $ $ &eiiNdNdNfFrwx~rArmnU[[UPaObbcdecfMghrAB Motu{o|}o|jk2Cv2M5::,,.SXS\S\ScScSeeo|}~Mo%]bci]jk]jXYuzz88aSk9RS]jkl  [`ag[hi[hVWuyy77 tA3Z8PQ[hij kk% %%*G"g-LNMJ3A  m PQ UV 34 m ~kis??J))J$:J)AK3AK8,AK=$J)) K038K++K0c6\rSrSr\S5r\S5rSrg)UnslothCheckpointFunctionicXlX l[U6Ul[ UR5uUlUlU(ad[R"5Ul SUl [UR5n[USS5(aSUl [U6uUlUl/Ul/Ul/nSUlSn['U5GHupx[R("U5(GaTUS:XGaUR*(Ga[,(a [.S- q[0S- qSUlUR35n U [4:Ga[0[.:wd [,(GaSnUR6n U R8n [:U n [<U n [>U n[@(aSq Sq![B[E[F5:a5[RH"XRJSSS9n[FRMU5 [F[BnURNnXR35:aURQU 5 XR35:aU RQU 5 USU RSU5nURUU 5 [WU5 URYUSS9 SSS5 U U[BXU4Ul-[BS- q!URMS5 [\(a [_S 5 Sq.OS Ul-URMU5 OURMU5 UR"RMU5 UR RMS5 OUR RMU5 GM UR$(aUR`"U6 [Rb"5 U"U6nSSS5 U(aW RUW5 W$!,(df  GN2=f!,(df  N:=f) NF _initializedTrr rRrrSz5Unsloth: Will smartly offload gradients to save VRAM!)NNNNNN)2 run_functionpreserve_rng_staterr$rdevice_autocast_kwargscpu_autocast_kwargsrW get_rng_state fwd_cpu_statehad_device_in_fwdrgetattrr fwd_devicesfwd_device_statesinputstensor_indices_requires_gradient enumerate is_tensor requires_gradrrrnumelrrUindexrrrrrrKrrr+rshaperesize_view wait_streamtorch_gpu_streamcopy__saved_metadatarrrYrX)r\rrr[r tensor_inputsuse_gpu_bufferrargnew_sizerU device_index GPU_BUFFER MAIN_STREAM EXTRA_STREAMrroutputss r<r`!UnslothCheckpointFunction.forwardsB (!3,d3>R OO? ;"C$;  % 3 3 5C  %*C !.s?M}ne<<(,%9JD9Q6!6  !&oFAs##6c///"z&* $)$-1C*"yy{H ,.5E5V[e[e)- "%'-|| '2\'B '3\'B '4\'B )=,1M()I%K(88 % HiiRWfj kA'..q1' 2 # #ggi/81D#&6&6&88*:L:LX:VixL--e4%00=-l;GGCG=<08 !"YZ-2N.S+%,,S1!((-""))!, !!$' !!#& W&X  ! !3#8#8-#H ]]_"D)G ;22<@=<;4_sN?O? O  Oc  UR(dg[RR5(d [ S5e[ UR 5nURnURnURupVpxpUbl[USURU5n [USURU5n U RU 5 [U 5 U RU SS9 SSS5 O[!U5S:wa USX#S'[#USSSS9H upXMX.'M SqSqSq/nUR*(aUR,(a UR.n[R0R3XR*UR4S9 UR*(aZ[R6"UR85 UR,(a)[;UR.UR<UR4S 9 [R>RAUR45(a6[R>RB"S S UR40URDD6O[FRH"5n/nUHgn[KU[RL5(dUROU5 M5URQ5n URRU l)UROU 5 Mi Ub7U RU 5 W RQ5n U RUS5 U US'[RV"5 U [R>RB"S0URXD6 URZ"U6nSSS5 SSS5 SSS5 SSS5 [KW[RL5(aU4n/n/n[][!U55H`n [R^"UU 5(dM#UU RR(dM9UROUU 5 UROX5 Mb [!U5S:XaO [RRaUU5 [cS W55n[][!U55H n SUU 'SX-'M S U-$!,(df  GN=f!,(df  GNG=f!,(df  GNQ=f!,(df  GN[=f!,(df  GNd=f)NzWhen use_reentrant=True, torch.utils.checkpoint is incompatible with .grad() or passing an `inputs` parameter to .backward(). To resolve this error, you can either set use_reentrant=False, or call .backward() without passing the `inputs` argument.TrSrr )startF)renabledr$r#r$c3|# UH2n[U[R5(a UROSv M4 g7frc) isinstancerWTensorrj).0inps r< 5UnslothCheckpointFunction.backward..gs0 &#3 55CHH4 ?&s:<)NNrn)rR)2rrWrh_is_checkpoint_valid RuntimeErrorlistrrrdrrrrrrrrKrrrrrrrrandomfork_rngr$rrrrampis_autocast_availableautocastrr nullcontextrrrrerrfrgrrr2rrir)r\r[rrtensorsrrrrrrbufferrridx rng_devicesdevice_autocast_ctxdetached_inputsrroutputs_with_gradargs_with_gradgradss r<ri"UnslothCheckpointFunction.backwards5%%d~~2244N cjj!++##NQNaNaK+   .y9>>uEFI&y166u=A  $ $[ 1!,/ Qt 40/>"a',3AJa()  qr 2A>FA!*FK?      ! !c&;&;//K \\ " ")?)?S__# %%##C$5$56((%coos7L7LZ]ZiZij00AA#())"4"4#OO#/2/I/I#GQG]G]G_ !O!#u||44#**3/JJL"%"3"3&&q)  $'' 5MMO  &%&" ""$&9599;M;M;oWZWnWn;o**O< 9S S S S" S%% S4rnN)rprqrrrsrur`rirxrnr>r<rrs8rrf w$w$p r>r)ContextManager_DEFAULT_DETERMINISM_MODE'_checkpoint_without_reentrant_generatornoop_context_fnF)rF context_fndeterminism_checkdebugrrrcUc[R"SSS9 SnURSS5nU(a+U(a$[SSR S U55-5eU(a0U[ LdUS La [S 5e[ R"X/UQ76$[XX#U/UQ70UD6n[U5 U"U0UD6n [U5 g![a U s$f=f) aCheckpoint a model or part of the model. Activation checkpointing is a technique that trades compute for memory. Instead of keeping tensors needed for backward alive until they are used in gradient computation during backward, forward computation in checkpointed regions omits saving tensors for backward and recomputes them during the backward pass. Activation checkpointing can be applied to any part of a model. There are currently two checkpointing implementations available, determined by the :attr:`use_reentrant` parameter. It is recommended that you use ``use_reentrant=False``. Please refer the note below for a discussion of their differences. .. warning:: If the :attr:`function` invocation during the backward pass differs from the forward pass, e.g., due to a global variable, the checkpointed version may not be equivalent, potentially causing an error being raised or leading to silently incorrect gradients. .. warning:: The ``use_reentrant`` parameter should be passed explicitly. In version 2.4 we will raise an exception if ``use_reentrant`` is not passed. If you are using the ``use_reentrant=True`` variant, please refer to the note below for important considerations and potential limitations. .. note:: The reentrant variant of checkpoint (``use_reentrant=True``) and the non-reentrant variant of checkpoint (``use_reentrant=False``) differ in the following ways: * Non-reentrant checkpoint stops recomputation as soon as all needed intermediate activations have been recomputed. This feature is enabled by default, but can be disabled with :func:`set_checkpoint_early_stop`. Reentrant checkpoint always recomputes :attr:`function` in its entirety during the backward pass. * The reentrant variant does not record the autograd graph during the forward pass, as it runs with the forward pass under :func:`torch.no_grad`. The non-reentrant version does record the autograd graph, allowing one to perform backward on the graph within checkpointed regions. * The reentrant checkpoint only supports the :func:`torch.autograd.backward` API for the backward pass without its `inputs` argument, while the non-reentrant version supports all ways of performing the backward pass. * At least one input and output must have ``requires_grad=True`` for the reentrant variant. If this condition is unmet, the checkpointed part of the model will not have gradients. The non-reentrant version does not have this requirement. * The reentrant version does not consider tensors in nested structures (e.g., custom objects, lists, dicts, etc) as participating in autograd, while the non-reentrant version does. * The reentrant checkpoint does not support checkpointed regions with detached tensors from the computational graph, whereas the non-reentrant version does. For the reentrant variant, if the checkpointed segment contains tensors detached using ``detach()`` or with :func:`torch.no_grad`, the backward pass will raise an error. This is because ``checkpoint`` makes all the outputs require gradients and this causes issues when a tensor is defined to have no gradient in the model. To avoid this, detach the tensors outside of the ``checkpoint`` function. Args: function: describes what to run in the forward pass of the model or part of the model. It should also know how to handle the inputs passed as the tuple. For example, in LSTM, if user passes ``(activation, hidden)``, :attr:`function` should correctly use the first input as ``activation`` and the second input as ``hidden`` preserve_rng_state(bool, optional): Omit stashing and restoring the RNG state during each checkpoint. Note that under torch.compile, this flag doesn't take effect and we always preserve RNG state. Default: ``True`` use_reentrant(bool): specify whether to use the activation checkpoint variant that requires reentrant autograd. This parameter should be passed explicitly. In version 2.5 we will raise an exception if ``use_reentrant`` is not passed. If ``use_reentrant=False``, ``checkpoint`` will use an implementation that does not require reentrant autograd. This allows ``checkpoint`` to support additional functionality, such as working as expected with ``torch.autograd.grad`` and support for keyword arguments input into the checkpointed function. context_fn(Callable, optional): A callable returning a tuple of two context managers. The function and its recomputation will be run under the first and second context managers respectively. This argument is only supported if ``use_reentrant=False``. determinism_check(str, optional): A string specifying the determinism check to perform. By default it is set to ``"default"`` which compares the shapes, dtypes, and devices of the recomputed tensors against those the saved tensors. To turn off this check, specify ``"none"``. Currently these are the only two supported values. Please open an issue if you would like to see more determinism checks. This argument is only supported if ``use_reentrant=False``, if ``use_reentrant=True``, the determinism check is always disabled. debug(bool, optional): If ``True``, error messages will also include a trace of the operators ran during the original forward computation as well as the recomputation. This argument is only supported if ``use_reentrant=False``. args: tuple containing inputs to the :attr:`function` Returns: Output of running :attr:`function` on :attr:`*args` Naetorch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.5 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.) stacklevelTrzUnexpected keyword arguments: ,c3$# UHov M g7frcrn)rrs r<r%unsloth_checkpoint..s7NvvsFzKPassing `context_fn` or `debug` is only supported when use_reentrant=False.) warningswarnpopr0joinrrrrnext StopIteration) rrFrrrr[rpreservegenrets r<unsloth_checkpointr-|sr  C   zz.5H - ,sxx7Nv7N/N N   _ ,U0B' )..xIDII5  u GK OU  S ''  I J s; C CCct[RRRRS:wao[ U5 [RRR[RRl[[RRl[RRRRS:wae[RRR[RRl[[RRlgg)Nrr-) rWrrCheckpointFunctionrpr_old_CheckpointFunctionrrr-r*s r<rrs {{0099=XX1%89>9O9O9b9b 64M 1 {{((115II161G1G1R1R .,> )Jr>c[RRRRS:XGa[ [RRS5(GaX[RRR [RRl[[[55HTn[ [US5(a[URS5 [[5[LdMKS[U'MV [[[55HTn[ [US5(a[URS5 [[5[LdMKS[U'MV Sq Sq [RR5 [ R""5 [RRRRS:Xal[ [RRS5(aB[RRR$[RRlggg)Nrr0rrr-r)rWrrr/rprIr0r2rKrrr,rrr" empty_cachegccollectrrs r<rr+s~ 11::>YY &&(ABB49KK4J4J4b4b 1s;'(A{1~y11;q>3I3I!3LK D(4+a.)s;'(A{1~y11;q>3I3I!3LK D(4+a.)      ))226JJ &&(9::,1KK,B,B,R,R ) ; Kr>c8[b[cg[[5S:Xag[[[55HnU[:aD[Ub8[ [US5(a[UR [5 MMMOMQ[Ub0[ [US5(a[UR S5 S[U'M [[5[:a [[S2 [[[55HEn[UcM[ [US5(dM)[UR [5 MG Sq Sq Sq Sq Sq Sq[R R#5 [$R&"5 g)ax All Unsloth Zoo code licensed under LGPLv3 Resets CPU_BUFFERS and GPU_BUFFERS to their initial sizes after training. This function should be called after trainer.train() completes to free up memory that was allocated during training while keeping the buffers ready for another potential training run. Unlike unpatch_unsloth_smart_gradient_checkpointing, this does NOT destroy the buffers or unpatch the checkpointing - it just resets them to their initial state. Usage: trainer.train() reset_unsloth_gradient_checkpointing_buffers() # Free memory, stay ready # Can run trainer.train() again without re-initializing NrrT)rrrKr2INITIAL_CPU_BUFFER_COUNTrIrINITIAL_CPU_BUFFER_SIZEINITIAL_GPU_BUFFER_SIZErrrrrrrWr"r2r3r4r5s r<rrEsV6k1 ;13{# $ ' '1~)gk!ni.P.PA&&'>?/Q)1~)gk!ni.P.PA&&q)!KN%  ;22 01 23{# $ q> %'+a.)*L*L N " "#: ;% IMMJN JJJJLr>c[[5S:Xa[USR5 [R "U/UQ76$)Nr)rKrrr+rrrs r<rrs7 ;11$q'--@ $ * *8 ;d ;;r>)r))r)T)r'Nrc)MrWnumpyr.typingrrrrrrrr$r3rr r r$r r rrrr__all__r8r9r7 __version__ torch_versionr"r  custom_fwdrv custom_bwdrwr-strr=rboolrrhFunctionrr_disable_dynamorrrrrtorch.utils.checkpointrrrrrrrrrstreamrrrrrrrrrrr-rrrrrnr>r<rHsA" >>  & &%(!!  =GG,, ::>>44 ::>>44 99//f/E 99//f/E ,2c3h( #Y0 9?$U38_5 #Y 9?-1$A$A$U38_5$A%TN$A $AJ enn.E.E < ENN$;$; 6AE@@((     7;/$/!zz((Eyy''  <zr  7 7r f %)FU6 ]D>]U>>#ABBC ]  ]  ]]| ?S.BFKO<< r>