You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Idea: implement automatically adding the default executor for special-purpose symbols - i.e. the ability for symbols to request "if you see me and don't have any other executor for it, include executor foo".
Apparently, we often have transforms that introduce new symbols linked to one particular executor.
This means that the user has to apply the transform and specify the executor to be used in addition to the transform where it would seem to be evident that we need it for the new symbol.
So for e.g. quantization you currently need
jm = thunder.jit(
model_fp_reference,
executors=(bitsandbytes_executor,),
early_transforms=[BitsAndBytesLinearQuant4bit()],
)
but clearly, the bnb_matmul_nf4 added to the graph by BitsAndBytesLinearQuant4bit belongs to the bitsandbytes_executor, so why would the user need to specify executors here (and decide whether they also want the default executors or not etc...).
Idea: implement automatically adding the default executor for special-purpose symbols - i.e. the ability for symbols to request "if you see me and don't have any other executor for it, include executor foo".
Apparently, we often have transforms that introduce new symbols linked to one particular executor.
This means that the user has to apply the transform and specify the executor to be used in addition to the transform where it would seem to be evident that we need it for the new symbol.
So for e.g. quantization you currently need
but clearly, the
bnb_matmul_nf4
added to the graph byBitsAndBytesLinearQuant4bit
belongs to thebitsandbytes_executor
, so why would the user need to specify executors here (and decide whether they also want the default executors or not etc...).This is from a discussion with @tfogal
The text was updated successfully, but these errors were encountered: