Text Generation
Merging Models
Task Arithmetic

A New Type of Merge

Task Arithmetic. Computes "task vectors" for each model by subtracting a base model. Merges the task vectors linearly and adds back the base. The config looks like this:

models:
  - model: ise-uiuc_Magicoder-S-CL-7B
    parameters:
      weight: 1
  - model: glaiveai_glaive-coder-7b
    parameters:
      weight: .5
  - model: Xwin-LM_XwinCoder-7B
    parameters:
      weight: .5
  - model: nvidia_OpenMath-CodeLlama-7b-Python-hf
    parameters:
      weight: .5
merge_method: task_arithmetic
base_model: LargeWorldModel_LWM-Text-Chat-1M
parameters:
  normalize: true
  int8_mask: true
dtype: float16
SLERPTIES