Skip to content

Instantly share code, notes, and snippets.

@brando90

brando90/training_guidelines.md

Created November 29, 2024 21:59
Show Gist options
  • Save brando90/2cfbbd9548d96d60bde9c35d08598698 to your computer and use it in GitHub Desktop.
Save brando90/2cfbbd9548d96d60bde9c35d08598698 to your computer and use it in GitHub Desktop.
Download ZIP
nothing bellow 16 bits for training
Raw
training_guidelines.md

Training Guidelines Summary

  • SFT: Use bf16 or fp32 for training; avoid 8bit. For evaluation, fp16, bf16, or fp32 is fine. Follow established scripts for reliability.
  • Unsloth: Train LoRA with fp16, bf16, or fp32. Avoid 8bit or lower unless validated through replication of original experiments. No QLoRA unless core setups are stable and everything before this has worked.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment