Fine-Tuning Small Language Models (SLMs) on Custom Domain Data with LoRA and Unsloth

The Power of Specialized Small Language Models

While general-purpose frontier models excel at general knowledge, fine-tuned 3B to 8B parameter models frequently outperform 70B models on specialized enterprise domain tasks—including medical triage, legal contract parsing, and SQL generation.

Using Unsloth and QLoRA, training these models requires only a single consumer GPU (such as an RTX 4080 or RTX 4090).

💻 Complete Training Script with Unsloth

from unsloth import FastLanguageModel
import torch

max_seq_length = 2048
dtype = None # Auto detection
load_in_4bit = True # 4bit quantization

# 1. Load Pre-trained Base Model
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Qwen2.5-7B-Instruct-bnb-4bit",
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
)

# 2. Add LoRA Adapters
model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_alpha = 16,
    lora_dropout = 0,
    bias = "none",
    use_gradient_checkpointing = "unsloth",
)

# 3. Export to GGUF / Ollama for Local Inference
model.save_pretrained_gguf("custom_slm_q4_k_m", tokenizer, quantization_method = "q4_k_m")
print("Fine-tuning and GGUF quantization complete!")

Summary & Best Practices

1. Dataset Quality over Quantity: 1,000 pristine instruction-response pairs yield higher accuracy than 50,000 noisy scraped samples.
2. Strict Evaluation Benchmarks: Test fine-tuned models against an un-seen validation split to prevent catastrophic forgetting.