Before you fine-tune a language model on your own data, the key question is which GPU it will fit on. We explain what consumes VRAM, how LoRA, QLoRA and full training differ, approximate memory needs by model size and how to keep costs under control.
Open source language models let you build a company-specific assistant, classifier or text generator by fine-tuning on your own data. But before you start, there is a practical question to answer: which GPU will this training fit on?
A wrong guess is expensive in two ways. Pick a card with too little memory and training stops with an "out of memory" error, wasting hours. Pick a card that is larger than needed and you pay for capacity you do not use.
In this article we explain what consumes VRAM, how LoRA, QLoRA and full training differ in memory terms, and which GPUs roughly suit each model size.
What Consumes VRAM During Training?
During training, GPU memory holds far more than the model weights. Memory need is the sum of several components:
- Model weights: 2 bytes per parameter at 16-bit precision (FP16 or BF16). With 4-bit quantisation, roughly 0.5 to 0.6 bytes per parameter.
- Gradients: one gradient is kept for each trained parameter. In full training, that means as much extra memory as the weights.
- Optimizer states: common optimizers such as Adam and AdamW keep two extra values per parameter. These are usually stored in 32-bit, so they can be the largest memory item.
- Activations: intermediate values computed in the forward pass. They grow with batch size and sequence length. Gradient checkpointing reduces them at the cost of somewhat slower training.
- Framework and CUDA overhead: extra space used by libraries and memory fragmentation, usually a few GB.
A widely used rule of thumb is that full training with mixed precision and the Adam optimizer needs roughly 16 bytes per parameter or more, with activations on top. So even full training of a 7 billion parameter model needs more than 110 GB before activations.
LoRA, QLoRA and Full Training Compared
Full fine-tuning
All model parameters are updated. It is the most flexible method but the most expensive in memory and time. Large models almost always need multiple GPUs and memory sharding techniques such as FSDP or DeepSpeed ZeRO.
LoRA
LoRA (Low-Rank Adaptation) freezes the base model weights and trains only small adapter matrices. The number of trained parameters is usually a very small percentage of the total. That dramatically reduces the memory needed for gradients and optimizer states. The base model stays in 16-bit, so the weights still take about 2 bytes per parameter.
QLoRA
QLoRA combines the LoRA approach with a 4-bit quantised base model. Base weights shrink to about a quarter, while adapters are trained at higher precision. On many tasks it delivers results close to LoRA while running on much smaller cards. The trade-off is that training is usually a little slower because of quantisation overhead.
Approximate VRAM Needs by Model Size
The table below contains approximate figures for initial planning. Real memory use can vary significantly with sequence length, batch size, LoRA rank, whether gradient checkpointing is used and library versions. Always run a short trial before deciding.
| Model size | QLoRA (4-bit base) | LoRA (16-bit base) | Full training (16-bit, Adam) |
|---|---|---|---|
| 7B / 8B | About 8-16 GB; L4 24 GB, RTX 4090 24 GB | About 18-30 GB; A40 48 GB, L40S 48 GB | About 120 GB or more; multiple GPUs (for example several A100 / H100 80 GB) or H200 141 GB |
| 13B / 14B | About 12-24 GB; RTX 4090 24 GB (tight), A40 48 GB | About 32-48 GB; A40 / L40S 48 GB (tight), A100 80 GB | About 220 GB or more; multiple GPUs |
| 32B / 34B | About 24-40 GB; A40 48 GB, L40S 48 GB | About 70-100 GB; RTX PRO 6000 96 GB, H200 141 GB | About 520 GB or more; multiple GPUs |
| 70B | About 45-70 GB; A100 / H100 80 GB, RTX PRO 6000 96 GB | About 150-180 GB; B200 180 GB (tight) or multiple GPUs | About 1.1 TB or more; large multi-GPU cluster |
The takeaway is clear: if you want to work on a single GPU, QLoRA for models up to 70B and LoRA for small and medium models are the most practical routes. Full training requires a serious multi-GPU setup except for small models, and in most business scenarios LoRA-based methods are good enough.
If you need a multi-GPU configuration, mention it at the quote stage; exact capacity and price are confirmed in the quote. You can see all card options on our GPU cloud page.
Which GPU Should You Choose?
| GPU | VRAM | Starting price (hourly, excl. VAT) | Typical fine-tuning use |
|---|---|---|---|
| L4 | 24 GB | $0.74 | QLoRA experiments on small models |
| RTX 4090 | 24 GB | $1.11 | 7B-13B QLoRA, fast experiments |
| A40 | 48 GB | $0.74 | 7B LoRA, 32B QLoRA, economical long runs |
| L40S | 48 GB | $1.36 | 7B-13B LoRA, 32B QLoRA |
| RTX PRO 6000 | 96 GB | $3.11 | 32B LoRA, 70B QLoRA |
| A100 | 80 GB | $2.24 | 13B LoRA, 70B QLoRA, mature software support |
| H100 | 80 GB | $4.99 | Long training runs where speed matters |
| H200 | 141 GB | $6.61 | LoRA on large models, full training of small models |
| B200 | 180 GB | $9.99 | The heaviest training jobs, newest generation |
Once memory is sufficient, the decision comes down to speed versus hourly price. H100 is substantially faster than A100 for transformer training; because it finishes sooner, it can be cheaper overall despite its higher hourly price. The best way to find out is to run the same small trial on both cards and compare tokens processed per second.
Data Preparation: The Invisible Half of Training
GPU hours are expensive, so finish as much data preparation as possible before you start the GPU server.
- Settle the format: instruction-response pairs, chat format or plain text; it must match the template your training library expects.
- Clean it: remove duplicate, broken or very short examples. Less but clean data often beats more but messy data.
- Check sequence length: choose a sensible maximum length that fits most examples. Unnecessarily long sequences increase activation memory and time.
- Hold out a validation set: keep part of the data out of training to see whether the model really improves.
- Remove personal data: strip or anonymise names, phone numbers, ID numbers and similar details from training data.
Checkpoints and S3
On CloudRUX GPU servers, the temporary disk is deleted when the server stops. To avoid losing the results of hours of training, regularly save checkpoints and LoRA adapters to a persistent volume or to S3 object storage.
A practical setup looks like this: the training script saves a checkpoint every set number of steps, checkpoints are copied to an S3-compatible bucket, and when training ends you keep only the best adapter and the evaluation report. LoRA adapters are usually far smaller than the base model, so storing them is cheap.
This approach also adds flexibility: you can start training on one card and, if needed, resume from a checkpoint on another.
Keeping Costs Under Control
- Try a small model first: validate the data pipeline and hyperparameters on a 7B-8B model with a cheap card.
- Start with QLoRA: if the results are good enough, you do not need larger cards.
- Run a short measurement: measure tokens per second and calculate total time and cost.
- Shut down idle servers: as long as your data lives on persistent storage, stopping the server is safe.
- Use ready templates: PyTorch and CUDA templates cut paid hours spent on drivers and libraries. Your own container image can pin the environment completely.
For example, suppose you finish QLoRA training of an 8B model on an A40 in 20 hours: 20 x 0.74 = $14.80 + VAT. Doing the same job with LoRA on an A100 will usually cost more because of the higher hourly price, but since it finishes sooner, the gap may be smaller than you expect.
Data Location and KVKK
CloudRUX GPU servers run in our partner's data centers abroad. If training data contains personal data, this can count as a cross-border transfer under KVKK, the Turkish data protection law. We therefore recommend preparing training data without personal data or anonymising it. Where you must work with personal data, we help with the standard contract processes.
Frequently Asked Questions
Is a LoRA-trained model as good as a fully trained one?
For many business tasks, such as writing in a certain style, classification or learning domain terminology, LoRA-based methods can come close to full training. Results depend on your data and task, so starting with a small trial is best.
Can I train a 13B model on a 24 GB card?
With QLoRA, gradient checkpointing and a sensible sequence length it is often possible, but tight. For longer sequences or larger batches, a 48 GB card gives more comfortable headroom.
Are the VRAM figures in the table exact?
No, they are approximate values for initial planning. Real usage changes with sequence length, batch size, LoRA rank and library versions.
Where should I save the model after training?
The temporary disk is deleted when the server stops, so save the model and adapters to a persistent volume or to S3 object storage.
Fine-Tuning on CloudRUX GPU Servers
The CloudRUX GPU service is in early access, offering cards from L4 to B200, ready PyTorch/CUDA templates, SSH and Jupyter access, e-invoicing in Turkey and Turkish-speaking engineer support. Check our GPU cloud page to pick the card that matches your model size and method, or contact us to review your training plan together and get a quote.