Tesla V100 llama.cpp installation report
Generated: 2026-09-05 (America/New_York)
Result
The Tesla V100-class GPU is configured for CUDA inference with llama.cpp. Two model presets are installed in the llama.cpp web router, which is available at http://127.0.0.1:8080:
- -
Qwen3.8-27B - -
Gemma-4-26B-A4B-it
The router starts with both models unloaded and has models-max = 1. Selecting another model stops and unloads the active model before loading the selected model. This behavior was verified in the router log and through /v1/models.
The official Qwen3.8 release is 27B, so that is the model installed here. Its official sources are Qwen/Qwen3.8-27B and ggml-org/Qwen3.8-27B-GGUF.
Gemma uses the instruction-tuned release from google/gemma-4-26B-A4B-it and ggml-org/gemma-4-26B-A4B-it-GGUF.
Hardware and software
| Component | Installed configuration |
|---|---|
| OS | Ubuntu 26.04, kernel 7.0.0-31-generic |
| GPU | Tesla PG500-216, Volta compute capability 7.0, 32,768 MiB VRAM |
| NVIDIA driver | 580.173.02 |
| GPU power limit | 250 W |
| CUDA toolkit | 12.4.131 (nvcc 12.4) |
| llama.cpp | Commit 74a7c897f049c17e7080423aa2111776eff6ebbf |
| Build target | Release, CUDA enabled, CUDA architecture 70, OpenSSL and ccache enabled |
| CUDA host compiler | GCC/G++ 13.4 |
| Default compiler | GCC/G++ 15.2 |
| CMake / Ninja | 4.2.3 / 1.13.2 |
Installed packages include nvidia-cuda-toolkit, build-essential, gcc-13, g++-13, cmake, ninja-build, git, libssl-dev, and ccache. llama.cpp executables and libraries are installed under /home/v100test/.local and are already on the user's PATH.
The CUDA build is at /home/v100test/llama.cpp/build-cuda. Flash Attention is disabled because the current upstream CUDA Flash Attention test attempts a dynamic shared-memory configuration that the V100 rejects for an extreme attention shape. The regular CUDA backend passed its representative 802-case backend test run. Quantized V cache also requires Flash Attention in llama.cpp, so the tuned profiles use Q8 K cache and FP16 V cache.
Installed model files
| Model file | Quantization | Size | SHA-256 |
|---|---|---|---|
Qwen3.8-27B-Q4_K_M.gguf |
Q4_K_M | 18,973,870,432 bytes | 31629f53165ab6a7dad8c9847dcfd1fdf55829dac1e6e748f4a68581b0033d34 |
mtp-Qwen3.8-27B-Q4_0.gguf |
Q4_0 MTP | 1,680,271,648 bytes | 051a1764cff8c4f3ee6ae8b00593a0364c7539c67fa50ffc58f3f96509fca38e |
gemma-4-26B-A4B-it-Q4_0.gguf |
Q4_0 | 14,618,145,824 bytes | d208665ab1cd3a69f7a9a4bc59430e8448c8093d9b06334f566ac59d6d504a03 |
mtp-gemma-4-26B-A4B-it-Q4_0.gguf |
Q4_0 MTP | 251,939,520 bytes | e00b0ce326d03d8f6f6972ad2320fe90b6ff25a3da5fc51bec15efe283052aa2 |
All four checksums match the official Hugging Face repository metadata. Base-model links are under /home/v100test/models/base; MTP files are under /home/v100test/models/drafts.
Shared model configuration
Both router presets use:
- - Context size: 98,304 tokens (96 Ki tokens)
- - Parallel slots: 1
- - Main-model GPU layers: 999, which fully offloads all available layers
- - Flash Attention: off
- - K cache: Q8_0
- - V cache: FP16
- - Multimodal projector auto-loading: off; these profiles are optimized for text chat and coding
- - Jinja chat templates: enabled
- - Speculative decoding:
draft-mtp - - MTP draft GPU layers: 999
- - Draft K cache: Q8_0
- - Draft V cache: FP16
- - Maximum MTP draft length: 3 tokens
MTP means multi-token prediction and is the performance feature enabled here. MCP means Model Context Protocol; it provides external tools and data sources and does not accelerate token generation. No external MCP server configuration was supplied or installed.
Qwen3.8-27B
| Setting | Value |
|---|---|
| Base model | /home/v100test/models/base/Qwen3.8-27B-Q4_K_M.gguf |
| MTP model | /home/v100test/models/drafts/mtp-Qwen3.8-27B-Q4_0.gguf |
| Context | 98,304 |
| Batch / microbatch | 512 / 128 |
| Peak VRAM during 96K MTP validation | 26,733 MiB |
The smaller microbatch is required to leave enough V100 memory for the full context, target model, draft model, KV caches, and CUDA compute buffers.
Gemma-4-26B-A4B-it
| Setting | Value |
|---|---|
| Base model | /home/v100test/models/base/gemma-4-26B-A4B-it-Q4_0.gguf |
| MTP model | /home/v100test/models/drafts/mtp-gemma-4-26B-A4B-it-Q4_0.gguf |
| Context | 98,304 |
| Batch / microbatch | 2,048 / 512 |
| Peak VRAM during 96K MTP validation | 23,789 MiB |
Gemma's mixture-of-experts design leaves enough VRAM for the larger prompt batch.
Performance and thermal validation
| Model and mode | Prompt processing | Token generation |
|---|---|---|
| Qwen3.8-27B baseline, 512-token prompt | 746.26 tokens/s | 33.27 tokens/s |
| Qwen3.8-27B, 96K profile with MTP | 107.3 tokens/s on the short validation prompt | 45.0 tokens/s |
| Gemma-4-26B-A4B-it baseline, 512-token prompt | 1,559.69 tokens/s | 103.65 tokens/s |
| Gemma-4-26B-A4B-it, 96K profile with MTP | 23.0 tokens/s on the short validation prompt | 155.4 tokens/s |
The short MTP prompt rates include initialization and are not directly comparable with the 512-token prompt benchmark. The generation measurements showed about a 35% increase for Qwen and 50% for Gemma in these validation runs. Router API requests also recorded MTP draft acceptance of 71.4% for Qwen and 62.5% for Gemma.
| Test | Peak temperature | Peak power | Peak GPU utilization | HW thermal slowdown | SW thermal slowdown |
|---|---|---|---|---|---|
| Qwen baseline | 42 C | 251.82 W | 100% | Not active | Not active |
| Qwen 96K + MTP | 42 C | 232.84 W | 99% | Not active | Not active |
| Gemma baseline | 35 C | 85.93 W | 100% | Not active | Not active |
| Gemma 96K + MTP | 36 C | 124.52 W | 99% | Not active | Not active |
No thermal throttling occurred during any model test. Qwen reached the configured 250 W power limit during the baseline run while both NVIDIA hardware and software thermal-slowdown flags remained inactive.
Web interface, API, and commands
The web router runs as the enabled user service llama-web.service and listens only on localhost.
- - Web UI: http://127.0.0.1:8080
- - OpenAI-compatible API:
http://127.0.0.1:8080/v1 - - Model list:
http://127.0.0.1:8080/v1/models - - Start or restart:
systemctl --user restart llama-web.service - - Stop:
systemctl --user stop llama-web.service - - Status:
systemctl --user status llama-web.service - - Logs:
journalctl --user -u llama-web.service -f - - Direct Qwen terminal chat:
qwen38 - - Direct Gemma terminal chat:
gemma4 - - Manual router start, if the service is stopped:
llama-web
The main configuration is /home/v100test/.config/llama.cpp/models.ini. The service definition is /home/v100test/.config/systemd/user/llama-web.service, and the launch scripts are under /home/v100test/.local/bin.
The router and direct terminal commands should not be run at the same time because each active 26-27B model expects to use most of the GPU.