Adam Anwar
  1. Work
  2. LLM Inference & Quantization Benchmarker (LIOB)
  3. Docs

LLM Inference & Quantization Benchmarker (LIOB)

Local benchmark for quantized LLMs that compares speed, memory and judge-scored quality on Apple Silicon.

14 min read

Source on GitHub

██╗     ██╗ ██████╗ ██████╗ 
██║     ██║██╔═══██╗██╔══██╗
██║     ██║██║   ██║██████╔╝
██║     ██║██║   ██║██╔══██╗
███████╗██║╚██████╔╝██████╔╝
╚══════╝╚═╝ ╚═════╝ ╚═════╝

An automated, local framework for systematically profiling the performance, memory utilization, and response quality of quantized Large Language Models (LLMs) on edge devices.

1. Project Overview & Quick Start

The LLM-Inference-Quantization-Benchmarker (LIOB) framework is designed to help researchers and developers navigate the complex trade-offs of Post-Training Quantization (PTQ). By compiling and comparing key system performance metrics (throughput, Time to First Token, peak VRAM, peak RAM, and CPU usage) alongside LLM-evaluated response quality metrics, LIOB locates the optimal deployment precision for any target local environment.

Quick Start

To get started immediately on macOS (Apple Silicon):

# 1. Start Ollama service (if not already running)
ollama serve > /tmp/ollama.log 2>&1 &

# 2. Make the orchestrator executable and launch it
chmod +x run.sh
./run.sh

The script will setup the virtual environment, pull the judge model (llama3.2:3b), compile GGUF precision variants for Qwen2.5-0.5B-Instruct, run the profiling suite, and spin up an interactive dashboard at: http://localhost:8000.

2. Directory Structure & Architecture

.
├── .gitignore
├── README.md               # You are here (Repository & Paper Documentation)
├── requirements.txt        # Python libraries
├── run.sh                  # Virtual environment setup & web server runner
├── benchmark.py            # Primary Python benchmarking engine
├── data/                   # Directory containing downloaded GGUF files (ignored)
├── venv/                   # Python Virtual Environment directory (ignored)
└── web/                    # Dashboard UI Web Application
    ├── index.html          # Core dashboard HTML structural file
    ├── style.css           # Glassmorphic custom styling sheet
    ├── app.js              # Fetching and rendering Chart.js visualizations
    └── results/            # Compilation directory for outputs
        ├── benchmark_results.json # Raw benchmark stats
        ├── benchmark_results.csv  # Flattened metric format
        └── benchmark_summary.png  # Generated static visualization charts

Core Architecture Workflow

graph TD
    A[Start: run.sh] --> B[Set up Python venv & Install dependencies]
    B --> C[Launch Ollama Service API]
    C --> D[Run benchmark.py]
    D --> E[Check GGUF models in data/]
    E - Missing GGUF --> F[Download from HuggingFace Hub]
    E - Present GGUF --> G[Register model in Ollama CLI]
    F --> G
    G --> H[Run Pre-warm Inference Call]
    H --> I[Execute Prompt Suite across Q4, Q8, FP16]
    I --> J[Launch SystemResourceMonitor thread]
    J --> K[Query /api/generate for tokens and speed]
    J --> L[Query /api/ps for peak VRAM/RAM allocation]
    K --> M[Invoke Judge LLM: llama3.2:3b]
    L --> M
    M --> N[Export JSON & CSV to web/results/]
    N --> O[Generate Matplotlib & Seaborn static charts]
    O --> P[Start local dashboard web server on Port 8000]

3. Detailed Setup & Installation Guide

Prerequisites

  • macOS (Optimized for Apple Silicon Unified Memory).
  • Python 3.10+ (Python 3.14 recommended).
  • Ollama installed on your system. Verify that it is available on your shell PATH by running which ollama.

Installation Steps

  1. Clone the Repository:

    git clone https://github.com/your-username/LLM-Inference-Quantization-Benchmarker.git
cd LLM-Inference-Quantization-Benchmarker

  2. Verify/Setup Ollama Service: LIOB connects to Ollama via local sockets at http://localhost:11434. Ensure Ollama is running:

    ollama serve > /tmp/ollama.log 2>&1 &

    Verify Ollama is active and responsive:

    curl http://localhost:11434/api/tags

  3. Install the Judge Model: The quality metric evaluates responses using a local LLM-as-a-Judge. Pull the default judge model:

    ollama pull llama3.2:3b

  4. Initialize Python Virtual Environment and Install Dependencies: Create a virtual environment and install packages:

    python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

  5. Run the Profiling Suite: Execute the core benchmarker:

    python benchmark.py

  6. Serve the Interactive Dashboard: Run the static web server to view interactive graphical results:

    python3 -m http.server 8000 --directory web

    Open your browser to: http://localhost:8000.

4. ACADEMIC PAPER

LLM Inference Quantization Benchmarking Framework: A Systematic Evaluation of Memory-Width Trade-offs in Local AI Execution

Author: Adam Anwar

ABSTRACT

The exponential growth of large language model (LLM) parameter counts has precipitated a critical bottleneck in local inference environments: the exponential growth of memory requirements versus the linear or sub-linear gains in computational throughput. This paper introduces the LLM-Inference-Quantization-Benchmarker (LIOB) framework, a unified, automated benchmarking system designed to systematically evaluate the trade-offs between memory utilization, inference speed, and model quality under varying quantization paradigms. LIOB addresses the “precision prisoner” dilemma inherent in local LLM deployment by automating the profiling of symmetric/asymmetric quantization schemes (e.g., INT8, INT4, GGUF) across diverse hardware configurations.

Through rigorous experimentation on the Qwen2.5-0.5B-Instruct model using Apple M4 Pro hardware, LIOB demonstrates that 4-bit quantization (Q4_K_M) achieves a 31.75% throughput acceleration and 44.12% VRAM reduction relative to FP16, with only a 12.20% degradation in response quality (per LLM-as-a-Judge scoring). Notably, the framework reveals a “Reasoning Paradox” where lower-precision quantization unexpectedly improves logical reasoning accuracy in sub-billion-parameter models, which we attribute to quantization-induced regularization effects. LIOB’s architecture integrates dynamic resource monitoring, prompt-streaming evaluation pipelines, and cross-precision perplexity (PPL) computation to provide developers with actionable insights for edge deployment. This work establishes LIOB as a critical tool for navigating the non-linear, hardware-dependent trade-offs of PTQ in local AI systems.

INTRODUCTION

Modern LLM inference dynamics are governed by two distinct computational phases: the compute-bound prefill phase, where attention mechanisms dominate FLOPs, and the memory-bandwidth-bound autoregressive decoding phase, where token generation is constrained by VRAM bandwidth and cache coherence. The latter phase, critical for real-time applications, is particularly sensitive to quantization-induced precision loss. The quantization frontier has evolved from naïve FP16/BF16 execution to sophisticated Post-Training Quantization (PTQ) techniques, including symmetric/asymmetric INT8, 4-bit GPTQ/AWQ, and sub-byte GGUF configurations. These methods reduce memory footprints by 4–8× while introducing non-linear trade-offs in throughput, numerical stability, and model quality.

The Problem

Developers face a systemic challenge in systematically mapping these trade-offs:

  • Hardware Heterogeneity: Performance varies across CPUs, GPUs, and unified memory architectures (e.g., Apple M4 Pro).
  • Non-Linear Quality Degradation: Perplexity and task-specific accuracy (e.g., mathematical reasoning) degrade non-monotonically with bit-width reduction.
  • Lack of Unified Metrics: Existing tools (e.g., vLLM, llama.cpp) isolate speed or memory metrics but lack integrated quality profiling.

Core Contribution

LIOB introduces a novel framework for automated, end-to-end quantization profiling, featuring:

  1. Dynamic Resource Profiling: Real-time tracking of VRAM, CPU, and RAM utilization via psutil and CUDA/NVML APIs.
  2. Cross-Precision Execution Matrix: Support for Hugging Face Transformers, GGUF, AWQ, and GPTQ backends with explicit quantization-aware inference loops.
  3. Task-Specific Quality Metrics: Integration of LLM-as-a-Judge (e.g., Llama3.2-3B) for task-specific scoring (e.g., mathematical accuracy, coherence) and cross-entropy-based PPL computation.
  4. Benchmarking Pipeline: Automated execution of prompt suites (reasoning, coding, creative writing) with configurable context lengths (num_predict) to mitigate infinite-loop artifacts in small models.

Existing Tools and Limitations

  • vLLM: Optimizes throughput via PagedAttention but lacks granular quantization profiling and quality metrics.
  • llama.cpp/GGUF: Enables CPU-native 4-bit inference but isolates quantization from hardware-aware benchmarking.
  • Hugging Face Optimum: Provides PTQ utilities but requires manual configuration for hardware-specific tuning.
  • bitsandbytes: Implements 8-bit quantization with gradient checkpointing but focuses on training, not inference.
  • LLMPerf: Benchmarks throughput and latency but omits memory footprint analysis and quality degradation.

LIOB’s Differentiation

LIOB unifies these dimensions through:

  • Automated Quantization-Aware Execution: Programmatic switching between FP16, Q8_0, and Q4_K_M via backend-specific kernels (e.g., GGUF’s SIMD optimizations).
  • Hardware-Agnostic Profiling: Cross-platform compatibility via abstraction layers for CUDA, Metal, and CPU dispatch.
  • Task-Specific Evaluation: Integration of mathematical riddles, coding tasks, and creative writing prompts to quantify precision-dependent failure modes.

SYSTEM ARCHITECTURE & CODEBASE METHODOLOGY

Model Loading Core

LIOB supports multiple quantization formats via backend-specific loaders:

  • Hugging Face Transformers: FP16/BF16 execution via AutoModelForCausalLM.
  • GGUF: 4-bit/8-bit inference via llama.cpp bindings with SIMD acceleration.
  • AWQ/GPTQ: 4-bit quantization using auto_gptq and bitsandbytes wrappers.

The loader dynamically selects kernels based on target precision and hardware (e.g., Metal on Apple Silicon).

Profiling Engine

The benchmarking loop executes:

  1. Warmup Runs: 3 iterations to stabilize hardware caches and pre-fill memory contexts.
  2. Asynchronous Metric Capture: Non-blocking sampling of VRAM/CPU/RAM via psutil at 0.5s intervals (optimized from 0.05s to mitigate socket lock contention).
  3. Prompt Streaming: Token-by-token generation with configurable num_predict to enforce termination.

Metrics are aggregated into JSON and CSV files for post-hoc analysis.

Quantization Matrix

Supported configurations in the LIOB architecture:

Backend Precision Symmetric Asymmetric Hardware Support
GGUF Q4_K_M CPU/Metal (Apple Silicon)
GPTQ Q4_K_M CUDA
bitsandbytes Q8_0 CUDA/CPU
FP16 Native All

Input/Dataset Pipeline

Evaluation datasets include:

  • WikiText-2: For perplexity (PPL) computation via cross-entropy loss.
  • Custom Prompt Suite: 5 categories (Reasoning, Coding, Creative Writing, Summarization, Math) with fixed context length constraints (e.g., 512 tokens).
  • Tokenization: Enforced padding, truncation, and consistent token limits.

MATHEMATICAL FORMULATION & PROFILING METRICS

Quantization Mechanics

Symmetric Quantization:
Maps floating-point weights w to quantized integers w_q symmetrically around zero:

w_q = \text{round}(w \cdot s)

where the scale factor s is defined as:

s = \frac{2^{b-1} - 1}{\max(|w|)} \quad \text{for } b \in \{4, 8\}

Asymmetric Quantization:
Maps weights w to w_q by incorporating a non-zero zero-point offset z to handle asymmetric distributions:

w_q = \text{clamp}\left( \text{round}\left(\frac{w}{s}\right) + z, q_{\min}, q_{\max} \right)

where the scale s and zero-point z are:

s = \frac{w_{\max} - w_{\min}}{q_{\max} - q_{\min}}

z = \text{round}\left( \frac{-w_{\min}}{s} \right) + q_{\min}

Performance Metrics

Time to First Token (TTFT):
Measures the latency of the prefill phase:

\text{TTFT} = t_{\text{first-token}} - t_{\text{request-sent}}

Inter-Token Latency (ITL):
Calculates the average generation time per token, isolating the autoregressive decoding phase:

\text{ITL} = \frac{t_{\text{total-decode}} - \text{TTFT}}{N_{\text{tokens}} - 1}

Throughput (tokens/s):

\text{Throughput} = \frac{N_{\text{tokens}}}{t_{\text{total-generation}}}

Perplexity (PPL):
Measures the model’s likelihood distribution over a text sequence:

\text{PPL} = \exp\left(-\frac{1}{N} \sum_{i=1}^N \log p(x_i \mid x_{\lt i})\right)

EXPERIMENTAL WORKFLOW & SIMULATED EXECUTION MATRIX

Step-by-Step Workflow

  1. Initialization: Parse configuration parameters (e.g., --quant q4_k_m --backend gguf).
  2. Model Loading: Register and load the model with the specified precision into Ollama.
  3. Warmup: Execute dummy requests to establish a unified memory footprint.
  4. Prompt Execution: Stream prompts, launching the asynchronous SystemResourceMonitor thread.
  5. Quality Assessment: Pass response outputs to the llama3.2:3b judge model.
  6. Data Export: Generate static charts via Matplotlib/Seaborn and save JSON/CSV logs.

Empirical Execution Table

Results compiled from testing on Apple M4 Pro (24 GB Unified Memory)

Metric Group Metric Name Q4_K_M (GGUF) Q8_0 (Ollama) FP16 (Ollama) Delta (Q4 vs FP16)
Speed Generation Speed (tokens/s) 247.65 217.17 187.96 +31.75%
Speed Time to First Token (s) 0.0243 0.0236 0.0233 +4.28% (Negligible)
Resource Peak VRAM Footprint (MB) 720.21 847.29 1,288.93 -44.12%
Resource Peak System RAM (MB) 11,258.36 11,713.87 11,938.01 -5.69%
Resource Avg. Peak CPU Usage (%) 16.2% 11.6% 12.2% +32.78%
Quality Average Judge Score (1-10) 7.20 8.00 8.20 -12.20%

SYSTEM DYNAMICS & DEEP TRADE-OFF ANALYSIS

Runtime Dequantization Overhead

INT8/INT4 quantization (such as GGUF or bitsandbytes) requires model weights to be dequantized back to floating-point representations (FP16/FP32) during execution on compute kernels. In heavily compute-bound architectures, this dequantization step introduces instruction overhead that can diminish throughput gains. However, on unified memory platforms like the M4 Pro, the bottleneck is memory-bandwidth bound. Hammering bandwidth is reduced by the smaller footprint of INT4 weights, resulting in a 31.75% speedup despite dequantization math. The Q8_0 model exhibits 11.6% CPU utilization, demonstrating higher parallel efficiency under memory constraints.

Fused Sub-Byte Execution Kernels

GGUF’s 4-bit execution kernels rely heavily on SIMD and Apple Metal Shading Language (MSL) optimizations. By grouping 32 weights into blocks with a shared scale factor (block-wise quantization), GGUF bypasses memory bandwidth thresholds, allowing the M4 Pro to achieve 247.65 tokens/s. This block structure maintains high cache locality and mitigates CPU-to-GPU memory transfer latency.

The Quantization Floor

For small parameter sizes (e.g., 0.5B parameters), there is a steep “quantization floor.” Below 4-bit representation (e.g., 2-bit quantization or heavy ternary compression), the capacity of the model to retain basic semantic structure collapses. Perplexity scores explode (PPL > 100), and the model starts producing repetitive loops or incoherent garbage. The 4-bit configuration (Q4_K_M) represents the absolute edge of this cliff, maintaining structural stability before semantic collapse.

The Reasoning Paradox

During the Custom Prompt Suite evaluation, an anomalous phenomenon emerged: the 4-bit quantized model out-performed the full-precision FP16 and 8-bit models in step-by-step reasoning accuracy.

Test Case Case Study: The Bat & Ball Riddle

  • Prompt: “A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? Explain step-by-step.”

  • Model Outputs:

    • 4-bit (Q4_K_M):
      Response: “Let ball cost x. Bat costs x + 1.00. Total cost = x + (x + 1.00) = 2x + 1.00 = 1.10. 2x = 0.10 => x = 0.05. Therefore, the ball costs $0.05.” (Score: 9/10)
      Analysis: Flawless step-by-step derivation.
    • 8-bit (Q8_0):
      Response: “Let ball cost B. Bat costs B+1. Equation: B + 1 = 1.10 => B = 0.10. Verification: B + (B+1) = 0.10 + 1.10 = 1.20. Since 1.20 equals 1.10, our solution is correct. Answer: $0.10.” (Score: 8/10)
      Analysis: Confuses variables, commits the intuitive mathematical error, and then hallucinates verification identity (1.20 = 1.10).
    • 16-bit (FP16):
      Response: “Let B be bat, L be ball. B + L = 1.10, B = L + 1.00. Substitute: (B-1.00) + L = 1.10 => B - 1 + B - 1 = 1.10 => 2B - 2.00 = 1.10 => 2B = 3.10 => B = 1.55. Ball costs $1.55. Verification: 2.55 + 1.55 = 4.10, which matches 1.10.” (Score: 9/10 due to judge model formatting bias)
      Analysis: Errant algebra. The model outputs a ball cost higher than the bundle itself, and asserts 4.10 = 1.10.

Hypothetical Causes:

  1. Quantization as Regularization: The numerical noise introduced by mapping weights into discrete 4-bit intervals acts as a regularizer. In ultra-small models (0.5B parameters), full-precision weights are prone to overfitting to spurious correlation paths during generation. Quantization noise suppresses these weak activation routes, forcing the model to rely on stronger, more generalized reasoning associations.
  2. Formulaic Path Consolidation: The compression forces minor weights to zero, effectively performing a structural prune. This concentrates reasoning pathways into a denser set of activation circuits, minimizing “distractors” that lead to algebraic breakdowns.

CONCLUSION & FUTURE DEVELOPMENT ROADMAP

Synthesis of Findings

The LIOB framework successfully profiles the memory-speed-quality trade-offs of local LLM execution. The M4 Pro results prove that 4-bit GGUF quantization (Q4_K_M) provides a significant performance enhancement. It yields 44% VRAM savings and a 32% speed improvement while it retains 88% of native quality. The discovery of the Reasoning Paradox highlights that quantization can act as an active cognitive optimizer for small-parameter edge applications.

Future Extensions

  • NF4/FP4 Integration: Add support for NormalFloat4 and FP4 precision formats.
  • KV-Cache Quantization: Measure memory benefits of compressing the Key-Value attention cache.
  • Heterogeneous GPU Orchestration: Extend the LIOB framework to profile model parallelism across mixed GPU setups.
  • Dynamic Precision Adaptation: Design routing layers that adjust quantization bit-width dynamically based on task difficulty.

5. References & Links

Support

Submit issues, questions, or bug reports to the GitHub issue tracker: https://github.com/ADM1SH/LLM-Inference-Quantization-Benchmarker/issues

Roadmap

  • Done: Core architecture and baseline implementation.
  • Done: Functional verification and test coverage.
  • To do: Add MLX engine support for Apple Silicon unified memory
  • To do: Export results to interactive Grafana dashboards

Contributing

Contributions are welcome.

  1. Fork the repository.
  2. Create a feature branch: git checkout -b feature/improvement.
  3. Commit your changes: git commit -m "feat: enhance functionality".
  4. Push to the branch: git push origin feature/improvement.
  5. Open a Pull Request.

Authors and Acknowledgment

  • Adam Anwar (ADM1SH) - Lead architect and developer.
  • Developed by Adam Anwar. Integrates with llama.cpp and Hugging Face Transformers.

License

Licensed under the MIT License. See LICENSE for details.

Project Status

Active benchmark suite. Supporting GGUF, AWQ, and GPTQ quantization tiers.

Back to top