← Back to Cybersecurity

Running a Private Local LLM for SOC Analysis

Qwen3-14B on an AMD RX 9070 XT, Windows 11 · October 2026

At a glance

  • Goal: a private model on my own PC, so case data never goes to a third party
  • Setup: Qwen3-14B (4-bit) on a 16 GB AMD Radeon RX 9070 XT, about 53 tokens per second, reachable only from this machine
  • Key finding: it only became trustworthy once it answered strictly from reference documents I supplied
  • Build: about two hours, six roadblocks, all documented below

Built with AI, on purpose. I used Claude (Sonnet 5.5, from Anthropic) to help plan this build, troubleshoot it, write the scripts and draft this write-up. I am not presenting the scripts as my own coding. What I am showing is the project itself: choosing the goal, running it on my own hardware, testing the results, and using AI tools to move faster on work I am still learning. See "How this was made" near the end of this page.

The goal: a private language model that runs entirely on my own PC, so alert summaries and log review never send case data to a third party.

The result: Qwen3-14B (4-bit) now runs fully on a 16 GB AMD Radeon RX 9070 XT. It answers through a local web page and an OpenAI-compatible API reachable only from this machine, at about 53 tokens per second. The build took about two hours and hit six roadblocks, which are documented below.

The most useful finding was about trust, not speed. The model got a basic security question wrong until I made it answer only from reference documents I supplied.

The setup

Component Detail
OS Windows 11, PowerShell
GPU AMD Radeon RX 9070 XT, 16 GB VRAM
Server llama.cpp release b8407, AMD-validated ROCm 7.2.1 build
Model Qwen3-14B, Q4_K_M quantization (about 9 GB), from Qwen's official Hugging Face repository (Apache 2.0)
Python 3.12 in a virtual environment (used for the first model test)
Network Bound to 127.0.0.1 only

Why 4-bit: a 14B model needs about 28 GB at full precision, which does not fit a 16 GB card. At 4-bit it needs about 9 GB, which leaves room for context.

Why llama.cpp and not PyTorch: AMD's PyTorch packages for Windows require Python 3.12, and a 16-bit 14B model was out of reach anyway. AMD's validated llama.cpp build ran the 4-bit model directly.

I skipped community re-uploads, including "abliterated" variants with safety training removed, and used the publisher's official repository. I checked the model file's hash against the value on its Hugging Face page.

How the build went

  1. Choosing a route. AMD supports PyTorch on Windows through ROCm 7.2.1 and lists the RX 9070 XT. The catch: its packages need Python 3.12, not the newer release I had installed.
  2. Building the environment. I installed Python 3.12 beside the newer version and created a virtual environment with it.
  3. Testing a first model. I ran a 1.5B DeepSeek distill. It generated text, which proved PyTorch was reaching the GPU. Its answer to "Explain Kerberoasting" was wrong, so it was too small to trust for security work.
  4. Moving to 14B. I switched to llama.cpp and Qwen3-14B at 4-bit.
  5. Getting the server to start. This took the most troubleshooting; see below.

Roadblocks and fixes

# Symptom Cause Fix
1 Newest Python incompatible with AMD's packages AMD's Windows packages are built for 3.12 only Installed 3.12 side by side and built the virtual environment with it
2 SyntaxError on line 1 of my test script I had pasted the Python startup banner into the file Saved the script alone, so line 1 became import torch
3 Wrong answer to a known question The 1.5B model is too small to hold niche security knowledge Moved to Qwen3-14B and checked answers against facts I already knew
4 ERR_CONNECTION_REFUSED at 127.0.0.1:8080 The server was not running Traced why it had stopped (rows 5 and 6)
5 Start command returned to the prompt with no output Windows ended the program at launch; exit code -1073741515 means a required DLL was not found Read the exit code on its own line; confirmed Windows Defender had flagged nothing
6 Exit code persisted ggml-cpu.dll needs libomp140.x86_64.dll, the LLVM OpenMP runtime, which was missing Scanned the program's imports with a short script to name the file, then placed a copy of the native runtime beside llama-server.exe

Rows 4 to 6 were one chain: the browser error was a symptom, the silent exit explained it, and the missing DLL caused it. I first suspected the Visual C++ redistributable, which was wrong. I avoided DLL download sites on purpose, since they are a common source of malware.

Starting the server

llama-server.exe -m Qwen3-14B-Q4_K_M.gguf -c 8192 -ngl 99 -fa on --jinja --host 127.0.0.1 --port 8080
  • -ngl 99 places every layer on the GPU.
  • -c 8192 sets an 8,192-token context window.
  • -fa on turns on flash attention to save memory.
  • --jinja applies the model's built-in chat template.
  • --host 127.0.0.1 keeps the server off the network. Never change it to 0.0.0.0 on a work network.

Can I trust it? Testing on a known answer

I asked about Kerberoasting, a technique I know well. The model's answer was only partly correct:

  • Right: Event ID 4769 as the key log, offline cracking of service tickets, the parent technique T1558, and the Credential Access tactic.
  • Wrong ATT&CK ID: it answered T1558.002, which is Silver Ticket. The correct ID is T1558.003.
  • Wrong on mechanics: it called the technique "passive" with no network signs (the attacker actively requests tickets, which logs Event ID 4769). It said Mimikatz is needed to extract a TGT (any authenticated domain user already has one). It never named the Service Principal Name, the core of the technique. And it suggested the wrong fixes, mixing up pre-authentication with ticket encryption and leaving out long random service-account passwords, managed service accounts and disabling RC4.

Running locally keeps case data private. It does not make answers more accurate.

The fix: answer only from reference text

I wrote a script that sends the model only the excerpts of my reference files that match the question, labeled [S1], [S2] and so on.

  • The system prompt requires a citation after every claim, and the exact reply NOT IN REFERENCE when the excerpts lack the answer.
  • If no excerpt matches, the script skips the model entirely.
  • After the answer, it warns about missing or invalid citations, and about any ATT&CK ID, CVE, IP address or hash that is absent from the reference text.

Result: with a reference file built from the MITRE ATT&CK Kerberoasting page, the same question returned T1558.003 with a citation. A question outside the reference's scope (default password policy for Azure AD service accounts) returned NOT IN REFERENCE instead of a guess from memory.

Limit: the check proves an identifier appears in the reference, not that it belongs to the right technique. In testing against a mock server, a wrong answer of T1558.002 passed because that ID sits in the reference under Silver Ticket. So the script prints the surrounding reference text for each identifier, and I read it before relying on the answer.

Lessons learned

  • Check version requirements first. Match the Python version to the hardware vendor's packages before installing anything.
  • Keep script files clean. Never paste interpreter output into a .py file.
  • Test every model on known answers before trusting it with alerts.
  • Do the memory math before downloading. Roughly 2 GB of VRAM per billion parameters at 16-bit, about 0.6 GB at 4-bit (estimates), plus room for context.
  • Read exit codes. A command that returns silently has usually crashed. Treat -1073741515 as "a DLL is missing."
  • Find the exact missing file. Scanning imports named the DLL in one step, where guessing did not.
  • Source files carefully. No DLL download sites. Models from the publisher's official repository, with the hash checked.
  • Keep the server local so SOC data stays on the machine.

How this was made

I want to be direct about who did what.

  • Written with Claude (Sonnet 5.5): the helper scripts (the reference-only question script, the server client and the test scripts), the troubleshooting guidance, and the first draft of this write-up. I reviewed the results and the text, but I did not write that code from scratch, and I don't claim to.
  • Mine: the goal (a private, local model for SOC work), the hardware and environment, running every command on my own PC, working through the failures, testing the model against answers I already knew, and deciding what to trust. The safeguards in this project, such as using the publisher's official model, verifying the file hash, avoiding DLL download sites and keeping the server local, were part of how I approached it.
  • Why I'm saying so: in a SOC, being honest about what a tool did and what you verified is part of the job. The same applies to AI-assisted work. The Kerberoasting test above is the clearest example: the model sounded confident and was wrong, so I checked it.

Using AI tools to learn faster, build real projects and then verify the output is a method I plan to keep using and keep being open about.

Still open

  • Finish the known-answer tests without a reference file, starting with what Windows Event ID 4769 records.
  • Add Qwen's recommended sampling flags to the server start command.
  • Run the helper scripts on real input; so far they have been tested only against a mock server.

Sources: Qwen3-14B-GGUF model card (Hugging Face); AMD Radeon documentation for PyTorch on Windows, Windows support matrices, and llama.cpp pre-built binaries (checked October 22, 2026); MITRE ATT&CK, Kerberoasting (T1558.003).

← Back to Cybersecurity