Local LLM Series, Part 2: Deployment Record, Qwen3-14B on AMD RX 9070 XT (Windows 11)

Written October 22, 2026 · updated October 24, 2026

Series: Overview · Part 1, Setup guide · Part 2, Deployment record (this page) · Part 3, Keeping reference data current · Part 4, Testing on a phishing email. Each part can be read on its own.

Built with AI, on purpose. I used Claude (Sonnet 5.5, from Anthropic) to help plan this build, troubleshoot it, write the scripts and draft this record. I am not presenting the scripts as my own coding. What I am showing is the project itself: choosing the goal, running it on my own hardware, testing the results, and using AI tools to move faster on work I am still learning. See "How this was made" near the end of this page.

Summary

Qwen3-14B, a 14-billion-parameter language model compressed to 4-bit, now runs locally on an AMD Radeon RX 9070 XT (16 GB) under Windows 11. It answers through a local web page and an API on the same PC, so SOC data never leaves the machine. The build took about two hours on October 22, 2026.

The final setup uses AMD's validated llama.cpp build, not PyTorch. A 14B model needs about 28 GB at full precision, which does not fit a 16 GB card, but about 9 GB at 4-bit does.

Six roadblocks came up, and the last three were symptoms and causes of one missing Windows DLL. The first model tried, a 1.5B DeepSeek distill, ran correctly but gave a wrong answer about Kerberoasting. That result drove the move to a larger model. The 14B model also gave a wrong ATT&CK ID, so a script that answers only from supplied reference documents was added; with the ATT&CK page as its reference, the model returned the correct ID.

Goal and starting environment

The goal was a private, local language model from Hugging Face to support SOC research and analysis, such as alert summaries and log review. It had to run on one Windows 11 PC with an AMD GPU.

Item Starting state
Operating system Windows 11, PowerShell
GPU AMD Radeon RX 9070 XT, 16 GB VRAM
Python Latest release installed system-wide and on PATH
Project folder <project folder>
LLM tooling None installed

The GPU was first entered as "9700xt". Its 16 GB of memory matches the RX 9070 XT; the Radeon AI PRO R9700 is a different, 32 GB card. This record assumes the RX 9070 XT.

How the build unfolded

The work ran in five stages. Each stage ended at a roadblock or a decision, and the table in the next section lists every problem in one place.

1. Choosing a route. The first question was whether an AMD GPU could run Hugging Face models on Windows at all. AMD supports PyTorch on Windows through ROCm 7.2.1, and its support matrix lists the RX 9070 XT. The catch was that AMD's packages require Python 3.12, not the newer release already installed.

2. Building the Python environment. Python 3.12 was installed beside the newer version, and a virtual environment was created in <project folder> with py -3.12. AMD's ROCm runtime and its PyTorch 2.9.1 packages went in next, followed by transformers, huggingface-hub and accelerate.

3. Testing a first model. The test script loaded DeepSeek-R1-Distill-Qwen-1.5B, a small model AMD has tested on Windows. The first run failed with a syntax error caused by a pasted Python banner on line 1 of the file. Once fixed, the script generated text, which shows PyTorch was reaching the GPU. The answer to a Kerberoasting question was wrong, so the model was too small to trust for security work.

4. Moving to a 14B model. A 14B model needs about 28 GB at full precision, so 16-bit loading in PyTorch was ruled out. The plan switched to AMD's validated llama.cpp build (release b8407) and Qwen3-14B at 4-bit (Q4_K_M, about 9 GB) from Qwen's official Hugging Face repository. Community re-uploads, including "abliterated" variants with safety training removed, were skipped on purpose.

5. Getting the server to start. The browser first showed a connection-refused error because the server was not running. Starting it returned silently to the prompt, and an exit code of -1073741515 showed Windows could not find a required DLL. A short Python script that scanned the program's imports named the missing file, libomp140.x86_64.dll. Placing a copy of that OpenMP runtime next to the server fixed it, and the web interface loaded at 127.0.0.1:8080.

Roadblocks and fixes

Six problems came up. Rows 4 to 6 are one chain: the browser error was a symptom, the silent exit explained it, and the missing DLL caused it.

# Stage Symptom Cause Fix
1 Environment The newest Python was incompatible with AMD's PyTorch packages (caught before installing) AMD's Windows packages are built for Python 3.12 only Installed Python 3.12 beside the newer version; built the virtual environment with py -3.12
2 First test SyntaxError: invalid decimal literal on line 1 of test_llm.py The Python startup banner (Python 3.12.10 ...) had been saved into the file Replaced the file contents with the script alone, so line 1 became import torch
3 First test Wrong answer to "Explain Kerberoasting" The 1.5B model is too small to hold niche security knowledge Moved to Qwen3-14B at 4-bit; checked answers against facts already known
4 Server start ERR_CONNECTION_REFUSED at 127.0.0.1:8080 The server was not running; port 8080 had no listener Traced why it had stopped (rows 5 and 6)
5 Server start The start command returned to the prompt with no output Windows ended the program at launch; exit code -1073741515 means a required DLL was not found Checked the exit code on its own line (on the same line as the command it is read as an argument); confirmed Windows Defender had flagged nothing and the package's own DLLs were present
6 Server start Exit code -1073741515 persisted ggml-cpu.dll needs libomp140.x86_64.dll, the LLVM OpenMP runtime, which was not on the system A short script listed every unresolved import; a copy of the native libomp140.x86_64.dll OpenMP runtime was placed beside llama-server.exe

One early guess was wrong. The Microsoft Visual C++ redistributable was suspected, but it does not normally include this LLVM runtime. DLL download websites were avoided on purpose, since they are a common source of malware.

Final working configuration

The server loads the model fully onto the GPU and serves a chat page and an OpenAI-compatible API at http://127.0.0.1:8080, reachable only from this PC.

Component Detail
Operating system Windows 11, PowerShell
GPU AMD Radeon RX 9070 XT, 16 GB
Python environment Python 3.12.10 in <project folder>\.venv
AMD software ROCm 7.2.1; PyTorch 2.9.1 (used for the first model test only)
Server software llama.cpp release b8407, AMD-validated ROCm 7.2.1 build for gfx110X, gfx115X and gfx120X GPUs
Server folder <project folder>\llama_cpp_binaries\llama-b8407-windows-rocm-7.2.1-gfx110x-gfx115x-gfx120x-x64
Model Qwen3-14B-Q4_K_M.gguf, about 9 GB, from Qwen's official repository (Apache 2.0 license)
Model folder <project folder>\models

Download the model (from the activated .venv window):

cd <project folder>
hf download Qwen/Qwen3-14B-GGUF Qwen3-14B-Q4_K_M.gguf --local-dir <project folder>\models

Start the server (from the server folder; leave the window open):

cd <project folder>\llama_cpp_binaries\llama-b8407-windows-rocm-7.2.1-gfx110x-gfx115x-gfx120x-x64
.\llama-server.exe -m <project folder>\models\Qwen3-14B-Q4_K_M.gguf -c 8192 -ngl 99 -fa on --jinja --host 127.0.0.1 --port 8080
  • -ngl 99 places every model layer on the GPU.
  • -c 8192 sets an 8,192-token context window, using roughly 1.3 GB of VRAM beyond the weights (an estimate).
  • -fa on turns on flash attention to save memory.
  • --jinja applies the model's built-in chat template.
  • --host 127.0.0.1 keeps the server off the network. Do not change it to 0.0.0.0 on a work network.

The missing-DLL fix is a copy of the OpenMP runtime file placed in the server folder.

DLL fix applied: the file copied into the server folder was the native libomp140.x86_64.dll, not a renamed libiomp5md.dll. Its original source path was not recorded.

Model verification results

Qwen3-14B's Kerberoasting answer was not reliable. It opened plausibly but contained several technical errors, and it gave the MITRE ATT&CK ID as T1558.002, which is Silver Ticket. The correct ID is T1558.003. Measured speed on this run: 53.29 tokens per second (2,084 tokens in 39 seconds).

Check Result
Explain Kerberoasting Partly correct, with several errors (listed below)
Give Kerberoasting's MITRE ATT&CK ID Wrong: answered T1558.002 (Silver Ticket); correct is T1558.003
Say what Windows Event ID 4769 records Not recorded
Model file hash vs. the value on Hugging Face Verified

What it got right: Event ID 4769 as the key log, offline cracking of service tickets, the parent technique T1558 and the Credential Access tactic. Errors found in the full response, checked against MITRE ATT&CK:

  • It called Kerberoasting "passive" with no network signs. The attacker actively requests tickets, which logs Event ID 4769.
  • It said the attacker needs Mimikatz to extract a TGT. Any authenticated domain user already holds one, and MITRE describes requesting tickets for any SPN with a valid TGT.
  • It never named the Service Principal Name (SPN), the core of the technique.
  • It said repeated requests mean an attacker is "testing multiple passwords". Cracking happens offline, and TGT requests are routine.
  • It suggested watching for lsass.exe process creation. LSASS is a normal Windows process.
  • It offered "pre-authentication (AES256)" as a fix, mixing up pre-authentication (an AS-REP roasting issue) with ticket encryption. It left out the standard fixes: long random service-account passwords, managed service accounts, and disabling RC4.
  • Its related-technique list was wrong: T1558.001 is Golden Ticket, and T1076 is a retired ID for Remote Desktop Protocol, not "Compromised Credentials".

Technique IDs, event IDs and other identifiers should be checked against the source. Until reliability improves, use the model to summarize and draft from evidence you supply, and treat anything it states from memory as unverified.

Reference-grounded questions

The wrong ATT&CK ID shows the model cannot be trusted on niche security facts from memory. Running locally keeps case data private but does not make answers more accurate, so the fix is to make the model answer only from reference text you supply. ask_grounded.py in <project folder> does this.

  • It sends the model only the excerpts from your reference files that match the question, labeled [S1], [S2] and so on.
  • The system prompt requires a citation after every claim, and the exact reply NOT IN REFERENCE when the excerpts lack the answer.
  • If no excerpt matches the question, the script skips the model and prints NOT IN REFERENCE.
  • It uses Qwen's sampling values and /no_think by default; --think enables reasoning.
  • After the answer it warns about missing or invalid citations, and about any ATT&CK ID, CVE, IP address or hash that is absent from the reference text.

Example, run from the .venv window with the server up: python ask_grounded.py -r refs\kerberoasting.txt -q "What is the MITRE ATT&CK ID for Kerberoasting?"

Limit: the check only proves an identifier appears in the reference, not that it belongs to the right technique. In testing against a mock server, a wrong answer of T1558.002 passed because that ID sits in the reference under Silver Ticket. The script therefore prints the surrounding reference text for each identifier; read it before relying on the answer. Keep one topic per reference file. Live test on Qwen3-14B with a reference file built from the ATT&CK Kerberoasting page: asked for Kerberoasting's ATT&CK ID, the model answered T1558.003 with a citation, and the grounding check passed. That is the same question it got wrong without a reference. Out-of-scope test: asked about the default password policy for Azure AD service accounts, which the reference does not cover, the model declined to answer from memory and returned NOT IN REFERENCE.

Later changes (October 24, 2026): this section records the first version of ask_grounded.py, as tested on October 22. The script has since changed: it reads a large reference library, caps how many excerpts one source can contribute, checks that URLs in the answer match the evidence, and has a browser page that runs code-side email checks and a runbook checklist. Part 1 has the current scripts, Part 3 covers the reference library and runbooks, and Part 4 covers the phishing tests.

Lessons learned

These practices would have prevented or shortened most of the problems above.

  • Check version requirements first. Match the Python version to the hardware vendor's packages before installing anything.
  • Keep script files clean. Save code with a text editor, and never paste interpreter output, such as a startup banner or >>> prompts, into a .py file.
  • Test every model on known answers. A small model can answer confidently and be entirely wrong. Ask questions whose answers you already know before trusting it with alerts.
  • Do the memory math before downloading. At 16-bit, a model needs about 2 GB of VRAM per billion parameters; at 4-bit, about 0.6 GB (both estimates). Leave room for context.
  • Read exit codes. A command that returns silently has usually crashed. Run $LASTEXITCODE on its own line, and treat -1073741515 as "a DLL is missing".
  • Find the exact missing file. Scanning a program's imports named the DLL in one step, where guessing at redistributable packages did not.
  • Source files carefully. Do not take DLLs from download sites. Prefer a copy already on the machine, and get models from the publisher's official repository, checking the file hash against the value shown on its page.
  • Keep the server local. Bind it to 127.0.0.1 so SOC data stays on this PC.
  • Recheck vendor docs before a rebuild. AMD's Radeon documentation used here covered ROCm through 7.2.1 and has since moved to unified documentation.

Open items and next steps

Three items are done and two follow-ups remain before this setup is ready for real casework.

  • Record which OpenMP DLL was copied into the server folder: the native libomp140.x86_64.dll (original path not recorded).
  • Compare the model file's hash with the value on its Hugging Face page.
  • Finish the known-answer tests on Qwen3-14B without a reference file: check what Windows Event ID 4769 records. With a reference file, the Kerberoasting ATT&CK ID (T1558.003) already came back correctly.
  • Add Qwen's recommended sampling flags to the start command (--temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 --presence-penalty 1.5) and confirm the server still starts. Those values are for thinking mode; Qwen advises temperature 0.7 and top-p 0.8 with /no_think. Optional: the helper scripts send these values with every request, so the flags only affect the browser chat page.
  • Point the Python helper scripts at the server's API using Qwen's sampling values (its model card advises against greedy decoding, so avoid a temperature near 0), and add /no_think to prompts that need fast answers. Done and tested live in ask_grounded.py (see Reference-grounded questions). ask_server.py and soc_llm.py were also rewritten to call the server with the same settings (soc_llm.py now imports ask from ask_server.py and no longer loads its own 1.5B model). They were tested against a mock server only; a live run on real input is still to do.

How this was made

I want to be direct about who did what.

  • Written with Claude (Sonnet 5.5): the troubleshooting guidance, the helper scripts described here, and the first draft of this record. I ran the commands on my own PC and checked the results, but I did not write that code from scratch, and I don't claim to.
  • Mine: the goal (a private, local model for SOC work), the hardware and environment, running everything on my own PC, working through the failures, testing the model against answers I already knew, and deciding what to trust.
  • Why I'm saying so: in a SOC, being honest about what a tool did and what you verified is part of the job. The same applies to AI-assisted work. The Kerberoasting test in Part 2 is the clearest example: the model sounded confident and was wrong, so I checked it.

Using AI tools to learn faster, build real projects and then verify the output is a method I plan to keep using and keep being open about.

Sources

Pages checked on October 22, 2026. AMD's Radeon documentation was current through ROCm 7.2.1 at that time.

← Back to Cybersecurity