Handle confidential talk with your own Qwen, safely.

Confidential data you can't paste into an outside AI. Sound familiar?

You know AI would be faster. But these conversations can't leave the company.

It can't go out, so you don't use AI. Between those two options lies "run it on your own servers."

* 1 and 2 are examples based on cases described in explainer videos (What the videos say).

What you really wanted: confidential data, with Qwen in-house

Same documents, same meetings. AI does the work, and the data stays inside the company.

You can build all of this with the steps and checklist on this page. How to split the work, including the follow-up after meetings, is covered in Where SIMY fits.

What is a local LLM? Why run Qwen in-house

A local LLM means running the published weights of a model on your own PC or your company's servers. The text you enter never leaves that machine.

What a local LLM can do

• Summarize and draft confidential documents: meeting minutes, contracts, design documents, customer correspondence.
• Search and ask questions over internal documents (RAG): feed it internal policies and manuals and have it answer questions.
• Translation and code help: translation between Chinese, English and other languages, explaining internal code and suggesting fixes.

Small models are not as smart as large cloud models. Use them to produce first drafts rather than final versions, and expectations will match reality more closely.

Why choose Qwen

• Apache 2.0 models
• Many languages
• A choice of sizes

• License: many models, such as Qwen3 and Qwen3.8-27B, are released under Apache 2.0. But not all of them (see the table below).
• Languages: with the Qwen3 generation, Qwen claims support for 119 languages and dialects.
• Size: from 0.6B to 8B that run on a PC, to 27B to 32B that fit on one GPU when quantized, up to 235B that needs multiple GPUs.

Licenses differ by model

Scroll the table sideways →

Checked against the model information in the Qwen organization on Hugging Face on October 1, 2026. Not every Qwen model is Apache 2.0. Before commercial use, check the LICENSE in the repository of the model you use with your legal team.

Choosing an inference engine: Ollama, llama.cpp, vLLM, SGLang, LM Studio

The software that runs a model is the "inference engine." For security, the biggest difference is where it listens if you change nothing.

Scroll the table sideways →

Checked against each project's official documentation (and the vLLM source) as of October 2026. In the LM Studio row, items we could not fully confirm are marked "Check the official docs."

• Check first: use llama.cpp or Ollama to see whether it helps your work.
• Company-wide use: vLLM. Speed holds up even with many concurrent users.
• Agents sending the same long instructions in volume: SGLang.

A vLLM caveat: the official Security page explains that --api-key only protects endpoints such as /v1, and not /invocations, /pooling and others. That is why the project itself names a reverse proxy that passes only the endpoints you want to expose as the most effective measure.

Hardware guidelines: GPU memory needed for each Qwen model

Think of the GPU memory you need as the sum of three parts.

• WeightsParameters × bytes per parameter (BF16 is 2, FP8 is 1, 4-bit is about 0.5)
• + KV cacheProportional to context length × concurrent users
• + HeadroomWorking space at run time

Scroll the table sideways →

The Qwen3 figures come from Qwen's official speed benchmark (GPU memory right after startup, processing a short input with transformers). Qwen3.8-27B has no official table, so its figures are estimates calculated from the parameter count plus an individual's measurements, and they exclude the KV cache. The amount needed varies widely with the quantization and context length (as of October 2026).

• Qwen3.8-27B and a 32 GB GPU: BF16 needs about 54 GB for the weights alone (a calculated estimate), so it doesn't fit on one RTX 5090 (32 GB). The official FP8 version is also about 28.8 GB for the weights alone, and one estimate shows it won't fit in 32 GB at the maximum context. Adjust by using a 4-bit version or lowering the context limit.
• Two 16 GB GPUs are not the same as one 32 GB GPU: splitting across two cards costs you the inter-card traffic and the working space each card needs.
• Long contexts mean longer waits: in one individual's measurement, an input of about 250,000 tokens took over 4 minutes before the first character appeared.
• Mac: Apple silicon shares memory with the GPU. You need memory well above the size of the model file.

How to size it: decide first how many people will use it at once and how long the documents they feed it will be. The KV cache grows in proportion to both, so if you assume the weights alone are enough, you'll run short in production.

Setup in a closed network: Qwen + vLLM on an internal server

The example uses the FP8 version of Qwen3.8-27B with vLLM (Docker) on a Linux server. Commands match the official documentation as of October 2026. If GPU memory runs short, lower the context limit (--max-model-len) or choose a 4-bit version.

* This setup is an illustration

Reading the commands: replace anything in angle brackets, such as <verified commit SHA>, with your own values. Version numbers are examples as of October 2026.

1. Fetch the model at a pinned version

   Transfer machine (internet side)

   Download from the official Qwen/ organization's repository, specifying a commit. Save the inference engine's container at a pinned version too.

   bash

   pip install -U huggingface_hub
   hf download Qwen/Qwen3.8-27B-FP8 \
     --revision <verified commit SHA> \
     --local-dir ./Qwen3.8-27B-FP8
   
   docker pull vllm/vllm-openai:v0.30.0
   docker inspect --format '{{index .RepoDigests 0}}' vllm/vllm-openai:v0.30.0
   docker save vllm/vllm-openai:v0.30.0 -o vllm-openai-v0.30.0.tar

   Copy the digest that docker inspect prints (the value starting with sha256:) into your ledger.

   Why pin the version: even in the same repository, contents can change with a new revision. In one quantized version, a new revision reportedly removed the layers used for speculative decoding (MTP), so the speed-up stopped working.

2. Build a SHA-256 ledger

   Transfer machine

   The lfs.oid returned by the Hugging Face API is the SHA-256 of a large (LFS) file. Check your local files against these values.

   bash

   # Extract the values from Hugging Face (LFS files only)
   curl -s "https://huggingface.co/api/models/Qwen/Qwen3.8-27B-FP8/tree/<verified commit SHA>?recursive=true&expand=true" \
     | jq -r '.[] | select(.lfs) | "\(.lfs.oid)  ./\(.path)"' > HF.sha256
   
   # Check the local files and build a ledger of every file
   cd Qwen3.8-27B-FP8
   sha256sum -c ../HF.sha256
   find . -type f ! -path './.cache/*' -print0 | sort -z | xargs -0 sha256sum > ../MODEL.sha256
   cd .. && sha256sum vllm-openai-v0.30.0.tar > IMAGE.sha256

   Have someone other than the person who downloaded the files sign off on the ledger (follow your internal standard for signing).

3. Bring it in on media and verify before loading

   Closed-network server

   Move the ledger and files on media, and check them again on the closed side. If even one line says FAILED, don't use it.

   bash

   cd /srv/llm/models/Qwen3.8-27B-FP8 && sha256sum -c /srv/llm/incoming/MODEL.sha256
   cd /srv/llm/incoming && sha256sum -c IMAGE.sha256
   docker load -i vllm-openai-v0.30.0.tar

4. Start it listening on 127.0.0.1 only

   Closed-network server

   vLLM listens on all interfaces if you omit --host. Always write --host 127.0.0.1, and pass the key in a file (on the command line it is visible in ps).

   bash · vLLM (Docker)

   # Create the key file with permissions 600
   sudo install -D -m 600 /dev/null /etc/llm/vllm.env
   echo "VLLM_API_KEY=$(openssl rand -hex 32)" | sudo tee /etc/llm/vllm.env > /dev/null
   
   sudo docker run -d --name qwen --gpus all --ipc=host --network host \
     -v /srv/llm/models:/models:ro \
     -e HF_HUB_OFFLINE=1 -e VLLM_NO_USAGE_STATS=1 -e DO_NOT_TRACK=1 \
     --env-file /etc/llm/vllm.env \
     vllm/vllm-openai:v0.30.0 \
     --model /models/Qwen3.8-27B-FP8 \
     --served-model-name qwen3.8-27b \
     --host 127.0.0.1 --port 8000 \
     --max-model-len 65536 --gpu-memory-utilization 0.90
   
   # Confirm it listens on 127.0.0.1:8000 only
   ss -ltnp | grep 8000

   • Don't add --trust-remote-code. This keeps code bundled with the model from running.
   • Don't use VLLM_SERVER_DEV_MODE=1 in production. It exposes development endpoints.
   • Thinking mode and tool calls: Qwen's official documentation gives --reasoning-parser qwen3 and --enable-auto-tool-choice --tool-call-parser hermes as examples for Qwen3. Check the model card for the settings for the 3.8 generation.
   bash · With Ollama (systemd)

   sudo systemctl edit ollama
   # Add these 3 lines to the file that opens, then save
   [Service]
   Environment="OLLAMA_HOST=127.0.0.1:11434"
   Environment="OLLAMA_NO_CLOUD=1"
   
   sudo systemctl restart ollama

   bash · With SGLang or llama.cpp

   python3 -m sglang.launch_server --model-path /models/Qwen3.8-27B-FP8 \
     --host 127.0.0.1 --port 30000 --api-key <key> --admin-api-key <admin key>
   
   llama-server -m /models/qwen.gguf --host 127.0.0.1 --port 8080 \
     --api-key-file /etc/llm/keys --offline --no-webui --jinja

   In the SGLang example too, don't write the keys directly; in practice pass them from a file or environment variable. The model card examples use --host 0.0.0.0, so don't copy them as they are.

5. Add TLS, an allowlist and authentication with nginx

   Closed-network server (gateway)

   Users touch only nginx. Pass only the parts of /v1 you need, and return 404 for everything else.

   nginx

   server {
     listen 443 ssl;
     server_name llm.internal.example;
     ssl_certificate     /etc/pki/llm.crt;
     ssl_certificate_key /etc/pki/llm.key;
     ssl_protocols TLSv1.2 TLSv1.3;
   
     allow 10.20.0.0/16;   # internal segment only
     deny  all;
   
     location = /v1/chat/completions {
       auth_request /_auth;
       auth_request_set $user $upstream_http_x_auth_request_user;
       proxy_set_header Authorization "Bearer <upstream key>";
       proxy_pass http://127.0.0.1:8000;
       proxy_buffering off;   # for streaming responses
     }
     location = /v1/models {
       auth_request /_auth;
       auth_request_set $user $upstream_http_x_auth_request_user;
       proxy_set_header Authorization "Bearer <upstream key>";
       proxy_pass http://127.0.0.1:8000;
     }
     location = /_auth {
       internal;
       proxy_pass http://127.0.0.1:4180/oauth2/auth;   # connect to your internal IdP via oauth2-proxy or similar
     }
     location / { return 404; }   # don't expose /metrics, /invocations and the like
   }

   Users sign in with your internal SSO (OIDC or SAML), and the upstream API key is never handed out to users. auth_request is an nginx module; check whether your build includes it with nginx -V.

6. Block outbound traffic

   Closed-network server

   Block all traffic from the server to the internet by default. Allow only internal DNS, time sync and your log collector.

   bash · ufw

   sudo ufw default deny incoming
   sudo ufw default deny outgoing
   sudo ufw allow in on <admin IF> to any port 22 proto tcp    # allow this first or SSH drops
   sudo ufw allow in on <internal IF> to any port 443 proto tcp
   sudo ufw allow out to <internal DNS> port 53 proto udp
   sudo ufw allow out to <internal NTP> port 123 proto udp
   sudo ufw allow out to <log collector> port 514 proto tcp
   sudo ufw enable
   
   # Confirm it can't reach the outside (failure means it's correct)
   curl -m 5 https://huggingface.co

   If you publish a port with Docker's -p, Docker adds its own packet rules, so traffic from outside can get through even when you think ufw has closed it. These steps use --network host and --host 127.0.0.1 to keep listening local.

   bash · When using -p

   # Publish on 127.0.0.1 only on the host (-p 8000:8000 publishes on all interfaces)
   sudo docker run -d --name qwen --gpus all --ipc=host \
     -p 127.0.0.1:8000:8000 \
     -v /srv/llm/models:/models:ro \
     -e HF_HUB_OFFLINE=1 -e VLLM_NO_USAGE_STATS=1 -e DO_NOT_TRACK=1 \
     --env-file /etc/llm/vllm.env \
     vllm/vllm-openai:v0.30.0 \
     --model /models/Qwen3.8-27B-FP8 --served-model-name qwen3.8-27b \
     --host 0.0.0.0 --port 8000 --max-model-len 65536
   # ↑ listens on 0.0.0.0 inside the container, exposed on 127.0.0.1 only on the host

   In this form, vLLM inside the container listens on 0.0.0.0, but it can't be reached from outside the host. Block outbound traffic at the network firewall as well.

7. Keep logs and audit trails

   Gateway and monitoring

   Record who used which endpoint and when in the nginx logs, and send them to your internal log platform (such as a SIEM). Count tokens with /metrics (Prometheus format), exposed only to the monitoring network.

   nginx · inside http { }

   log_format llm '$time_iso8601 user=$user ip=$remote_addr "$request" '
                  'status=$status bytes=$body_bytes_sent rt=$request_time';
   access_log /var/log/nginx/llm.log llm;

   If you enable --enable-log-requests in vLLM at DEBUG level, the prompt text ends up in the logs. Decide whether to keep the text, and for how long, according to your rules on personal and confidential data.

8. Decide users and permissions

   UI (Open WebUI or similar)

   If people will chat through a UI, put a front end such as Open WebUI in the same closed network, and limit which models each role and group can use. Make sure people who leave are cut off through your internal IdP.

   env · Open WebUI

   OFFLINE_MODE=true
   ENABLE_SIGNUP=false
   DEFAULT_USER_ROLE=pending
   ENABLE_COMMUNITY_SHARING=false

   With OFFLINE_MODE enabled, document search (RAG) won't work unless the embedding model is installed beforehand. Bring in the embedding model through the same flow as steps 1 to 3.

9. Set up the update flow in advance

   Transfer machine → test environment → production

   vLLM releases roughly every two weeks, and almost every release includes security fixes. Even in a closed network, upgrade regularly through "fetch → verify → record → production."

   bash

   # 1. Fetch, check and bring in the new version as in steps 1 to 3
   # 2. In the test environment, send the same questions to the old and new versions and compare
   # 3. List the environment variable names in use and check against the release notes that none were removed
   sudo docker inspect --format '{{range .Config.Env}}{{println .}}{{end}}' qwen \
     | grep -E '^(VLLM_|HF_)' | cut -d= -f1
   # 4. Once in production, log one line on who upgraded when, and when it was rolled back
   echo "$(date -I) vllm <old version>→<new version> owner:<name> verified:OK rollback:<old version>.tar" >> /srv/llm/CHANGELOG

   Keep the previous container and model, with their digests noted, so you can roll back. Compare values that change between versions, such as the context length limit, before and after.

Local LLM security checklist: 20 items before production

Go through these from top to bottom before and after setup. The checkboxes work only on this screen and nothing is sent anywhere.

0 / 20 checked

Common mistakes: the gaps that "I thought it was fine" leaves in a local LLM

Common mistakes on the left, the fix from this page's steps on the right.

• --host 0.0.0.0 with no authUsing a sample start command as is and exposing it to the whole LAN.
  127.0.0.1 + nginxListen locally only. Users go through the gateway.
• Modified versions of unknown originUsing community GGUFs such as "Uncensored" builds without checking where they came from.
  From the official Qwen/, pinned by SHAIf you use a third-party version, record its author and process.
• "It has an api-key, so it's safe"Assuming vLLM's key protects every endpoint.
  Narrow it with an allowlistPass only what you need, such as /v1/chat/completions.
• "It's on-premises, so it's safe"Skipping permissions, logs and RAG access control because it sits inside the company.
  Manage it in-house like anything externalSSO, roles, logs, and search that respects access rights.
• Using it from outside via a tunnelExposing an internal tool to the outside with Cloudflare Tunnel or similar.
  Only from the internal networkIf outside access is needed, go through the company's official remote access.
• Missing default outbound trafficvLLM's usage stats, or auto-updates in the Ollama desktop app.
  Both environment variables and outbound blockingStop it in settings, and stop it at the firewall too.
• Trusting small models too muchIn agents and tool calls, they invent files or links that don't exist.
  Use it for first draftsRun things in a sandbox, and have a person check the results.
• Breaking on updateAfter upgrading, removed environment variables or changed limits stop it from running.
  Compare before and after in a test environmentCheck against the release notes and record who upgraded it.

What the videos say: real examples of self-hosting Qwen

We went through 18 explainer videos in English, Chinese and Japanese published between February and October 2026, down to their audio transcripts.

Where SIMY fits: Qwen inside the closed network, SIMY outside

To be upfront: SIMY does not currently run on Qwen, and it does not run inside a closed network or offline.

* This division of work is an illustration

• Picks up from meetings and chat
• Your way of working
• Returns only decisions

Confidential content is handled by Qwen in the closed network. After that, for "who does what by when," SIMY moves forward only the part the company has decided may go outside, within tools the company allows.

• How SIMY runs: cloud runs use your own ChatGPT account (Codex). Codex usage falls within your ChatGPT plan.
• Running on your PC: the SIMY desktop app can use Claude Code on your PC to do the work. Download it here.
• Reading chat history: the desktop app reads your Claude Code and Codex chat history and turns work you repeat into "Suggestions from SIMY." Chat content is not stored on our servers. Decide which PCs and what scope according to your company's rules.

Where to draw the line: your company's rules decide which information stays in the closed network and what may be handled by outside tools. How SIMY handles data is published in Security and the Privacy Policy.

Local LLM and Qwen FAQ

Sources

Steps, default settings and licenses were checked against the following official sources (checked October 1, 2026). Versions and defaults change often, so check each official page before you build.

• Security (vLLM docs), Usage Stats Collection, vllm serve
• vLLM release notes (GitHub)
• Server Arguments (SGLang docs)
• FAQ (Ollama docs)
• llama.cpp server README (GitHub)
• Deploying with vLLM (Qwen docs), Speed Benchmark (Qwen docs)
• Qwen (Hugging Face), Qwen3.8-27B model card
• Hugging Face CLI, Environment variables (Hugging Face Hub)
• ngx_http_auth_request_module (nginx docs), ngx_http_ssl_module
• Environment Variable Configuration (Open WebUI docs)
• Packet filtering and firewalls (Docker docs)
• qwen3.8 (Ollama library)