NVIDIA LLM Design Rules, Qwen 3.8 #84
Today's Letter
NVIDIA outlines hardware-friendly LLM co-design rules
- NVIDIA published a technical guide on hardware-aware LLM design on July 10, 2026, focused on improving throughput and interactivity without changing accuracy targets
- The post argues that transformer linear layers should use near-square dimensions and align with GPU tile sizes in multiples of 128, with 256 or 512 presented as preferred targets
- NVIDIA says model width, intermediate projection size, and layer count determine how well decoder-style LLMs map to GPU arithmetic intensity, memory movement, and parallel scaling
- The guidance highlights a width-over-depth balance for better GPU utilization, with model shape affecting whether GEMM workloads stay memory-bound or move into the compute-bound region
- For low-latency decoding, the post notes that workloads are typically memory-bound at low concurrency, so reducing memory access cost matters more than raw FLOPS
- NVIDIA positions NVFP4 quantization, supported through TensorRT Model Optimizer and LLM Compressor, as a way to raise throughput with limited accuracy loss across linear layers
- The post also points to expert parallelism, pipeline parallelism, and Helix Parallelism in TensorRT-LLM for scaling large Mixture-of-Experts deployments across multi-node Blackwell NVLink systems
- As one example of optimization priority, NVIDIA notes that if attention accounts for 77% of runtime, feed-forward tuning alone will have limited effect under Amdahl's law
Source: developer.nvidia.com
Qwen 3.8, 2.4T-parameter model introduced
- Alibaba Qwen introduced Qwen 3.8, described as a 2.4 trillion-parameter model, with open weights planned for a later release
- The company said the model is still evolving and positioned it as compatible with leading frontier AI systems
- Qwen3.8-Max-Preview is available now through Alibaba Token Plan, Qoder, and QoderWork
- Token Plan has separate pricing pages for international and China users, with international Lite, Standard, and Pro tiers priced at $6, $18, and $68 per month during the promotion
- The three tiers include 2,500, 10,000, and 40,000 credits per 7 days, plus 700, 3,000, and 12,000 credits per 5 hours
- Concurrent agent limits are listed at 1-2, 3-4, and 6-8 by plan tier
- The service can be connected from tools that support OpenAI and Anthropic-style protocols by setting an API key and base URL
- The release adds another large-model entry to the current China AI model race, while the open-weight release remains pending
Source: news.hada.io
More: officechai.com · bloomberg.com · scmp.com
Jocoletter curates AI, software, and product trends for developers and builders.
#NVIDIA #Qwen