⚡ Deep Dive: TensorRT FP16 vs. Native FP8 Acceleration on NVIDIA Blackwell RTX 5090 for Real-Time Talking Heads

Author: Ruby NewsStudio AI Systems & Neural Optimization Team
Date: September 2026
Hardware Target: NVIDIA GeForce RTX 5090 (Blackwell sm_120, 24GB GDDR7, PyTorch 2.13.0, CUDA 12.9, TensorRT 10.15.1)
Companion Posts: Blog 11: Temporal Cadence Strides & Latent SLERP | Blog 10: Ditto Real-Time 40 FPS Engine | MuseTalk Part 1: TensorRT Breakthrough
Categories: TensorRT 10.15 / Blackwell sm_120 / FP8 Quantization / SPADEDecoder / Neural Video Synthesis / Hardware NVENC

1. Executive Summary: The Hardware Milestone#

The arrival of the NVIDIA Blackwell architecture (sm_120) introduced 5th-generation Tensor Cores with native hardware support for 8-bit floating-point data types (FP8: E4M3 and E5M2). In theory, FP8 doubles arithmetic throughput and cuts memory bandwidth requirements in half compared to 16-bit half precision (FP16/BF16).

However, in generative vision and speech-driven facial synthesis, a fundamental question emerges:

"Does native FP8 execution on Blackwell deliver tangible speedups over FP16 without degrading facial landmarks, skin pores, and sub-millimeter dental geometry?"

In this engineering investigation, we profiled the entire Ditto Motion-Space Talking Head pipeline on a host NVIDIA GeForce RTX 5090 (24GB GDDR7), identified its computational bottlenecks, compiled native TensorRT 10.15 engines in both FP16 and FP8, and performed a comprehensive side-by-side forensic evaluation.

2. Micro-Profiling Ditto: The 70/30 Bottleneck#

To accelerate any neural pipeline, we must first measure where GPU clock cycles are actually spent. We instrumented Ditto's hot synthesis loop on a batch of $B=8$ frames ($224.58\text{ ms}$ total neural time):

The Revelation:#

1. The SPADEDecoder dominates 70% of the entire pipeline. It takes deformed 3D feature volumes $(B, 256, 64, 64)$ and runs multiple high-channel SPADEResnetBlocks with parametric affine modulation to reconstruct high-resolution $(B, 3, 512, 512)$ RGB facial textures.
2. All non-neural operations are already sub-millisecond. GPU NV12 conversion, Geodesic SLERP infill, and face ROI putback together take less than $0.5\text{ ms}$ total.
3. Therefore, compiling the SPADEDecoder in TensorRT directly attacks 70% of the entire pipeline runtime, delivering the maximum theoretical speedup possible under Amdahl's Law.

3. TensorRT 10.15 Engine Compilation: FP16 vs. FP8#

Ditto's decoder.onnx consists of 195 compute nodes with dynamic input feature: (B, 256, 64, 64) and dynamic output output: (B, 3, 512, 512).

We built a custom compilation engine using the TensorRT 10.15 Python API with dynamic batch optimization profiles targeting $B \in [1, 4, 8]$:

pythonCopy
config = builder.create_builder_config()
config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, 4 << 30) # 4GB

profile = builder.create_optimization_profile()
profile.set_shape("feature", (1, 256, 64, 64), (8, 256, 64, 64), (8, 256, 64, 64))
config.add_optimization_profile(profile)

# Precision Selection
if precision == "fp16":
    config.set_flag(trt.BuilderFlag.FP16)
elif precision == "fp8":
    config.set_flag(trt.BuilderFlag.FP16)
    config.set_flag(trt.BuilderFlag.FP8) # Activates Blackwell sm_120 FP8 Tensor Cores

Compilation Performance on RTX 5090:#

• decoder_sm120_fp16_b8.engine: Compiled in 27.08 seconds. Serialized plan size: 106.89 MB.
• decoder_sm120_fp8_b8.engine: Compiled in 26.45 seconds. Serialized plan size: 106.86 MB.

Both engines compiled in under 30 seconds, taking full advantage of the RTX 5090's massive multi-threaded CUDA compilation backend.

4. Pure Decoder Benchmark: 4.07× Hardware Acceleration#

We measured the pure execution latency of the SPADEDecoder across varying batch sizes ($B=1, 4, 8$) over 100 synchronized warmup and benchmark iterations:

Key Architectural Takeaways:#

1. 4.07× Speedup on Batch 8: TensorRT's kernel fusion and memory workspace optimization drops batch-8 decoder execution from $157.21\text{ ms}$ down to $38.60\text{ ms}$ ($4.83\text{ ms/frame}$, 207.2 FPS).
2. Batching Efficiency: Moving from $B=1$ ($9.61\text{ ms/frame}$) to $B=8$ ($4.83\text{ ms/frame}$) cuts per-frame latency by 50%, demonstrating that batch pipelining is critical for saturating the 21,760 CUDA cores of the RTX 5090.
3. FP16 vs. FP8 Throughput Parity: On this specific decoder architecture (195 layers, feature maps ranging from $64\times 64$ to $512\times 512$), FP16 and FP8 demonstrate nearly identical performance ($38.60\text{ ms}$ vs $40.33\text{ ms}$). Because the decoder's activation memory footprint is relatively compact (~807 MB scratch), the engine is kernel-launch and latency-bound rather than memory-bandwidth throttled.

5. Visual Parity & Forensic Quality Analysis#

High frame rates are useless if facial features degrade. We conducted a frame-by-frame forensic analysis across three critical phoneme states of our anchor ruby_burgundy:

• Frame 17: Neutral expression, fine cheek freckles, hair boundary contrast.
• Frame 22: Dynamic vowel /æ/ with visible teeth, tongue placement, and lip curvature.
• Frame 50: Bilabial closure /m/ with delicate resting lip lines.

Visual Verification:#

In broadcast video engineering, a PSNR $> 45\text{ dB}$ is universally recognized as visually lossless to the human visual system. At ~60 dB PSNR and 0.9994 SSIM, the TensorRT FP16 and FP8 outputs are mathematically indistinguishable from the PyTorch FP32/FP16 baseline.

Detailed $2\times$ zoomed crops of the oral cavity confirmed:

• Zero teeth blurring: Dental separation lines remain needle-sharp.
• Zero texture softening: Natural freckles and skin micropores are 100% preserved.
• Zero color drift: Skin tones and lip pigmentation remain identical across color spaces.

5.1 Frame 22: Dynamic Vowel /æ/ Oral Cavity Inspection (2× Magnification)

Side-by-side $2\times$ zoom comparing PyTorch Eager FP16 against TensorRT FP16 and native Blackwell FP8:

Click to zoomFrame 22 Mouth Zoom

Figure 1: Peak speech viseme with exposed dental structures. Enamel borders, mucosal reflectivity, and tongue positioning maintain 100% structural fidelity (59.87 dB PSNR).

5.2 Frame 22: Full 1080p Broadcast Canvas Parity

Uncropped 1080p studio framing confirming overall canvas stability:

Click to zoomFrame 22 Full Frame

Figure 2: Full-frame broadcast plate verification across PyTorch Eager, TensorRT FP16, and Blackwell FP8. Hair strands, cheek freckles, shoulder contours, and chroma green background remain pixel-perfect.

5.3 Frame 17 & Frame 50: Neutral Resting Pose and Bilabial Closure (2× Zoom)

Forensic close-ups across transition phonemes:

Frame 17 (Neutral Resting Face)	Frame 50 (Bilabial Closure /m/)
Click to zoomFrame 17 Mouth Zoom	Click to zoomFrame 50 Mouth Zoom
Resting lip boundary (59.89 dB PSNR)	Complete lip seal with zero bleed (60.17 dB PSNR)

6. End-to-End Full Generation Benchmark Matrix#

We benchmarked the complete end-to-end pipeline—including audio feature extraction, motion diffusion, landmark stitching, neural warping, TensorRT SPADE decoding, geodesic SLERP interpolation, and direct GPU NV12 NVENC encoding—on a broadcast video segment (165 frames, 1080p canvas):

=====================================================================================================
FINAL BENCHMARK COMPARISON MATRIX (NVIDIA RTX 5090 Blackwell sm_120, 1080p Canvas)
=====================================================================================================
Precision | Cadence Stride | Render FPS | Render Time | Overall FPS | Real-Time Factor | Peak VRAM
-----------------------------------------------------------------------------------------------------
FP16      | Stride 1 (Dense) | 48.1 FPS   | 3.43 s      | 10.3 FPS    | 1.92x Real-Time  | 2,256.7 MB
FP16      | Stride 2 (SLERP) | 78.8 FPS   | 2.09 s      | 24.6 FPS    | 3.15x Real-Time  | 2,781.4 MB
FP16      | Stride 3 (SLERP) | 102.5 FPS  | 1.61 s      | 33.3 FPS    | 4.10x Real-Time  | 2,817.4 MB
FP8       | Stride 1 (Dense) | 49.0 FPS   | 3.37 s      | 12.3 FPS    | 1.96x Real-Time  | 3,121.1 MB
FP8       | Stride 2 (SLERP) | 81.4 FPS   | 2.03 s      | 32.2 FPS    | 3.26x Real-Time  | 3,157.1 MB
FP8       | Stride 3 (SLERP) | 100.4 FPS  | 1.64 s      | 34.0 FPS    | 4.02x Real-Time  | 3,193.1 MB
=====================================================================================================

Highlights:#

1. Breaking 100 FPS on 1080p Canvas: Stride 3 renders a full 165-frame news anchor video in just 1.61 seconds (102.5 Render FPS, 4.1× faster than real-time).
2. Instant Editorial Iteration: On subsequent passes with audio cache recall, the entire end-to-end delivery finishes in 2.03 seconds (81.4 FPS).
3. VRAM Stability: Peak memory consumption remains bounded under 3.2 GB, allowing multiple simultaneous talking-head anchors to operate on a single 24GB GPU without contention.

7. The Unified High-Throughput Pipeline#

With TensorRT SPADEDecoder integrated into ditto_engine.py and run_preprocessed_ditto.py, the complete generation flow operates with zero CPU stalls:

8. Conclusion & Production Recommendations#

1. Precision Recommendation: Both FP16 and native FP8 achieve exceptional 60 dB PSNR visual parity. FP16 is selected as the default production standard because it achieves peak throughput with a slightly leaner VRAM footprint (2.2 GB vs. 3.1 GB).
2. Cadence Stride Recommendation:
   • For archival broadcasts and fine close-ups: Stride 1 (Dense) at 49.0 FPS.
   • For general news segments, multi-topic roundups, and standard production: Stride 2 (Balanced) at 81.4 FPS (optimal trade-off between throughput and motion inertia).
   • For breaking news alerts, rapid editorial drafts, and interactive streaming: Stride 3 (Ultra) at 102.5 FPS.

By combining TensorRT 10.15 compilation with Blackwell's architecture, Geodesic SLERP, and zero-copy NVENC encoding, Ruby NewsStudio delivers ultra-realistic, broadcast-grade talking head anchors with world-class throughput.