Squeeze out GPU's performance through hardware-aware inference R&D

Such a task arises for businesses where the development is in phase:

• - all ETL processes are agreed and debugged
• - all machine learning pipelines are optimized at the upper levels of the PE to produce results intended for the "long haul"
• - various optimizations performed on infrastructure and logic layer regarding load distribution within GPU cluster.

In this research we consider possibilities of Minimize overheads in layers close to Hardware. Where the task "Squeeze out performance" means extracting maximum from available equipment.

In context of AI-inference it means minimizing time required for GPU processor to perform inference task (e.g., passing through neural network layers).

Thesis:

Problem

The problem we solve: Standard frameworks (PyTorch/TensorFlow) often cannot fully utilize specific instructions of next-gen GPU for very small models due to abstraction overhead.

Modern machine learning tasks (especially when working with large models) require an enormous amount of matrix operations. To achieve ultra-low latency and high throughput.

Project Goal:

Develop and implement high-performance inference (output) for "small" neural networks (NLP models), minimizing memory consumption and execution latency on the target device.

In computing engineering, (kernel) refers to the lowest level of abstraction where main processor operations are performed.

Consider:

• - Hardware consider how these kernels interact with memory, caches and compute blocks (e.g., tensor cores on GPU or specialized CPU instructions).
• - GPU Architecture:Study how CUDA works (Warp architecture, Shared Memory, Registers).

Mark a project with hardware-oriented solution.

• The solution must achieve maximum possible efficiency of executing critical computational operations directly at the processor level, bypassing overheads of OS and high-level abstractions. This can be:

Solution with a Warp-architecture addition

CUDA/Triton Consideration

N	technology	Link
1	Triton	github.com/triton-inference-server
1	Triton Inference Server on NVIDIA GPU Cloud (NGC)	github.com/triton-inference-server/server
1	TensorRT-LLM Backend	github.com/triton-inference-server/tensorrtllm_backend
1	The inflight_batcher_llm directory contains the C++ implementation of the backend supporting inflight batching, paged attention and more.	github.com/triton-inference-server/TensorRT-LLM/tree/main/triton_backend/all_models/inflight_batcher_llm
1	Programming for accelerators such as GPUs is critical for modern AI systems. This often means programming directly in proprietary low-level languages such as CUDA.	github.com/gpu-mode/Triton-Puzzles

Triton I found to be the most popular solution applicable to our task.

This is a good solution and unambiguously requires attention and detailed study, but I set it aside into a separate development section because this implementation requires a radically different approach.

Solution with a direct per-core streaming.

Consider: As all high-level libraries and operating systems that introduce additional delays in memory management, thread scheduling, and context switching. Replace them ( Сarefully! ) as much as possible.

• Dedicated, heavily optimized functions: Which directly perform mathematical and logical operations necessary for neural network inference (e.g., matrix multiplication, convolutions, activations).

Provide Direct access to hardware:

• Select solutions on CUDA, OpenCL, or use specific SIMD/AVX instructions. Any (at this stage) solution that directly interacts with the processor architecture or accelerator.
• Optimize parallelism: Distribute operations correctly so they utilize all available compute resources to maximum capacity.
• Manage memory efficiently: Organize data in memory such that minimize access to slow external memory and maximize usage of fast cache memory.
• Use architecture specifics: Use the fastest processor instructions for performing specific mathematical operations. Ensure calculations are done in minimum possible time.

Extension and features for Custom Kernel Implementation

Of course, a project on optimization using hardware-oriented programming (Kernel Optimization) in context of AI-inference often focuses on quantization, compression and high-performance computing (HPC). GEMM.

CUDA Kernel Optimization for INT8 GEMM in LLM Inference: This is also excellent as a solution due to math and quantization. Quality let's us put aside just focus provide ML developers an additional optimization tool.

I will consider combinations with emphasis on throughput and batching methods:

• High-Performance LLM Inference with Continuous Batching and Kernel Tuning
• PagedAttention Kernel Implementation
• Memory & Asynchronous Scheduling Optimization for PagedAttention