SqueezeBits

This article introduces Fits on Chips, an LLMOps toolkit for performance evaluation.

A brief review of the research paper from our team, published at ICML 2024.

This article is about an open-source library for direct conversion of PyTorch models to TensorRT-LLM.

This article explores the rise and fall of ONNX, from its early success as a unifying stasndard for AI frameworks to its gradual shift into a niche tool in the era of PyTorch 2.0.

This article provides a comparative analysis of serving vision-language models on vLLM and TensorRT-LLM.

In this blog series, we thoroughly evaluate Intel's AI accelerator, the Gaudi series, focusing on its performance, features, and usability.

In this blog series, we thoroughly evaluate Intel's AI accelerator, the Gaudi series, focusing on its performance, features, and usability.

This article provides a comparative analysis of automatic prefix caching.

This article provides a comparative analysis of speculative decoding.

This article provides a comparative analysis of multi-LoRA serving capabilities of vLLM and TensorRT-LLM frameworks.

In this blog series, we thoroughly evaluate Intel's AI accelerator, the Gaudi series, focusing on its performance, features, and usability.

This article provides a comparative analysis of different parallelism strategies on vLLM and TensorRT-LLM frameworks.

In this blog series, we thoroughly evaluate Intel's AI accelerator, the Gaudi series, focusing on its performance, features, and usability.

This article provides a comparative analysis of the effects of KV cache quantization on vLLM and TensorRT-LLM frameworks.

This article provides a comparative analysis of the effects of weight-activation quantization on vLLM and TensorRT-LLM frameworks.

This article provides a comparative analysis of the effects of weight-only quantization on vLLM and TensorRT-LLM frameworks.

This article provides a comparative analysis of vLLM and TensorRT-LLM frameworks, focusing on performance with fixed and dynamic datasets.

This article provides a comparative analysis of schedulers in vLLM and TensorRT-LLM frameworks.

This article provides a comparative analysis of vLLM and TensorRT-LLM frameworks with various sampling methods.

This article provides a comparative analysis of vLLM and TensorRT-LLM frameworks, focusing on batching configurations and thoroughly examining the effects of maximum batch size and maximum number of tokens.

This article provides a comparative analysis of vLLM and TensorRT-LLM frameworks for serving LLMs, evaluating their performance based on key metrics like throughput, TTFT, and TPOT to offer insights for practitioners in optimizing LLM deployment strategies.

Estimating the cost savings from model compression.

An introduction to tokenizers and their implications in language models.

Clarifying the misunderstandings in AI model compression

The Blackwell GPU from GTC 2024 was astonishing. Analysis of the Nvidia GPU evolution & what it means for GPU users.

Do I need to COMPRESS my AI model? : the short answer is “YES” — and here’s why.

AI model compression for acceleration is essential. The question is HOW? Here are 4 key methodologies.