Unlock the Potential of AI

In this article, we introduce how to run LLMs efficiently on Apple Silicon with disaggregated inference technique.

Trimming large multilingual vocabularies in Small Language Models (SLM) is a simple, low-risk way to boost efficiency to its limit. It accelerates the model inference significantly while keeping accuracy almost unchanged.

LoRA excels at efficient fine-tuning but suffers at higher ranks due to gradient entanglement. We introduce GraLoRA, which addresses these issues through finer-grained, block-wise updates, significantly enhancing performance and expressivity without overhead. GraLoRA outperforms LoRA across tasks, achieving up to +8.5% improvement in HumanEval+ Pass@1.

SqueezeBits has partnered with Intel to make Gaudi NPUs more usable in practice. We optimized LLMs and diffusion models for Gaudi-2 and created yetter, a generative AI API service.

This article discusses inference efficiency when running the FLUX.1 models on Intel Gaudi-2 hardware.

With TensorRT-LLM now open source, we can finally take a deep dive into the secret sauce behind its impressive performance.

A brief review of the research paper from our team, published at ICML 2024.

This article is about an open-source library for direct conversion of PyTorch models to TensorRT-LLM.

This article explores the rise and fall of ONNX, from its early success as a unifying stasndard for AI frameworks to its gradual shift into a niche tool in the era of PyTorch 2.0.

This article provides a comparative analysis of serving vision-language models on vLLM and TensorRT-LLM.

In this blog series, we thoroughly evaluate Intel's AI accelerator, the Gaudi series, focusing on its performance, features, and usability.

In this blog series, we thoroughly evaluate Intel's AI accelerator, the Gaudi series, focusing on its performance, features, and usability.

This article provides a comparative analysis of automatic prefix caching.

This article provides a comparative analysis of speculative decoding.

This article provides a comparative analysis of multi-LoRA serving capabilities of vLLM and TensorRT-LLM frameworks.

In this blog series, we thoroughly evaluate Intel's AI accelerator, the Gaudi series, focusing on its performance, features, and usability.

This article provides a comparative analysis of different parallelism strategies on vLLM and TensorRT-LLM frameworks.

In this blog series, we thoroughly evaluate Intel's AI accelerator, the Gaudi series, focusing on its performance, features, and usability.

This article provides a comparative analysis of the effects of KV cache quantization on vLLM and TensorRT-LLM frameworks.

This article provides a comparative analysis of the effects of weight-activation quantization on vLLM and TensorRT-LLM frameworks.

This article provides a comparative analysis of the effects of weight-only quantization on vLLM and TensorRT-LLM frameworks.

This article provides a comparative analysis of vLLM and TensorRT-LLM frameworks, focusing on performance with fixed and dynamic datasets.

This article provides a comparative analysis of schedulers in vLLM and TensorRT-LLM frameworks.

This article provides a comparative analysis of vLLM and TensorRT-LLM frameworks with various sampling methods.

This article provides a comparative analysis of vLLM and TensorRT-LLM frameworks, focusing on batching configurations and thoroughly examining the effects of maximum batch size and maximum number of tokens.

This article provides a comparative analysis of vLLM and TensorRT-LLM frameworks for serving LLMs, evaluating their performance based on key metrics like throughput, TTFT, and TPOT to offer insights for practitioners in optimizing LLM deployment strategies.

Estimating the cost savings from model compression.

An introduction to tokenizers and their implications in language models.

Clarifying the misunderstandings in AI model compression

The Blackwell GPU from GTC 2024 was astonishing. Analysis of the Nvidia GPU evolution & what it means for GPU users.

Do I need to COMPRESS my AI model? : the short answer is “YES” — and here’s why.

AI model compression for acceleration is essential. The question is HOW? Here are 4 key methodologies.