BULLETIN

A new study identifies how tuning sequence lengths during large language model (LLM) inference can sharply reduce energy consumption. Researchers tested models from 1B to 9B parameters on NVIDIA H100 GPUs using TensorRT-LLM and found clear "sweet spots" where energy use drops significantly.

The Story

The team developed an analytical model revealing a non-linear link between sequence length and energy efficiency. Peak efficiency occurs with short-to-moderate inputs and medium-length outputs. Long inputs or very short outputs cause energy use to spike. Their model predicts consumption with under 2% error, outperforming simple linear estimates.

The Context

LLMs power many AI applications but come with rising energy costs and environmental concerns. Current energy estimates often assume a straight line relationship with sequence length, missing critical nuances. This study challenges that view, showing the relationship is more complex.

Researchers tested popular models—OPT, LLaMA, Gemma, Falcon, Qwen2, and Granite—across input and output lengths from 64 to 4,096 tokens. Using NVIDIA’s TensorRT-LLM, they precisely measured energy consumption on H100 GPUs.

The findings offer practical guidance: trimming inputs, summarizing content, or adjusting output length dynamically can hit these efficiency sweet spots. This approach can lower operational costs and shrink AI’s carbon footprint.

Key Takeaways

• Energy efficiency peaks with short-to-moderate input lengths and medium output lengths.
• Long inputs or very short outputs cause sharp drops in efficiency.
• The team’s analytical model predicts energy use with a mean error of 1.79%.
• Techniques like truncation, summarization, and adaptive generation help align workloads with efficiency sweet spots.
• TensorRT-LLM enables accurate energy measurements critical for real-world LLM deployment.