Load-testing large language models (LLMs) is a critical step in ensuring they can handle the expected production traffic effectively. Unlike traditional machine learning (ML) models, LLMs generally have lower requests per second (RPS) and higher latency due to their computational demands. Token-based metrics, such as time to first token and total output tokens per second, offer a more accurate representation of an LLM's performance under varying input and output sizes. This article explores the use of LLMPerf, a load-testing tool, specifically for Amazon Bedrock, to help benchmark and optimize different LLMs. LLM-Specific Metrics Traditionally, performance metrics like RPS and latency have been used to evaluate models. However, for LLMs, these metrics are less informative because requests can vary widely in complexity and length. Key metrics for LLMs include: Time to First Token (TTFT): The duration it takes for the model to generate the first token of a response. This is particularly useful for streaming applications. Total Output Tokens Per Second: The total number of tokens generated per second, providing a more granular view of performance. Other metrics, such as inter-token latency, can also be valuable. These metrics help in understanding how an LLM performs with different types of queries and input sizes. LLMPerf Introduction LLMPerf is a load-testing tool built on Ray, a distributed computing framework. It simulates real-time production-level traffic to test the performance of LLMs. The tool exposes parameters that are crucial for load testing LLMs, including: Mean and standard deviation of input and output tokens. Maximum number of completed requests. Number of concurrent requests. Timeout duration. These parameters can be adjusted to match the expected production load, allowing for more accurate and tailored testing. Applying LLMPerf to Amazon Bedrock Setup For this demonstration, we use a SageMaker Classic Notebook Instance with a conda_python3 kernel and a high-compute instance (ml.g5.12xlarge). Ensure your AWS credentials are set up to access the hosted model on Bedrock or SageMaker. ```python import os from litellm import completion os.environ["AWS_ACCESS_KEY_ID"] = "Enter your access key ID" os.environ["AWS_SECRET_ACCESS_KEY"] = "Enter your secret access key" os.environ["AWS_REGION_NAME"] = "us-east-1" response = completion( model="anthropic.claude-3-sonnet-20240229-v1:0", messages=[{ "content": "Who is Roger Federer?","role": "user"}] ) output = response.choices[0].message.content print(output) ``` This code configures the completion API to work with Amazon Bedrock's Claude 3 Sonnet model and passes a prompt, demonstrating how messages are consistently formatted across different model providers. LLMPerf Bedrock Integration To execute a load test, use the token_benchmark_ray.py script provided by LLMPerf. Adjust the parameters to match your production expectations: sh python llmperf/token_benchmark_ray.py \ --model bedrock/anthropic.claude-3-sonnet-20240229-v1:0 \ --mean-input-tokens 1024 \ --stddev-input-tokens 200 \ --mean-output-tokens 1024 \ --stddev-output-tokens 200 \ --max-num-completed-requests 30 \ --num-concurrent-requests 1 \ --timeout 300 \ --llm-api litellm \ --results-dir bedrock-outputs This script runs for 300 seconds, generating a specified number of concurrent requests. The results are saved in the bedrock-outputs directory, including individual responses and summary statistics. Parsing Results To make the results more readable, parse the summary file using pandas: ```python import json from pathlib import Path import pandas as pd individual_path = Path("bedrock-outputs/bedrock-anthropic-claude-3-sonnet-20240229-v1-0_1024_1024_individual_responses.json") summary_path = Path("bedrock-outputs/bedrock-anthropic-claude-3-sonnet-20240229-v1-0_1024_1024_summary.json") with open(individual_path, "r") as f: individual_data = json.load(f) with open(summary_path, "r") as f: summary_data = json.load(f) summary_metrics = { "Model": summary_data.get("model"), "Mean Input Tokens": summary_data.get("mean_input_tokens"), "Stddev Input Tokens": summary_data.get("stddev_input_tokens"), "Mean Output Tokens": summary_data.get("mean_output_tokens"), "Stddev Output Tokens": summary_data.get("stddev_output_tokens"), "Mean TTFT (s)": summary_data.get("results_ttft_s_mean"), "Mean Inter-token Latency (s)": summary_data.get("results_inter_token_latency_s_mean"), "Mean Output Throughput (tokens/s)": summary_data.get("results_mean_output_throughput_token_per_s"), "Completed Requests": summary_data.get("results_num_completed_requests"), "Error Rate": summary_data.get("results_error_rate") } print("Claude 3 Sonnet - Performance Summary:\n") for k, v in summary_metrics.items(): print(f"{k}: {v}") ``` The output will display metrics like TTFT, inter-token latency, and output throughput, providing insights into the model's performance. Real-World Use Cases In a real-world scenario, LLMPerf can be used to compare different LLMs and deployment configurations. By running tests across multiple model providers and settings, you can identify the best model and stack for your specific use case. This holistic approach ensures that your LLM is optimized and robust, ready to handle the expected production traffic. Additional Resources & Conclusion The complete code for this demonstration is available in the associated GitHub repository. If you need to work with SageMaker endpoints, there are additional samples available, such as a Llama JumpStart deployment load testing example. Load testing and evaluation are essential for deploying performant LLMs. By using LLMPerf, you can gain valuable insights into your model's capabilities and ensure it meets the demands of your application. Future articles will cover more comprehensive testing and evaluation strategies, providing a well-rounded approach to productionizing generative AI applications. Industry Insights Industry insiders emphasize the importance of load testing in the deployment process of LLMs. According to a Machine Learning Architect at AWS, understanding how an LLM performs under production-like conditions is crucial for ensuring reliability and efficiency. LLMPerf, with its focus on token-based metrics, offers a robust solution for this challenge, making it a valuable tool for any tech company looking to deploy large language models. Company Profile Amazon Bedrock is a managed service that provides a scalable and secure environment for running and deploying large language models. It supports a variety of models, including Claude 3 Sonnet, and integrates seamlessly with AWS services like SageMaker. This service is designed to help developers and data scientists efficiently manage and scale their generative AI applications.