Bootstrap

LiteLLM


一、关于 LiteLLM🚅

使用 OpenAI 格式调用 LLM APIs。 使用 Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate, Groq (100+ LLMs)。


LiteLLM 管理一下:

  • 将输入转换到提供者的completionembeddingimage_generation端点
  • 一致的输出,文本响应将始终可用在['choices'][0]['message']['content']
  • 跨多个部署 重试/兜底 逻辑(例如Azure/OpenAI)— 路由器
  • 设置每个项目的预算和速率限制,api密钥,模型 OpenAI Proxy Server

跳转到OpenAI代理文档
跳转到支持的LLM提供程序


🚨**稳定版发布:**使用带有-stable标签的docker镜像。这些镜像在发布之前经过了12小时的负载测试。

支持更多提供者。缺少提供者或LLM平台,提出特征请求


企业级

对于需要更好的安全性、用户管理和专业支持的公司

与创始人交谈

这包括:

  • LiteLLM商业许可下的功能
  • ✅特征优先级
  • ✅自定义集成
  • ✅专业支持-专门的不和谐+松弛
  • ✅自定义SLA
  • ✅单点登录的安全访问

我们为什么要建造这个?

  • 需要简单:我们的代码开始变得极其复杂,在Azure、OpenAI和Cohere之间管理和转换调用。

二、用法

重要提示:

LiteLLM v1.0.0现在需要openai>=1.0.0。迁移指南在这里
LiteLLM v1.40.14+现在需要pydantic>=2.0.0。无需更改。


colab 示例:https://colab.research.google.com/github/BerriAI/litellm/blob/main/cookbook/liteLLM_Getting_Started.ipynb


安装

pip install litellm

from litellm import completion
import os
***
## set ENV variables
os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["COHERE_API_KEY"] = "your-cohere-key"

messages = [{ "content": "Hello, how are you?","role": "user"}]

# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)

# cohere call
response = completion(model="command-nightly", messages=messages)
print(response)

使用 model=<provider_name>/<model_name> 调用提供程序支持的任何模型。

这里可能有特定于提供程序的详细信息,因此详情可见 provider docs for more information


异步

详见文档:https://docs.litellm.ai/docs/completion/stream#async-completion

from litellm import acompletion
import asyncio

async def test_get_response():
    user_message = "Hello, how are you?"
    messages = [{"content": user_message, "role": "user"}]
    response = await acompletion(model="gpt-3.5-turbo", messages=messages)
    return response

response = asyncio.run(test_get_response())
print(response)

详见文档:https://docs.litellm.ai/docs/completion/stream

LiteLLM支持流式传输模型响应,传递stream=True以获得流式迭代器响应。
所有模型都支持流式传输(Bedrock、Huggingface、ToketherAI、Azure、OpenAI等)

from litellm import completion
response = completion(model="gpt-3.5-turbo", messages=messages, stream=True)
for part in response:
    print(part.choices[0].delta.content or "")

# claude 2
response = completion('claude-2', messages, stream=True)
for part in response:
    print(part.choices[0].delta.content or "")

日志可观测性

详见文档:https://docs.litellm.ai/docs/observability/callbacks

LiteLLM公开预定义的回调以将数据发送到Lunary、Langfuse、DynamoDB、s3 Buckets、Helicone、Prompttier、Traceloop、Athina、Slack

from litellm import completion
***
## set env variables for logging tools
os.environ["LUNARY_PUBLIC_KEY"] = "your-lunary-public-key"
os.environ["HELICONE_API_KEY"] = "your-helicone-auth-key"
os.environ["LANGFUSE_PUBLIC_KEY"] = ""
os.environ["LANGFUSE_SECRET_KEY"] = ""
os.environ["ATHINA_API_KEY"] = "your-athina-api-key"

os.environ["OPENAI_API_KEY"]

# set callbacks
litellm.success_callback = ["lunary", "langfuse", "athina", "helicone"] # log input/output to lunary, langfuse, supabase, athina, helicone etc

#openai call
response = completion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hi 👋 - i'm openai"}])

三、OpenAI代理

详见文档:https://docs.litellm.ai/docs/simple_proxy

跨多个项目跟踪支出+负载平衡

托管代理(预览版)

代理提供:

  1. Hooks for auth
  2. Hooks for logging
  3. Cost tracking
  4. Rate Limiting

📖代理端点

详见 Swagger Docs : https://litellm-api.up.railway.app/


快速启动代理-CLI

pip install 'litellm[proxy]'

第1步:启动litellm代理
$ litellm --model huggingface/bigcode/starcoder

#INFO: Proxy running on http://0.0.0.0:4000

第2步:向代理发出ChatCompletions请求
import openai # openai v1.0.0+
client = openai.OpenAI(api_key="anything",base_url="http://0.0.0.0:4000") # set proxy to base_url
# request sent to model set on litellm proxy, `litellm --model`
response = client.chat.completions.create(model="gpt-3.5-turbo", messages = [
    {
        "role": "user",
        "content": "this is a test request, write a short poem"
    }
])

print(response)

代理密钥管理

详见:https://docs.litellm.ai/docs/proxy/virtual_keys

将代理与Postgres DB连接以创建代理密钥

# Get the code
git clone https://github.com/BerriAI/litellm

# Go to folder
cd litellm

# Add the master key - you can change this after setup
echo 'LITELLM_MASTER_KEY="sk-1234"' > .env

# Add the litellm salt key - you cannot change this after adding a model
# It is used to encrypt / decrypt your LLM API Key credentials
# We recommned - https://1password.com/password-generator/ 
# password generator to get a random hash for litellm salt key
echo 'LITELLM_SALT_KEY="sk-1234"' > .env

source .env

# Start
docker-compose up

UI on /ui on your proxy server

外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传

跨多个项目设置预算和费率限制 POST /key/generate


请求
curl 'http://0.0.0.0:4000/key/generate' \
--header 'Authorization: Bearer sk-1234' \
--header 'Content-Type: application/json' \
--data-raw '{"models": ["gpt-3.5-turbo", "gpt-4", "claude-2"], "duration": "20m","metadata": {"user": "[email protected]", "team": "core-infra"}}'

预期反应

{
    "key": "sk-kdEXbIqZRwEeEiHwdg7sFA", # Bearer token
    "expires": "2023-11-19T01:38:25.838000+00:00" # datetime object
}

四、支持的 Providers

详见文档: https://docs.litellm.ai/docs/providers

ProviderCompletionStreamingAsync CompletionAsync StreamingAsync EmbeddingAsync Image Generation
openai
azure
aws - sagemaker
aws - bedrock
google - vertex_ai
google - palm
google AI Studio - gemini
mistral ai api
cloudflare AI Workers
cohere
anthropic
empower
huggingface
replicate
together_ai
openrouter
ai21
baseten
vllm
nlp_cloud
aleph alpha
petals
ollama
deepinfra
perplexity-ai
Groq AI
Deepseek
anyscale
IBM - watsonx.ai
voyage ai
[xinference Xorbits Inference]
FriendliAI

五、贡献

贡献:在本地克隆repo->进行更改->提交带有更改的PR。

以下是如何在本地修改repo:

第1步:克隆repo

git clone https://github.com/BerriAI/litellm.git

第2步:导航到项目中,并安装依赖项:

cd litellm
poetry install -E extra_proxy -E proxy

第3步:测试您的更改:

cd litellm/tests # pwd: Documents/litellm/litellm/tests
poetry run flake8
poetry run pytest .

第4步:提交包含更改的PR!🚀

  • 将您的fork推送到您的GitHub存储库
  • 从那里提交PR

2024-07-25(四)

;