VLLM

Python ENV dependency Run pip install poetry==1.8.0 to install Poetry, which is Python packaging and dependency management tool. openai = "^1.30.3" fastapi = "^0.111.0" transformers = "^4.41.1" tiktoken = "^0.6.0" torch = "^2.3.0" sse-starlette = "^2.1.0" sentence-transformers = "^2.7.0" sentencepiece = "^0.2.0" accelerate = "^0.30.1" pydantic = "^2.7.1" timm = "^1.0.3" pandas = "^2.2.2" vllm = "^0.4.2" vLLM(<=0.4.2) not support tool_call This codes is the best way for tool_call. from openai import OpenAI client = OpenAI() messages = [{"role": "user", "content": "What's the weather like in San Francisco, Tokyo, and Paris?"}] tools = [...] response = client.chat.completions.create( model="gpt-4o", messages=messages, tools=tools, tool_choice="auto", # auto is default, but we'll be explicit ) response_message = response.choices[0].message tool_calls = response_message.tool_calls Due to latest vLLM does not support using tool_call with OpenAI python SDK, releated PR #3237. We will use corresponding model Prompt, then insert tools into request....