Last updated May 16, 2025

The /v1/chat/completions endpoint generates conversational completions for a provided set of input messages. You can specify the model, adjust generation settings such as temperature , and opt to stream the responses in real time. You can also specify tools the model can choose to call.

Selecting a chat model: for the best intelligence, we recommend using a Claude Sonnet model; for cost-savings and fast inference we recommend Claude Haiku.

Request Body Parameters

Use parameters to manage how conversational completions are generated.

Required Parameters

Optional Parameters

extended_thinking Object

tools Array of Objects

tools lets you provide your model with an array of tools it can choose to call. Use tool_choice to specify how the model calls tools. When provided, your model may send back tool_calls in the role="assistant" generated message, asking your system to run the specified tool, and send back the result in a role="tool" message.

Note that these tools are given to the model in the form of an extended prompt and no further validation is done. Models may make up tool names that don’t exist in the tools array you have given them. To avoid this, we recommend you perform tool validation on your end when a model sends back a tool_calls assistant message.

function Object

Example tools Array

[
    {
      "type": "function",
      "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The city and state, e.g. Portland, OR"
            }
          },
          "required": ["location"]
        }
      }
    }
  ]

tool_choice Object

The tool_choice object specifies how the model should use the provided tools.

It can either be a string (none, auto, or required), or a tool_choice object. none will mean the model will call no tools. auto allows the model to call zero to many of the provided tools, and required forces the model to call at least one or more tools before responding to the user.

To force the model to call a specific tool, you may simply specify a single tool in the tools object and pass "tools": "required", or you can force the tool selection by passing a tool_choice object that specifies the required function.

messages Array of Objects

The messages object is an array of message objects.

Each message must specify a role field that determines the messages’s schema (see below).

Currently, the supported types are user, assistant, system, and tool.

If the most recent message uses the assistant role, the model will continue its answer starting from the content in that most recent message.

role=user message

user messages are the primary way to send queries to your model and prompt it to respond.

role=assistant message

Typically, assistant messages are only generated by the model, however you can create your own or pre-fill a partially completed assistant response to help influence the content that the model will generate on its next turn.

Tool Call Object

Tool Call Example

Here’s an example of what a tool_calls object might look like, when your model has decided to call a tool you’ve given to it as an option via tools.

"tool_calls": [
  {
    "id": "toolu_02F9GXvY5MZAq8Lw3PTNQyJK",
    "type": "function",
    "function": {
        "name": "get_weather",
        "arguments": "{\"location\":\"Portland, OR\"}"
    }
  }
]

Function Object

role=system message

A system message is sort of a prompt ‘prefix’ that is given to the model to help influence its responses.

role=tool message

A tool message object lets you communicate a specified tool’s result (output) to the model.

Request Headers

In the following example, we assume your model resource has an alias of “INFERENCE” (the default).

All inference curl requests must include an Authorization header containing your Heroku Inference key.

Response Format

When a request is successful, the API returns a JSON object with the following structure:

Choice Object

The object inside the choices array (length 1) has the following structure:

Message Object

Reasoning Object

If extended_thinking.include_reasoning is set to true, the model returns a reasoning object inside the message.

Usage Object

Information about token consumption.

Example Request

Let’s walk through an example /v1/chat/completions curl request.

First, use this command to set your Heroku environment variables as local variables. bash eval $(heroku config -a $APP_NAME --shell | grep '^INFERENCE_' | sed 's/^/export /' | tee >(cat >&2)) Next, send the curl request:

curl $INFERENCE_URL/v1/chat/completions \
 -H "Authorization: Bearer $INFERENCE_KEY" \
 -d @- <<EOF | jq
{
  "model": "$INFERENCE_MODEL_ID",
  "messages": [{"role": "user", "content": "Hello"}]
}
EOF

Example Response

{
  "id": "chatcmpl-1839afa8133ceda215788",
  "object": "chat.completion",
  "created": 1745619466,
  "model": "claude-3-7-sonnet",
  "system_fingerprint": "heroku-inf-1y38gdr",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hi! How can I help you today?",
        "refusal": null
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 8,
    "completion_tokens": 12,
    "total_tokens": 20
  }
}

Example Request with Tools

 curl $INFERENCE_URL/v1/chat/completions \
 -H "Authorization: Bearer $INFERENCE_KEY" \
 -d @- <<EOF | jq
 {
    "model": "$INFERENCE_MODEL_ID",
    "messages": [
        {
            "role": "user",
            "content": "What's the weather like in Portland?"
        }
    ],
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "get_current_weather",
                "description": "Get the current weather in a given location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g. Portland, OR"
                        }
                    },
                    "required": [
                        "location"
                    ]
                }
            }
        }
    ]
}
EOF

Example Response with Tools

{
  "id": "chatcmpl-1839adcc2079997417288",
  "object": "chat.completion",
  "created": 1745617422,
  "model": "claude-3-7-sonnet",
  "system_fingerprint": "heroku-inf-1y38gdr",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "I'll help you check the current weather in Portland. Since Portland could refer to either Portland, Oregon or Portland, Maine, I should specify the state.\nI'll check Portland, OR as it's the larger and more commonly referenced Portland.",
        "refusal": null,
        "tool_calls": [
          {
            "id": "tooluse_aFByQsacQ_2BmYMGHvkBmg",
            "type": "function",
            "function": {
              "name": "get_current_weather",
              "arguments": "{\"location\":\"Portland, OR\"}"
            }
          }
        ]
      },
      "finish_reason": "tool_calls"
    }
  ],
  "usage": {
    "prompt_tokens": 407,
    "completion_tokens": 107,
    "total_tokens": 514
  }
}