Tool calling in Lambda Inference API

Technical Help
1

While using the Public Inference API, I get a blank output when using tools field, but legitimate output when not using tools in the data.

This is the payload I’m using:


{
        "model": "llama3.1-8b-instruct",
        "messages": [
          {
            "role": "system",
            "content": "You are a helpful assistant named Hermes, made by Nous Research."
          },
          {
            "role": "user",
            "content": "Who won the world series in 2020?"
          },
          {
            "role": "assistant",
            "content": "The Los Angeles Dodgers won the World Series in 2020."
          },
          {
            "role": "user",
            "content": "Where was it played?"
          }
        ],
        "tools": [
          {
            "type": "function",
            "function": {
                "name": "say",
                "description": "Say",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "message": {
                            "type": "string",
                            "description": "message"
                        }
                    },
                    "required": ["message"]
                }
            }
          }
        ]
      }

Output:

{"id":"","object":"","created":0,"model":"","choices":null,"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0,"prompt_tokens_details":null,"completion_tokens_details":null},"system_fingerprint":""}
2

@dhruvmullick_aa I’m looking into this right now. I’ll post an update as soon as I have one.

1 Like
3

@dhruvmullick_aa The Lambda Inference API doesn’t support tool calling yet.

4

I see. Thanks for getting back!

1 Like