How to Use Function Calling in AI Agents (Step by Step)

Building AI agents that can actually do things beyond generating text requires a mechanism called function calling. Since OpenAI launched its dedicated function-calling API in June 2023, developers have used this feature to make large language models (LLMs) query databases, send emails, book flights, and control IoT devices. Without function calling, an AI agent is reactive and confined to chat. With it, an agent becomes active, autonomous, and capable of completing multi-step tasks in the real world. This guide walks you through exactly how to implement function calling in AI agents step by step, using real code patterns and production-tested strategies. Whether you are building a customer support bot or a code-generation pipeline, these principles apply across every major LLM provider.

Quick Answer: Function calling lets an LLM like GPT-4 request external tools by outputting structured JSON instead of plain text. You define functions as JSON schemas, pass them into the API call, and when the model decides it needs a tool, it returns a function name and arguments. Your code executes the function and sends the result back to the model for a final response. This loop turns a chat model into an agent.

What Is Function Calling and Why Do AI Agents Need It

The core mechanism behind tool-use

Function calling solves a fundamental limitation of LLMs: they cannot access real-time data, perform calculations reliably, or trigger external systems. The transformer architecture introduced in the 2017 paper "Attention Is All You Need" gave us powerful next-token predictors, but those predictors hallucinate facts and have no built-in ability to call APIs. Function calling bridges this gap. When you send a list of function definitions to the model alongside a user message, the model can respond with a structured JSON object that specifies which function to call and what arguments to pass. Your application code then executes that function and returns the result to the model. This feedback loop gives the agent perception and action capabilities without retraining the model.

Why 2023 was the inflection point

Before June 2023, developers had to parse model output with regex or prompt-engineering tricks to extract tool calls — a fragile approach. OpenAI's dedicated function-calling API changed that by guaranteeing structured output. According to Wikipedia's analysis of AI agent history, "Deployment of such agents began to accelerate in late 2023 after OpenAI's 'function-calling' API was made available." Anthropic followed in late 2024 with the Model Context Protocol (MCP), standardizing how agents gain contextual awareness and call tools. In December 2025, the Linux Foundation formed the Agentic AI Foundation (AAIF) to ensure transparent evolution of these systems.

Real example: A weather agent that retrieves live data

A user asks "What is the temperature in Tokyo right now?" Without function calling, the model guesses and says "Tokyo is around 22°C." With function calling, the model returns {"function": "get_weather", "args": {"city": "Tokyo"}}. Your code calls a real weather API, gets {"temperature": 18, "unit": "celsius"}, sends that back, and the model replies: "Tokyo is currently 18°C and partly cloudy." The agent delivered a factual, real-time answer.

How to Define Functions for an LLM Agent

Function schema design in JSON format

Every function you expose to an AI agent must be described as a JSON schema. This schema tells the model the function's name, description, and expected parameters. The description field is critical — the model uses it to decide when to call the function. Write descriptions that are clear and action-oriented. Here is the standard structure used by OpenAI, Anthropic Claude, and Google Gemini:

• name: A snake_case identifier like "search_database"
• description: A plain-English explanation of what the function does
• parameters: A JSON Schema object listing each parameter's name, type, and description
• required: An array of parameter names the model must provide

Writing descriptions that improve model accuracy

The LLM reads your function descriptions during inference. If you write a vague description like "Gets data from the system," the model may call the wrong function or omit arguments. Instead, write: "Retrieves customer account details including balance, status, and last transaction date by passing the customer_id string." Include edge-case guidance: "Returns null if no account is found." This reduces hallucinated calls and improves first-attempt accuracy from around 65% to over 90% in production systems.

Real example: E-commerce inventory lookup

For a shopping assistant agent, define: {"name":"check_inventory","description":"Returns current stock count for a product SKU. Use this when a user asks if an item is in stock. Returns integer quantity or -1 if SKU not found.","parameters":{"type":"object","properties":{"sku":{"type":"string","description":"The 8-character product SKU code"}},"required":["sku"]}}. When a user asks "Do you have the blue running shoes in size 10?", the model will extract the SKU from your catalog and return the correct call.

Implementing the Function Calling Loop Step by Step

Step 1: Define your tools array

1. Create a Python dictionary (or JSON object) that lists all functions your agent can call.
2. Each entry must contain type: "function", function.name, function.description, and function.parameters.
3. Keep the total number of functions under 20 per request to avoid confusing the model. Group related tools into a single function when possible.

Step 2: Make the initial API request

1. Send a message array that includes system instructions, user query, and the tools array to the LLM endpoint.
2. Set tool_choice: "auto" to let the model decide whether to call a function or respond directly.
3. For critical workflows, set tool_choice: {"type": "function", "function": {"name": "specific_tool"}} to force a specific tool call.

Step 3: Parse the response and detect tool calls

1. Check if the response contains a tool_calls field. If absent, the model gave a direct answer — return it to the user.
2. If present, extract the function.name and function.arguments (parse the JSON string).
3. Iterate through multiple tool calls if the model returns parallel calls (supported since GPT-4 Turbo in November 2023).

Step 4: Execute the function in your environment

1. Map the function name string to your actual Python/Node.js function using a dictionary lookup.
2. Pass the parsed arguments using **kwargs syntax.
3. Wrap execution in try/except blocks — if the function throws an error, return the error message to the model so it can adjust or apologize.

Step 5: Send results back to the model

1. Append the model's original response (with tool_calls) to the message history.
2. Add a new "tool" role message containing the function's output.
3. Re-send the full conversation to the model. It will generate a final answer incorporating the tool's data.

Real example: Flight booking agent

A user says "Book a flight from JFK to LHR on March 15 for 2 adults." The agent calls search_flights("JFK","LHR","2025-03-15",2), receives a list of options, shows them to the user, then calls book_flight("FL123","john.doe@email.com") after confirmation. Each call follows the same five-step loop above.

Comparing Function Calling Across LLM Providers

Not all providers implement function calling the same way. Understanding the differences helps you choose the right backend for your agent. Below is a comparison based on production use across OpenAI, Anthropic, and Google.

Feature	OpenAI (GPT-4 / GPT-4o)	Anthropic (Claude 3.5 Sonnet)
Release date	June 2023 (public beta)	Late 2024 (MCP standard)
Tool definition format	JSON Schema via tools[] array	JSON Schema via tools[] array
Parallel function calls	Supported (since Nov 2023)	Supported
Force tool choice	tool_choice: "any" or specific function name	tool_choice: {"type":"tool","name":"..."}
Context window max	128K tokens (GPT-4 Turbo)	200K tokens
Structured output guarantee	Structured Outputs mode (JSON schema enforcement)	No native JSON mode (use tool calls)
Error recovery	Returns error message as tool response; model adjusts	Returns error message; model adjusts

Both provider ecosystems continue to evolve. OpenAI released Structured Outputs in mid-2024 to enforce response schemas, while Anthropic's MCP protocol introduced a standardized client-server architecture for tool integration in late 2024.

Common Mistakes When Implementing Function Calling

Mistake 1: Overloading the tools array with too many functions

Why it hurts: LLMs lose accuracy when given more than 15-20 function options. The attention mechanism spreads across too many tokens, causing the model to pick the wrong function or forget arguments. Accuracy drops from ~85% to ~60% in benchmarks.

Fix: Limit tools to 12-15 per request. Group related actions under a single function with a discriminator parameter. For example, instead of create_user, update_user, delete_user, use manage_user with action parameter.

Mistake 2: Writing vague or missing function descriptions

Why it hurts: The LLM relies on your description text to decide when to call a function. A description like "Processes data" provides no signal. The model may call the function at the wrong time or omit it entirely.

Fix: Write descriptions that include trigger conditions, expected return values, and edge behavior. Example: "Fetches real-time stock price by ticker symbol. Use this when the user asks for current market data. Returns float value or error string if ticker is invalid."

Mistake 3: Not handling tool call errors gracefully

Why it hurts: A failed API call or database timeout will crash the agent loop if not caught. The model receives nothing and may hallucinate a fake response to fill the gap.

Fix: Wrap every function execution in try/catch and return structured error objects to the model. For example: {"error": "Rate limit exceeded. Try again in 60 seconds."} The model can then inform the user and retry.

Mistake 4: Forgetting to manage conversation history

Why it hurts: Every function call and response adds tokens to the conversation. After 5-10 rounds, you hit context limits or increase latency by 300%.

Fix: Implement a sliding window. Keep the system prompt, last user message, and last 2-3 function call cycles. Discard older tool responses that no longer affect the current state.

Mistake 5: Ignoring parallel function call ordering

Why it hurts: When the model issues parallel calls (e.g., "get weather in three cities"), it expects you to execute them concurrently but return results in the correct order. Returning results out of order confuses the next model response.

Fix: Use asyncio.gather() or Promise.all() but store results keyed by the function call id field. Return results sorted by their original index in the tool_calls array.

Pro Tips

• Always set temperature: 0 or 0.1 for function calling agents — higher temperatures increase hallucinated tool calls.
• Use tool_choice: "required" (OpenAI) when the agent must use a tool every turn, such as in code-generation agents.
• Cache function schemas server-side to reduce token overhead — send only the schema hash for repeated calls.
• Log every tool call with timestamps, arguments, and response times for debugging latency spikes and logic errors.
• Test with edge inputs: empty strings, null values, and missing required fields trigger the most agent failures.

FAQ

What exactly is function calling in the context of AI agents?

Function calling is a capability built into modern LLMs that allows the model to output structured JSON requesting the execution of a predefined external function. Instead of generating conversational text, the model outputs a function name and arguments, which your application code executes and returns as a result. This turns a static language model into a dynamic agent that can interact with databases, APIs, and hardware.

How does function calling differ from regular API calls or plugins?

Standard API calls are hardcoded endpoints your application triggers directly. Function calling is model-driven — the LLM decides when to call a tool based on the user's natural language request. Plugins (as introduced by OpenAI in March 2023) are a higher-level abstraction that wraps function calling with UI elements and manifest files. Function calling gives you more control because you manage the execution logic yourself.

Can I use function calling with open-source models like Llama 3?

Yes. Open-source models from Meta (Llama 3.1 and 3.2), Mistral, and others support function calling via compatible tool-use fine-tunes. However, accuracy varies. Llama 3.1 70B achieves roughly 85% tool-call accuracy compared to GPT-4's 95% in independent benchmarks. Libraries like Ollama and vLLM provide function-calling endpoints compatible with the OpenAI API format, so your code can switch providers with minimal changes.

What happens if the AI model calls a function with incorrect arguments?

The model passes the arguments as parsed JSON. If arguments are missing or malformed, your application code should validate them before execution. Return a clear error message to the model: "Error: parameter 'email' is missing. Please provide a valid email address." The model will then correct its call and retry. This retry loop usually resolves within 1-2 attempts if your function descriptions are well written.

Will function calling replace traditional programming or APIs entirely?

No. Function calling augments traditional programming — you still write and deploy the API endpoints behind each function. The LLM becomes a smart router that decides which endpoint to call based on user intent. According to the Linux Foundation's Agentic AI Foundation (formed December 2025), agentic AI systems will evolve alongside traditional software, not replace it. Function calling handles the orchestration layer; the underlying business logic remains hand-coded.

Conclusion

Function calling is the single most important feature for turning a language model into an autonomous AI agent. Since OpenAI released its dedicated API in June 2023, the ability to parse model output into executable tool calls has moved from experimental hacks to production-standard implementations. You define functions as JSON schemas, let the model decide when and how to call them, execute the logic, and feed results back into the conversation loop. This guide covered the full implementation — from schema design and provider comparison to error handling and parallel execution. The mistakes section highlighted the five biggest pitfalls, and the pro tips gave you battle-tested practices for production deployments.

• Define functions with clear descriptions and limit the tools array to 12-15 entries per request.
• Implement the five-step loop: send, parse, execute, return, and re-send for every tool interaction.
• Use temperature 0.1 or lower for function calling to reduce hallucinated tool requests.
• Handle errors with structured return messages so the model can self-correct in the next turn.

Sources

• Wikipedia: AI Agent
• Wikipedia: Large Language Model
• OpenAI: Function Calling Guide
• Anthropic: Tool Use Documentation
• Vaswani et al.: "Attention Is All You Need" (2017)