ChatOllama

This will help you getting started with ChatOllama chat models. For detailed documentation of all ChatOllama features and configurations head to the API reference.

Overview

Integration details

Ollama allows you to run open-source large language models, such as Llama 2, locally.

Ollama bundles model weights, configuration, and data into a single package, defined by a Modelfile. It optimizes setup and configuration details, including GPU usage.

This example goes over how to use LangChain to interact with an Ollama-run Llama 2 7b instance as a chat model. For a complete list of supported models and model variants, see the Ollama model library.

Class	Package	Local	Serializable	PY support	Package downloads	Package latest
ChatOllama	@langchain/ollama	✅	beta	✅

Model features

Tool calling	Structured output	JSON mode	Image input	Audio input	Video input	Token-level streaming	Token usage	Logprobs
✅	✅	✅	✅	❌	❌	✅	✅	❌

Setup

Follow these instructions to set up and run a local Ollama instance. Then, download the @langchain/ollama package.

Credentials

If you want to get automated tracing of your model calls you can also set your LangSmith API key by uncommenting below:

# export LANGCHAIN_TRACING_V2="true"
# export LANGCHAIN_API_KEY="your-api-key"

Installation

The LangChain ChatOllama integration lives in the @langchain/ollama package:

tip

See this section for general instructions on installing integration packages.

npm
yarn
pnpm

npm i @langchain/ollama

yarn add @langchain/ollama

pnpm add @langchain/ollama

Instantiation

Now we can instantiate our model object and generate chat completions:

TODO: Update model instantiation with relevant params.

import { ChatOllama } from "@langchain/ollama";

const llm = new ChatOllama({
  model: "llama3",
  temperature: 0,
  maxRetries: 2,
  // other params...
});

Invocation

TODO: Run cells so output can be seen.

const aiMsg = await llm.invoke([
  [
    "system",
    "You are a helpful assistant that translates English to French. Translate the user sentence.",
  ],
  ["human", "I love programming."],
]);
aiMsg;

AIMessage {
  "content": "Je adore le programmation.\n\n(Note: \"programmation\" is the feminine form of the noun in French, but if you want to use the masculine form, it would be \"le programme\" instead.)",
  "additional_kwargs": {},
  "response_metadata": {
    "model": "llama3",
    "created_at": "2024-08-01T16:59:17.359302Z",
    "done_reason": "stop",
    "done": true,
    "total_duration": 6399311167,
    "load_duration": 5575776417,
    "prompt_eval_count": 35,
    "prompt_eval_duration": 110053000,
    "eval_count": 43,
    "eval_duration": 711744000
  },
  "tool_calls": [],
  "invalid_tool_calls": [],
  "usage_metadata": {
    "input_tokens": 35,
    "output_tokens": 43,
    "total_tokens": 78
  }
}

console.log(aiMsg.content);

Je adore le programmation.

(Note: "programmation" is the feminine form of the noun in French, but if you want to use the masculine form, it would be "le programme" instead.)

Chaining

We can chain our model with a prompt template like so:

TODO: Run cells so output can be seen.

import { ChatPromptTemplate } from "@langchain/core/prompts";

const prompt = ChatPromptTemplate.fromMessages([
  [
    "system",
    "You are a helpful assistant that translates {input_language} to {output_language}.",
  ],
  ["human", "{input}"],
]);

const chain = prompt.pipe(llm);
await chain.invoke({
  input_language: "English",
  output_language: "German",
  input: "I love programming.",
});

AIMessage {
  "content": "Ich liebe Programmieren!\n\n(Note: \"Ich liebe\" means \"I love\", \"Programmieren\" is the verb for \"programming\")",
  "additional_kwargs": {},
  "response_metadata": {
    "model": "llama3",
    "created_at": "2024-08-01T16:59:18.088423Z",
    "done_reason": "stop",
    "done": true,
    "total_duration": 585146125,
    "load_duration": 27557166,
    "prompt_eval_count": 30,
    "prompt_eval_duration": 74241000,
    "eval_count": 29,
    "eval_duration": 481195000
  },
  "tool_calls": [],
  "invalid_tool_calls": [],
  "usage_metadata": {
    "input_tokens": 30,
    "output_tokens": 29,
    "total_tokens": 59
  }
}

Tools

Ollama now offers support for native tool calling. The example below demonstrates how you can invoke a tool from an Ollama model.

import { tool } from "@langchain/core/tools";
import { ChatOllama } from "@langchain/ollama";
import { z } from "zod";

const weatherTool = tool((_) => "Da weather is weatherin", {
  name: "get_current_weather",
  description: "Get the current weather in a given location",
  schema: z.object({
    location: z.string().describe("The city and state, e.g. San Francisco, CA"),
  }),
});

// Define the model
const llmForTool = new ChatOllama({
  model: "llama3-groq-tool-use",
});

// Bind the tool to the model
const llmWithTools = llmForTool.bindTools([weatherTool]);

const resultFromTool = await llmWithTools.invoke(
  "What's the weather like today in San Francisco? Ensure you use the 'get_current_weather' tool."
);

console.log(resultFromTool);

ResponseError: model "llama3-groq-tool-use" not found, try pulling it first
    at checkOk (/Users/bracesproul/code/lang-chain-ai/langchainjs/node_modules/ollama/dist/shared/ollama.09afdd6e.cjs:72:9)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async post (/Users/bracesproul/code/lang-chain-ai/langchainjs/node_modules/ollama/dist/shared/ollama.09afdd6e.cjs:119:3)
    at async Ollama.processStreamableRequest (/Users/bracesproul/code/lang-chain-ai/langchainjs/node_modules/ollama/dist/shared/ollama.09afdd6e.cjs:251:22)
    at async ChatOllama._streamResponseChunks (/Users/bracesproul/code/lang-chain-ai/langchainjs/libs/langchain-ollama/dist/chat_models.cjs:455:32)
    at async ChatOllama._generate (/Users/bracesproul/code/lang-chain-ai/langchainjs/libs/langchain-ollama/dist/chat_models.cjs:407:26)
    at async Promise.allSettled (index 0)
    at async ChatOllama._generateUncached (/Users/bracesproul/code/lang-chain-ai/langchainjs/langchain-core/dist/language_models/chat_models.cjs:177:29)
    at async ChatOllama.invoke (/Users/bracesproul/code/lang-chain-ai/langchainjs/langchain-core/dist/language_models/chat_models.cjs:61:24)
    at async evalmachine.<anonymous>:25:24 {
  error: 'model "llama3-groq-tool-use" not found, try pulling it first',
  status_code: 404
}

`.withStructuredOutput`

Since ChatOllama supports the .bindTools() method, you can also call .withStructuredOutput() to get a structured output from the tool.

import { ChatOllama } from "@langchain/ollama";
import { z } from "zod";

// Define the model
const llmForWSO = new ChatOllama({
  model: "llama3-groq-tool-use",
});

// Define the tool schema you'd like the model to use.
const schemaForWSO = z.object({
  location: z.string().describe("The city and state, e.g. San Francisco, CA"),
});

// Pass the schema to the withStructuredOutput method to bind it to the model.
const llmWithStructuredOutput = llmForWSO.withStructuredOutput(schemaForWSO, {
  name: "get_current_weather",
});

const resultFromWSO = await llmWithStructuredOutput.invoke(
  "What's the weather like today in San Francisco? Ensure you use the 'get_current_weather' tool."
);
console.log(resultFromWSO);

JSON mode

Ollama also supports a JSON mode that coerces model outputs to only return JSON. Here’s an example of how this can be useful for extraction:

import { ChatOllama } from "@langchain/ollama";
import { ChatPromptTemplate } from "@langchain/core/prompts";

const promptForJsonMode = ChatPromptTemplate.fromMessages([
  [
    "system",
    `You are an expert translator. Format all responses as JSON objects with two keys: "original" and "translated".`,
  ],
  ["human", `Translate "{input}" into {language}.`],
]);

const llmJsonMode = new ChatOllama({
  baseUrl: "http://localhost:11434", // Default value
  model: "llama3",
  format: "json",
});

const chainForJsonMode = promptForJsonMode.pipe(llmJsonMode);

const resultFromJsonMode = await chainForJsonMode.invoke({
  input: "I love programming",
  language: "German",
});

console.log(resultFromJsonMode);

Multimodal models

Ollama supports open source multimodal models like LLaVA in versions 0.1.15 and up. You can pass images as part of a message’s content field to multimodal-capable models like this:

import { ChatOllama } from "@langchain/ollama";
import { HumanMessage } from "@langchain/core/messages";
import * as fs from "node:fs/promises";

const imageData = await fs.readFile("../../../../../examples/hotdog.jpg");
const llmForMultiModal = new ChatOllama({
  model: "llava",
  baseUrl: "http://127.0.0.1:11434",
});
const multiModalRes = await llmForMultiModal.invoke([
  new HumanMessage({
    content: [
      {
        type: "text",
        text: "What is in this image?",
      },
      {
        type: "image_url",
        image_url: `data:image/jpeg;base64,${imageData.toString("base64")}`,
      },
    ],
  }),
]);
console.log(multiModalRes);

ResponseError: model "llava" not found, try pulling it first
    at checkOk (/Users/bracesproul/code/lang-chain-ai/langchainjs/node_modules/ollama/dist/shared/ollama.09afdd6e.cjs:72:9)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async post (/Users/bracesproul/code/lang-chain-ai/langchainjs/node_modules/ollama/dist/shared/ollama.09afdd6e.cjs:119:3)
    at async Ollama.processStreamableRequest (/Users/bracesproul/code/lang-chain-ai/langchainjs/node_modules/ollama/dist/shared/ollama.09afdd6e.cjs:231:25)
    at async ChatOllama._streamResponseChunks (/Users/bracesproul/code/lang-chain-ai/langchainjs/libs/langchain-ollama/dist/chat_models.cjs:474:24)
    at async ChatOllama._generate (/Users/bracesproul/code/lang-chain-ai/langchainjs/libs/langchain-ollama/dist/chat_models.cjs:407:26)
    at async Promise.allSettled (index 0)
    at async ChatOllama._generateUncached (/Users/bracesproul/code/lang-chain-ai/langchainjs/langchain-core/dist/language_models/chat_models.cjs:177:29)
    at async ChatOllama.invoke (/Users/bracesproul/code/lang-chain-ai/langchainjs/langchain-core/dist/language_models/chat_models.cjs:61:24)
    at async evalmachine.<anonymous>:39:23 {
  error: 'model "llava" not found, try pulling it first',
  status_code: 404
}

API reference

For detailed documentation of all ChatOllama features and configurations head to the API reference: https://api.js.langchain.com/classes/langchain_ollama.ChatOllama.html

ChatOllama

Overview

Integration details

Model features

Setup

Credentials

Installation

Instantiation

Invocation

Chaining

Tools

`.withStructuredOutput`

JSON mode

Multimodal models

API reference

Was this page helpful?

You can also leave detailed feedback on GitHub.

Overview​

Integration details​

Model features​

Setup​

Credentials​

Installation​

Instantiation​

Invocation​

Chaining​

Tools​

.withStructuredOutput​

JSON mode​

Multimodal models​

API reference​

Was this page helpful?

You can also leave detailed feedback on GitHub.

Overview

Integration details

Model features

Setup

Credentials

Installation

Instantiation

Invocation

Chaining

Tools

`.withStructuredOutput`

JSON mode

Multimodal models

API reference