Use Foundry Local native chat completions API

Prerequisites

• .NET 9.0 SDK or later installed.
• Azure role-based access control (RBAC): Not applicable.

Samples repository

You can find the sample in this article in the Foundry Local SDK Samples GitHub repository.

Set up project

Use Foundry Local in your C# project by following these Windows-specific or Cross-Platform (macOS/Linux/Windows) instructions:

Use native chat completions API

The following example demonstrates how to use the native chat completions API in Foundry Local. The code includes the following steps:

1. Initializes a FoundryLocalManager instance with a Configuration.

2. Gets a Model object from the model catalog using an alias.

   Note

   Foundry Local automatically selects the best variant for the model based on the available hardware of the host machine.

3. Downloads and loads the model variant.

4. Uses the native chat completions API to generate a response.

5. Unloads the model.

Copy and paste the following code into a C# file named Program.cs:

using Microsoft.AI.Foundry.Local;
using Betalgo.Ranul.OpenAI.ObjectModels.RequestModels;
using Microsoft.Extensions.Logging;

CancellationToken ct = CancellationToken.None;

var config = new Configuration
{
    AppName = "app-name",
    LogLevel = Microsoft.AI.Foundry.Local.LogLevel.Information
};

using var loggerFactory = LoggerFactory.Create(builder =>
{
    builder.SetMinimumLevel(Microsoft.Extensions.Logging.LogLevel.Information);
});
var logger = loggerFactory.CreateLogger<Program>();

// Initialize the singleton instance.
await FoundryLocalManager.CreateAsync(config, logger);
var mgr = FoundryLocalManager.Instance;

// Get the model catalog
var catalog = await mgr.GetCatalogAsync();

// Get a model using an alias
var model = await catalog.GetModelAsync("qwen2.5-0.5b") ?? throw new Exception("Model not found");

// Download the model (the method skips download if already cached)
await model.DownloadAsync(progress =>
{
    Console.Write($"\rDownloading model: {progress:F2}%");
    if (progress >= 100f)
    {
        Console.WriteLine();
    }
});

// Load the model
await model.LoadAsync();

// Get a chat client
var chatClient = await model.GetChatClientAsync();

// Create a chat message
List<ChatMessage> messages = new()
{
    new ChatMessage { Role = "user", Content = "Why is the sky blue?" }
};

var streamingResponse = chatClient.CompleteChatStreamingAsync(messages, ct);
await foreach (var chunk in streamingResponse)
{
    Console.Write(chunk.Choices[0].Message.Content);
    Console.Out.Flush();
}
Console.WriteLine();

// Tidy up - unload the model
await model.UnloadAsync();

References:

• Foundry Local SDK reference
• Foundry Local C# SDK API Reference
• CancellationToken

Optional: list model aliases available on your device

If you don't know which model alias to use, list the models available for your hardware.

// List available models and aliases
Console.WriteLine("Available models for your hardware:");
var models = await catalog.ListModelsAsync();
foreach (var availableModel in models)
{
    foreach (var variant in availableModel.Variants)
    {
        Console.WriteLine($"  - Alias: {variant.Alias}");
    }
}

References:

• Foundry Local SDK reference

Run the code by using the following command:

dotnet run -r:win-x64

dotnet run -r:win-arm64

dotnet run -r:osx-arm64

dotnet run -r:linux-x64

dotnet run -r:win-x64

Troubleshooting

• Build errors referencing net9.0: Install the .NET 9.0 SDK, then rebuild the app.
• Model not found: Run the optional model listing snippet to find an alias available on your device, then update the alias passed to GetModelAsync.
• Slow first run: Model downloads can take time the first time you run the app.

Prerequisites

• Node.js 20 or later installed.

Samples repository

You can find the sample in this article in the Foundry Local SDK Samples GitHub repository.

Set up project

Use Foundry Local in your JavaScript project by following these Windows-specific or Cross-Platform (macOS/Linux/Windows) instructions:

Use native chat completions API

The following example demonstrates how to use the native chat completions API in Foundry Local. The benefit of using the native chat completions API is there's no need for a REST web server running and therefore it provides a simplified deployment. The code includes the following steps:

1. Initializes a FoundryLocalManager instance with a configuration.
2. Gets a Model object from the model catalog using an alias.
3. Downloads and loads the model variant.
4. Uses the native chat completions API to generate a response.
5. Unloads the model.

Copy and paste the following code into a JavaScript file named app.js:

import { FoundryLocalManager } from 'foundry-local-sdk';

// Initialize the Foundry Local SDK
console.log('Initializing Foundry Local SDK...');

const manager = FoundryLocalManager.create({
    appName: 'foundry_local_samples',
    logLevel: 'info'
});
console.log('✓ SDK initialized successfully');

// Get the model object
const modelAlias = 'qwen2.5-0.5b'; // Using an available model from the list above
const model = await manager.catalog.getModel(modelAlias);

// Download the model
console.log(`\nDownloading model ${modelAlias}...`);
await model.download((progress) => {
    process.stdout.write(`\rDownloading... ${progress.toFixed(2)}%`);
});
console.log('\n✓ Model downloaded');

// Load the model
console.log(`\nLoading model ${modelAlias}...`);
await model.load();
console.log('✓ Model loaded');

// Create chat client
console.log('\nCreating chat client...');
const chatClient = model.createChatClient();
console.log('✓ Chat client created');

// Example chat completion
console.log('\nTesting chat completion...');
const completion = await chatClient.completeChat([
    { role: 'user', content: 'Why is the sky blue?' }
]);

console.log('\nChat completion result:');
console.log(completion.choices[0]?.message?.content);

// Example streaming completion
console.log('\nTesting streaming completion...');
await chatClient.completeStreamingChat(
    [{ role: 'user', content: 'Write a short poem about programming.' }],
    (chunk) => {
        const content = chunk.choices?.[0]?.message?.content;
        if (content) {
            process.stdout.write(content);
        }
    }
);
console.log('\n');

// Unload the model
console.log('Unloading model...');
await model.unload();
console.log(`✓ Model unloaded`);

Run the code

Run the code by using the following command:

node app.js