One-click Deployment of Mistral Large 2407 123B Model OpenAI Compatible API Service

Tutorial and Model Introduction

This tutorial is to deploy Mistral-Large-Instruct-2407-AWQ using the OpenAI compatible API.

* Video tutorials can be found in[OpenBayes official tutorial] Quick deployment of Mistral-Large & Llama-3.1-405B super large models"

Mistral-Large-Instruct-2407-AWQ is the new generation flagship AI model released by French artificial intelligence company Mistral AI, namely Mistral Large 2. The model has 123B parameters, a 128k context window, and supports dozens of languages including Chinese and more than 80 programming languages such as Python, Java, C, C++, JavaScript, and Bash. Mistral Large 2 has significantly improved multilingual capabilities, mathematics, reasoning, and code writing, and has achieved an accuracy of 84.0% on MMLU.

"OpenAI-compatible API" refers to an application programming interface (API) that follows the interface standards and specifications set by OpenAI, allowing developers to use these APIs to interact with large language models (such as OpenAI's GPT series models). This compatibility means that third-party developers can use the same request and response format as OpenAI to integrate similar functionality into their own applications. For example, if a developer builds a chatbot using OpenAI's API, they can easily switch to another service that also follows the OpenAI compatible API standard without making significant changes to their code.

Key features of the OpenAI-compatible API include:

• Standardized requests: API requests follow OpenAI's format, including required parameters and structure.
• Standardized responses: API responses also follow OpenAI’s format, making processing and parsing results consistent and predictable.
• Functionality consistency: Provides similar functionality to OpenAI, such as text generation, translation, summarization, etc.
• Easy to integrate: Developers can easily integrate these APIs into existing systems, leveraging familiar interfaces and patterns.

Run steps

1. Clone and start the container in the upper right corner of the tutorial interface

The OpenAI-compatible API will automatically start all services after successful deployment, without any additional intervention. 

2. Copy the API address to a new page and open it

You can see that a default 404 message is displayed.

3. Add an extra parameter '/v1/models' after the API address

You can see that the deployment information of the model is displayed.

4. At this point, you can connect to the model in any OpenAI compatible SDK. Here we take OpenWebUI as an example - use the local OpenWebUI to integrate this API

Start an OpenWebUI service locally, start an additional connection in "External Connections", fill in the API in "OpenAPI" and ➕ '/v1', there is no "API key" custom input here. Click Save in the lower right corner.

5. Deployment completed

You can see that the Mistral Large 2407 123B model already exists in the OpenWebUI interface. You can communicate with the large model by simply entering a message at the bottom of the page.