{"cells":[{"cell_type":"markdown","id":"4039e3e3-fec2-46f2-aa36-a50efc753c6e","metadata":{"execution":{"iopub.execute_input":"2025-12-19T08:13:02.672802Z","iopub.status.busy":"2025-12-19T08:13:02.672254Z","iopub.status.idle":"2025-12-19T08:13:02.678899Z","shell.execute_reply":"2025-12-19T08:13:02.677905Z","shell.execute_reply.started":"2025-12-19T08:13:02.672774Z"},"tags":[]},"source":["

One-click deployment of FunctionGemma-270m-it

$\"$

This Notebook provides a comprehensive tutorial for using the Functiongemma-270m-it model, including detailed instructions on model loading, input processing, inference implementation, and the construction of the openwebui interface.

"]},{"cell_type":"markdown","id":"334f6c31-411f-4a55-a41e-aba5f4a2fde5","metadata":{"execution":{"iopub.status.busy":"2025-12-19T08:12:53.166723Z","iopub.status.idle":"2025-12-19T08:12:53.167055Z","shell.execute_reply":"2025-12-19T08:12:53.166900Z","shell.execute_reply.started":"2025-12-19T08:12:53.166883Z"}},"source":["

1.Introduction

"]},{"cell_type":"markdown","id":"7cdea176-0352-4ed1-8b3d-616b7549d172","metadata":{},"source":["

1.1 Overview of FunctionGemma-270m-it

FunctionGemma-270m-it is a lightweight function call-specific model with 270 million parameters, released by Google DeepMind in December 2025. It is built based on the Gemma 3 270M architecture and trained using the same research techniques as the Gemini series. This model is specifically designed for function call scenarios, using 6T tokens of training data (as of August 2024), including public tool definitions and tool usage interaction data. FunctionGemma supports a maximum context length of 32K tokens and has undergone strict content security filtering and responsible AI development processes.

\n
This tutorial is deployed on a single NVIDIA GeForce RTX 5090.
\n

1.2 Contents of This Tutorial

This tutorial will guide you through:

✅ Launching a model inference service with vLLM (OpenAI API-compatible)
✅ Verifying the vLLM service is running correctly
✅ Starting the OpenWebUI frontend
✅ Interacting with the model through your browser

"]},{"cell_type":"markdown","id":"3f9738c4-83f6-4ec6-af6d-be572e7457e8","metadata":{},"source":["

2.Operating steps

"]},{"cell_type":"markdown","id":"8059e13f-e689-4a80-8851-0f87ab9b0fd5","metadata":{},"source":["

2.1 Launching the Model Service with vLLM

vLLM exposes an OpenAI-compatible API; the official way to start the inference server is through vllm.entrypoints.openai.api_server.

host
\n – IP address the vLLM API server listens on.
port
\n – Port the vLLM API server listens on.
\n – Make sure the port is free and opened in your firewall/security group.
model
\n – Local path to the model weights.
\n – Usually a read-only mount such as /openbayes/input/input0/....
served-model-name
\n – Public model name clients must use in the model field of their requests.
dtype
\n – Data type for inference (e.g., float16, bfloat16) to reduce GPU memory usage.
max-model-len
\n – Maximum context length in tokens the model accepts.
\n – Must fit the model architecture and available VRAM; too large a value causes OOM.
trust-remote-code
\n – Whether to allow custom Python code from the model repo.
\n – Required for many Hugging Face repos with custom model classes.

"]},{"cell_type":"code","execution_count":1,"id":"86d3c9b0-75ca-4994-84d9-e2e882506765","metadata":{"execution":{"iopub.execute_input":"2026-01-15T14:38:59.153745Z","iopub.status.busy":"2026-01-15T14:38:59.153153Z","iopub.status.idle":"2026-01-15T14:38:59.162506Z","shell.execute_reply":"2026-01-15T14:38:59.161564Z","shell.execute_reply.started":"2026-01-15T14:38:59.153689Z"},"tags":[]},"outputs":[],"source":["import subprocess\n","import pathlib\n","import datetime\n","import os\n","import sys\n","import time"]},{"cell_type":"code","execution_count":2,"id":"b84f922c-db10-4a5d-95f7-de7a2da00090","metadata":{"execution":{"iopub.execute_input":"2026-01-15T14:39:04.464590Z","iopub.status.busy":"2026-01-15T14:39:04.464007Z","iopub.status.idle":"2026-01-15T14:40:54.620291Z","shell.execute_reply":"2026-01-15T14:40:54.619565Z","shell.execute_reply.started":"2026-01-15T14:39:04.464553Z"},"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["Start the vLLM service and write logs: /output/vllm.log\n","Start the vLLM service, log writing: vllm.log\n","The vLLM service has started, PID = 1527\n","The logs are being monitored to confirm that it is in the startup state...\n","INFO 01-15 14:39:07 [__init__.py:216] Automatically detected platform cuda.\n","\u001B[1;36m(APIServer pid=1527)\u001B[0;0m INFO 01-15 14:39:10 [api_server.py:1839] vLLM API server version 0.11.0\n","\u001B[1;36m(APIServer pid=1527)\u001B[0;0m INFO 01-15 14:39:10 [utils.py:233] non-default args: {'host': '0.0.0.0', 'port': 80, 'model': '/openbayes/input/input0/FunctionGemma-270m-it', 'trust_remote_code': True, 'served_model_name': ['functiongemma-270m-it'], 'max_num_seqs': 16}\n","\u001B[1;36m(APIServer pid=1527)\u001B[0;0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.\n","\u001B[1;36m(APIServer pid=1527)\u001B[0;0m INFO 01-15 14:39:19 [model.py:547] Resolved architecture: Gemma2ForCausalLM\n","\u001B[1;36m(APIServer pid=1527)\u001B[0;0m `torch_dtype` is deprecated! Use `dtype` instead!\n","\u001B[1;36m(APIServer pid=1527)\u001B[0;0m INFO 01-15 14:39:19 [model.py:1510] Using max model len 8192\n","\u001B[1;36m(APIServer pid=1527)\u001B[0;0m INFO 01-15 14:39:21 [scheduler.py:205] Chunked prefill is enabled with max_num_batched_tokens=2048.\n","INFO 01-15 14:39:25 [__init__.py:216] Automatically detected platform cuda.\n","\u001B[1;36m(EngineCore_DP0 pid=1794)\u001B[0;0m INFO 01-15 14:39:28 [core.py:644] Waiting for init message from front-end.\n","\u001B[1;36m(EngineCore_DP0 pid=1794)\u001B[0;0m INFO 01-15 14:39:28 [core.py:77] Initializing a V1 LLM engine (v0.11.0) with config: model='/openbayes/input/input0/FunctionGemma-270m-it', speculative_config=None, tokenizer='/openbayes/input/input0/FunctionGemma-270m-it', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=functiongemma-270m-it, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={\"level\":3,\"debug_dump_path\":\"\",\"cache_dir\":\"\",\"backend\":\"\",\"custom_ops\":[],\"splitting_ops\":[\"vllm.unified_attention\",\"vllm.unified_attention_with_output\",\"vllm.mamba_mixer2\",\"vllm.mamba_mixer\",\"vllm.short_conv\",\"vllm.linear_attention\",\"vllm.plamo2_mamba_mixer\",\"vllm.gdn_attention\",\"vllm.sparse_attn_indexer\"],\"use_inductor\":true,\"compile_sizes\":[],\"inductor_compile_config\":{\"enable_auto_functionalized_v2\":false},\"inductor_passes\":{},\"cudagraph_mode\":[2,1],\"use_cudagraph\":true,\"cudagraph_num_of_warmups\":1,\"cudagraph_capture_sizes\":[32,24,16,8,4,2,1],\"cudagraph_copy_inputs\":false,\"full_cuda_graph\":false,\"use_inductor_graph_partition\":false,\"pass_config\":{},\"max_capture_size\":32,\"local_cache_dir\":null}\n","[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n","[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n","[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n","[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n","[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n","[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n","\u001B[1;36m(EngineCore_DP0 pid=1794)\u001B[0;0m INFO 01-15 14:39:31 [parallel_state.py:1208] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0\n","\u001B[1;36m(EngineCore_DP0 pid=1794)\u001B[0;0m WARNING 01-15 14:39:31 [topk_topp_sampler.py:66] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.\n","\u001B[1;36m(EngineCore_DP0 pid=1794)\u001B[0;0m INFO 01-15 14:39:31 [gpu_model_runner.py:2602] Starting to load model /openbayes/input/input0/FunctionGemma-270m-it...\n","\u001B[1;36m(EngineCore_DP0 pid=1794)\u001B[0;0m INFO 01-15 14:39:32 [gpu_model_runner.py:2634] Loading model from scratch...\n","\u001B[1;36m(EngineCore_DP0 pid=1794)\u001B[0;0m INFO 01-15 14:39:32 [cuda.py:366] Using Flash Attention backend on V1 engine.\n","\u001B[1;36m(EngineCore_DP0 pid=1794)\u001B[0;0m\n","Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00\n

Verify whether the startup was successful. If the returned message is \"curl: (7) Failed to connect to 127.0.0.1 port 80 after 0 ms: Connection refused\", then you still need to wait.

\n"]},{"cell_type":"code","execution_count":3,"id":"ff6399a3-b105-44d7-b0c8-fc475b0c6d8f","metadata":{"execution":{"iopub.execute_input":"2026-01-15T14:41:23.623039Z","iopub.status.busy":"2026-01-15T14:41:23.622468Z","iopub.status.idle":"2026-01-15T14:41:23.873626Z","shell.execute_reply":"2026-01-15T14:41:23.872748Z","shell.execute_reply.started":"2026-01-15T14:41:23.623004Z"},"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["{\"object\":\"list\",\"data\":[{\"id\":\"functiongemma-270m-it\",\"object\":\"model\",\"created\":1768488083,\"owned_by\":\"vllm\",\"root\":\"/openbayes/input/input0/FunctionGemma-270m-it\",\"parent\":null,\"max_model_len\":8192,\"permission\":[{\"id\":\"modelperm-7384dfd37bae4d99827873f43b19caf8\",\"object\":\"model_permission\",\"created\":1768488083,\"allow_create_engine\":false,\"allow_sampling\":true,\"allow_logprobs\":true,\"allow_search_indices\":false,\"allow_view\":true,\"allow_fine_tuning\":false,\"organization\":\"*\",\"group\":null,\"is_blocking\":false}]}]}"]}],"source":["! curl http://127.0.0.1:80/v1/models"]},{"cell_type":"markdown","id":"7e216a58-422b-41e0-936e-0252d9f58a04","metadata":{},"source":["

2.2 Verify whether the vLLM service has started successfully

"]},{"cell_type":"code","execution_count":5,"id":"0786a432-5386-4cfc-a716-5ba46dc1f5a8","metadata":{"execution":{"iopub.execute_input":"2026-01-15T14:41:54.195997Z","iopub.status.busy":"2026-01-15T14:41:54.195199Z","iopub.status.idle":"2026-01-15T14:42:20.263557Z","shell.execute_reply":"2026-01-15T14:42:20.262699Z","shell.execute_reply.started":"2026-01-15T14:41:54.195974Z"},"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["You got it! \n","\n","Here's a breakdown of what makes a large language model (LLM):\n","\n","**What is a Large Language Model?**\n","\n","* **AI with a Focus on Language:** LLMs are advanced computer programs designed to understand and process human language. They're not just about translating words – they aim for fluency, context, and nuance.\n","* **Huge Datasets:** They're trained on massive amounts of text data, like books, articles, websites, code, and even social media posts. This vast data allows them to learn the complex patterns and structures of human language.\n","* **Powerful Algorithms:** LLMs use deep learning algorithms, specifically a kind called transformer networks, which excel at understanding relationships between words and phrases. These algorithms allow LLMs to \"learn\" language like a human does.\n","* **Generative Capabilities:** This is where they get interesting. LLMs can generate human-like responses, write different creative text formats (poems, code, scripts, musical pieces, email, letters, etc.), and answer your questions in an informative way.\n","\n","**Examples of LLMs:**\n","\n","* **ChatGPT (OpenAI):** Famous for its conversational abilities and ability to generate different creative text formats.\n","* **LaMDA (Google):** Focused on dialogue and engaging in natural, informative conversations.\n","* **BERT (Google):** Primarily designed for understanding language and performing tasks like search queries.\n","* **GPT-3 (OpenAI):** Known for its impressive text generation abilities, used in various applications like content creation, translation, and summarizing text.\n","\n","\n","**Key Takeaways:**\n","\n","* LLMs are still under development, but they're rapidly advancing, with the potential to transform how we interact with technology and information.\n","* Their ability to understand and generate human language is a significant step forward in AI.\n","\n","\n","Let me know if you want to learn more about specific aspects of LLMs, like their limitations, ethical considerations, or specific applications! \n","\n"]}],"source":["from openai import OpenAI\n","\n","client = OpenAI(\n"," api_key=\"EMPTY\",\n"," base_url=\"http://127.0.0.1:80/v1\"\n",")\n","\n","resp = client.chat.completions.create(\n"," model=\"functiongemma-270m-it\",\n"," messages=[\n"," {\"role\": \"user\", \"content\": \"What is a large language model? It is a type of artificial intelligence that has the ability to understand and generate human-like language.\"}\n"," ],\n"," temperature=0.7,\n"," max_tokens=512,\n",")\n","\n","print(resp.choices[0].message.content)\n"]},{"cell_type":"markdown","id":"6502467a-a8b8-49fb-9a34-900b074d0b31","metadata":{},"source":["

3. Launching the OpenWebUI Frontend

3.1 What OpenWebUI Does

OpenWebUI gives you a browser-based GUI that talks to the vLLM OpenAI-compatible endpoint, letting you chat with the model, call tools, etc.

3.2 Starting the Service (example snippet, line-by-line)

log_file = pathlib.Path(\"open-webui.log\")
\n Create a Path object for the log file in the current directory.
webui_env = os.environ.copy()
\n Clone the current shell environment so we don’t pollute it.
webui_env.update({ … })
\n Overlay the keys Open-WebUI needs:
\n
- HF_HUB_OFFLINE=1 – disable HF network checks
- ENABLE_OPENAI_API=True – turn on OpenAI-style API support
- DEFAULT_LOCALE=en – default UI language English
- HF_ENDPOINT=https://hf-mirror.com – use a mirror (trailing space harmless)
- ENV=prod – run in production mode
- WEBUI_AUTH=False – skip login
- WEBUI_SECRET_KEY=… – path to secret-key file
- DATA_DIR=… – persist data under /openbayes/home/openwebui-data
- OPENAI_API_BASE_URLS=http://0.0.0.0:80/v1 – point to the local vLLM endpoint (trailing space harmless)
- OPENAI_API_KEYS=Empty – no API key required locally
- ENABLE_RAG_WEB_SEARCH=True – enable web-search plug-in
- ENABLE_SEARCH_QUERY_GENERATION=True – let the model auto-generate search queries
- RAG_WEB_SEARCH_RESULT_COUNT=6 – return 6 search hits
- RAG_WEB_SEARCH_ENGINE=duckduckgo – use DuckDuckGo as the engine
webui_cmd = [\"/input1/py12/bin/open-webui\", \"serve\", \"--port\", \"8080\"]
\n Build the launch command; binary lives in /input1/py12/bin/open-webui, listen on 8080.
print(\"Start Open WebUI service and write logs:\", log_file.resolve())
\n Tell the user where the log will be written (resolve gives the absolute path).
with log_file.open(\"wb\") as f:
\n Open the log file in binary write mode; with guarantees it’s closed afterward.
proc = subprocess.Popen(webui_cmd, env=webui_env, stdout=f, stderr=subprocess.STDOUT)
\n Fire off the background process:
\n
- run webui_cmd
- use the customized environment
- merge stdout & stderr into the same log file
  \n The call returns immediately; the service is now running.
print(\"Open WebUI service started, PID =\", proc.pid)
\n Print the process ID so you can manage (kill/monitor) it later.

\n
Click on the API address on the right to access.
\n

"]},{"cell_type":"code","execution_count":6,"id":"38892231-c267-44d4-a624-0164754af3c8","metadata":{"execution":{"iopub.execute_input":"2026-01-15T14:43:13.690271Z","iopub.status.busy":"2026-01-15T14:43:13.688983Z","iopub.status.idle":"2026-01-15T14:43:13.728356Z","shell.execute_reply":"2026-01-15T14:43:13.727451Z","shell.execute_reply.started":"2026-01-15T14:43:13.690231Z"},"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["Start Open WebUI service and write logs: /output/open-webui.log\n","Open WebUI service started, PID = 2552\n"]}],"source":["log_file = pathlib.Path(\"open-webui.log\")\n","\n","\n","webui_env = os.environ.copy()\n","webui_env.update({\n"," \"HF_HUB_OFFLINE\": \"1\",\n"," \"ENABLE_OPENAI_API\": \"True\",\n"," \"DEFAULT_LOCALE\": \"en\",\n"," \"HF_ENDPOINT\": \"https://hf-mirror.com\",\n"," \"ENV\": \"prod\",\n"," \"WEBUI_AUTH\": \"False\",\n"," \"WEBUI_SECRET_KEY\": \"/openbayes/home/.webui_secret_key\",\n"," \"DATA_DIR\": \"/openbayes/home/openwebui-data\",\n"," \"OPENAI_API_BASE_URLS\": \"http://0.0.0.0:80/v1\",\n"," \"OPENAI_API_KEYS\": \"Empty\",\n"," \"ENABLE_RAG_WEB_SEARCH\": \"True\",\n"," \"ENABLE_SEARCH_QUERY_GENERATION\": \"True\",\n"," \"RAG_WEB_SEARCH_RESULT_COUNT\": \"6\",\n"," \"RAG_WEB_SEARCH_ENGINE\": \"duckduckgo\"\n","})\n","\n","webui_cmd = [\"/input1/py12/bin/open-webui\", \"serve\", \"--port\", \"8080\"]\n","\n","\n","print(\"Start Open WebUI service and write logs:\", log_file.resolve())\n","\n","\n","with log_file.open(\"wb\") as f:\n"," proc = subprocess.Popen(\n"," webui_cmd,\n"," env=webui_env,\n"," stdout=f,\n"," stderr=subprocess.STDOUT\n"," )\n","\n","print(\"Open WebUI service started, PID =\", proc.pid)"]}],"metadata":{"kernelspec":{"display_name":"Python 3 (ipykernel)","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.18"},"toc-autonumbering":false,"toc-showcode":true,"toc-showmarkdowntxt":false},"nbformat":4,"nbformat_minor":5}

Command Palette

vLLM+Open WebUI 導入機能Gemma-270m-it

AIでAIを構築

HyperAI Newsletters

Command Palette

vLLM+Open WebUI 導入機能Gemma-270m-it

関連ノートブック

vLLM + Open WebUIを使用してApril-1.5-15b-Thinkerをデプロイする

vLLM+Open WebUI を使用した QwenLong-L1.5 のデプロイ

vLLM+Open WebUIを使用してIQuest-Coder-V1をデプロイする

vLLM+Open WebUI が Nemotron-3 Nano を導入

vLLM+Open WebUIを使用したLFM2.5-1.2B-Thinkingの導入

vLLM+Open WebUIを使用してGLM-4.7-Flashを展開する

vLLM-Omniを使用してQwen-Image-2512をデプロイする

vLLM-Omniを使用したQwen-Image-Editの導入

PaddleOCR-VL-1.5: vLLM ベースのローカル OCR

Qwen3-Coder-Next を使用した vLLM+Open WebUI の導入

AIでAIを構築

HyperAI Newsletters

Command Palette

vLLM+Open WebUI 導入機能Gemma-270m-it

関連ノートブック

vLLM + Open WebUIを使用してApril-1.5-15b-Thinkerをデプロイする

vLLM+Open WebUI を使用した QwenLong-L1.5 のデプロイ

vLLM+Open WebUIを使用してIQuest-Coder-V1をデプロイする

vLLM+Open WebUI が Nemotron-3 Nano を導入

vLLM+Open WebUIを使用したLFM2.5-1.2B-Thinkingの導入

vLLM+Open WebUIを使用してGLM-4.7-Flashを展開する

vLLM-Omniを使用してQwen-Image-2512をデプロイする

vLLM-Omniを使用したQwen-Image-Editの導入

PaddleOCR-VL-1.5: vLLM ベースのローカル OCR

Qwen3-Coder-Next を使用した vLLM+Open WebUI の導入

AIでAIを構築

HyperAI Newsletters

関連ノートブック

vLLM + Open WebUIを使用してApril-1.5-15b-Thinkerをデプロイする

vLLM+Open WebUI を使用した QwenLong-L1.5 のデプロイ

vLLM+Open WebUIを使用してIQuest-Coder-V1をデプロイする

vLLM+Open WebUI が Nemotron-3 Nano を導入

vLLM+Open WebUIを使用したLFM2.5-1.2B-Thinkingの導入

vLLM+Open WebUIを使用してGLM-4.7-Flashを展開する

vLLM-Omniを使用してQwen-Image-2512をデプロイする

vLLM-Omniを使用したQwen-Image-Editの導入

PaddleOCR-VL-1.5: vLLM ベースのローカル OCR

Qwen3-Coder-Next を使用した vLLM+Open WebUI の導入

関連ノートブック

vLLM + Open WebUIを使用してApril-1.5-15b-Thinkerをデプロイする

vLLM+Open WebUI を使用した QwenLong-L1.5 のデプロイ

vLLM+Open WebUIを使用してIQuest-Coder-V1をデプロイする

vLLM+Open WebUI が Nemotron-3 Nano を導入

vLLM+Open WebUIを使用したLFM2.5-1.2B-Thinkingの導入

vLLM+Open WebUIを使用してGLM-4.7-Flashを展開する

vLLM-Omniを使用してQwen-Image-2512をデプロイする

vLLM-Omniを使用したQwen-Image-Editの導入

PaddleOCR-VL-1.5: vLLM ベースのローカル OCR

Qwen3-Coder-Next を使用した vLLM+Open WebUI の導入