mnn-llm

mnn-llm是基于MNN开发的LLM推理引擎，支持目前主流的开源LLM模型。

模型导出与下载

llm模型导出onnx和mnn模型请使用llm-export

modelscope模型下载：

qwen

• modelscope-qwen-1.8b-chat

• modelscope-qwen-7b-chat

• modelscope-qwen-vl-chat

• modelscope-qwen1.5-0.5b-chat

• modelscope-qwen1.5-1.8b-chat

• modelscope-qwen1.5-4b-chat

• modelscope-qwen1.5-7b-chat

• modelscope-qwen2-0.5b-instruct

• modelscope-qwen2-1.5b-instruct

• modelscope-qwen2-7b-instruct

• modelscope-qwen2-vl-2b-instruct

• modelscope-qwen2-vl-7b-instruct

• modelscope-qwen2.5-0.5b-instruct

• modelscope-qwen2.5-1.5b-instruct

• modelscope-qwen2.5-3b-instruct

• modelscope-qwen2.5-7b-instruct

• modelscope-qwen2.5-coder-1.5b-instruct

• modelscope-qwen2.5-coder-7b-instruct

• modelscope-qwen2.5-math-1.5b-instruct

• modelscope-qwen2.5-math-7b-instruct

• modelscope-reader-lm-0.5b

• modelscope-reader-lm-1.5b

glm

• modelscope-chatglm-6b

• modelscope-chatglm2-6b

• modelscope-codegeex2-6b

• modelscope-chatglm3-6b

• modelscope-glm4-9b-chat

llama

• modelscope-llama2-7b-chat

• modelscope-llama3-8b-instruct

• modelscope-llama3.2-1b-instruct

• modelscope-llama3.2-3b-instruct

• modelscope-baichuan2-7b-chat

• modelscope-internlm-chat-7b

• modelscope-yi-6b-chat

• modelscope-deepseek-7b-chat

• modelscope-tinyllama-1.1b-chat

phi

• modelscope-phi-2

embedding

• modelscope-bge-large-zh

• modelscope-gte_sentence-embedding_multilingual-base

构建

CI构建状态：

4. 执行

# linux/macos
./cli_demo ./Qwen2-1.5B-Instruct-MNN/config.json # cli demo
./web_demo ./Qwen2-1.5B-Instruct-MNN/config.json ../web # web ui demo

# windows
.\Debug\cli_demo.exe ./Qwen2-1.5B-Instruct-MNN/config.json
.\Debug\web_demo.exe ./Qwen2-1.5B-Instruct-MNN/config.json ../web

# android
adb push libs/*.so build/libllm.so build/cli_demo /data/local/tmp
adb push model_dir /data/local/tmp
adb shell "cd /data/local/tmp && export LD_LIBRARY_PATH=. && ./cli_demo ./Qwen2-1.5B-Instruct-MNN/config.json"