一、GME-Qwen2-VL-7B 模型GME-Qwen2VL系列是统一的多模态Embedding模型基于Qwen2-VL训练支持动态分辨率。模型支持三种类型的输入文本、图像、图像-文本对所有输入类型都可以生成通用的向量表示并具有优秀的检索性能。使知识向量化不再局限于文本。基于该模型可以实现文搜文、文搜图图搜文图搜图等丰富的场景。GME-Qwen2-VL-7BModelScope地址https://modelscope.cn/models/iic/gme-Qwen2-VL-7B-Instruct本文基于GME-Qwen2-VL-7B模型本地化部署并实现文搜图案例效果如下所示二、GME-Qwen2-VL-7B 部署下载模型modelscope download--modeliic/gme-Qwen2-VL-7B-Instruct--local_dirgme-Qwen2-VL-7B-Instruct然后将下载后gme_inference.py和下面的服务代码放在一起读取模型并启动api服务importtimefromfastapiimportFastAPI,Requestfromfastapi.middleware.corsimportCORSMiddlewareimportuvicorn,jsonfromgme_inferenceimportGmeQwen2VLimporttorchimportbase64fromioimportBytesIOfromPILimportImage appFastAPI()app.add_middleware(CORSMiddleware,allow_origins[*],allow_credentialsTrue,allow_methods[*],allow_headers[*],)model_pathgme-Qwen2-VL-7B-InstructgmeGmeQwen2VL(model_pathmodel_path,devicecuda:0)defbase64_to_image(image_base64:str):image_database64.b64decode(image_base64)image_fileBytesIO(image_data)imageImage.open(image_file)returnimageapp.post(/v1/embeddings)asyncdefembeddings(request:Request):globalmodel,tokenizer json_post_rawawaitrequest.json()json_postjson.dumps(json_post_raw)messagesjson.loads(json_post)textsmessages.get(texts)imagesmessages.get(images)ifnottextsandnotimages:return{code:400,message:texts 和 images 至少传一个,}ttime.time()ifimages:images[base64_to_image(b)forbinimagesifb]iftextsandimages:embedsgme.get_fused_embeddings(textstexts,imagesimages).tolist()eliftexts:embedsgme.get_text_embeddings(textstexts).tolist()else:embedsgme.get_image_embeddings(imagesimages).tolist()use_timetime.time()-tiftorch.backends.mps.is_available():torch.mps.empty_cache()return{code:200,message:success,data:embeds,use_time:use_time}if__name____main__:uvicorn.run(app,host0.0.0.0,port8848,workers1)启动后大概占用17.5G显存。三、API 调用示例importbase64importrequestsdefencode_image(image_path):withopen(image_path,rb)asimage_file:returnbase64.b64encode(image_file.read()).decode(utf-8)defembeds(texts:[]None,images:[]None):ifnottextsandnotimages:raiseException(embed content is empty)ifimages:images[encode_image(p)forpinimagesifp]responserequests.post(http://127.0.0.1:8848/v1/embeddings,json{texts:texts,images:images})returnresponse.json()[data]defmain():texts[你好呀]images[img/1.png]print(embeds(textstexts,imagesimages))if__name____main__:main()调用结果向量维度为3584维。四、实现 文搜图 案例这里我准备了一些 猫、狗的图片通过GME-Qwen2-VL-7B模型向量化并持久化到Milvus向量库中 。importjsonimportosimportbase64importrequestsfrompymilvusimportMilvusClient,DataType clientMilvusClient(http://127.0.0.1:19530)collection_namegme_vl_testdefcreate_collection():client.drop_collection(collection_namecollection_name)schemaMilvusClient.create_schema(auto_idFalse,enable_dynamic_fieldFalse,)schema.add_field(field_nameid,datatypeDataType.VARCHAR,is_primaryTrue,max_length255)schema.add_field(field_namevector,datatypeDataType.FLOAT_VECTOR,dim3584)schema.add_field(field_namecontent,datatypeDataType.VARCHAR,max_length5000)schema.verify()index_paramsclient.prepare_index_params()index_params.add_index(field_namevector,index_typeIVF_FLAT,metric_typeL2,params{nlist:1024})# 创建 collectionclient.create_collection(collection_namecollection_name,schemaschema,index_paramsindex_params)defencode_image(image_path):withopen(image_path,rb)asimage_file:returnbase64.b64encode(image_file.read()).decode(utf-8)defembeds(texts:[]None,images:[]None):ifnottextsandnotimages:raiseException(embed content is empty)ifimages:images[encode_image(p)forpinimagesifp]responserequests.post(http://127.0.0.1:8848/v1/embeddings,json{texts:texts,images:images})returnresponse.json()[data]defto_milvus():forindex,imginenumerate(os.listdir(img)):img_pathos.path.join(img,img)embedembeds(images[img_path])content{type:img,content:img}client.upsert(collection_namecollection_name,data{id:str(index),vector:embed[0],content:json.dumps(content,ensure_asciiFalse)})print(save ---- ,img_path)defmain():## 创建collectioncreate_collection()## 向量持久化to_milvus()if__name____main__:main()通过文本进行图像召回检索importbase64importrequestsfrompymilvusimportMilvusClientimportmatplotlib.pyplotaspltfromPILimportImage plt.rcParams[font.sans-serif][SimHei]clientMilvusClient(http://127.0.0.1:19530)collection_namegme_vl_testdefencode_image(image_path):withopen(image_path,rb)asimage_file:returnbase64.b64encode(image_file.read()).decode(utf-8)defembeds(texts:[]None,images:[]None):ifnottextsandnotimages:raiseException(embed content is empty)ifimages:images[encode_image(p)forpinimagesifp]responserequests.post(http://127.0.0.1:8848/v1/embeddings,json{texts:texts,images:images})returnresponse.json()[data]defmain():whileTrue:questioninput(请输入)ifnotquestion:passifquestionq:breakvecembeds(texts[question])resclient.search(collection_name,datavec,limit2,output_fields[content])plt.figure()plt.axis(off)plt.title(f输入问题{question},fontsize20,fontweightbold)forindex,iteminenumerate(res[0]):img_nameitem[entity][content]plt.subplot(1,2,index1)imageImage.open(fimg/{img_name})plt.imshow(image)plt.show()if__name____main__:main()运行后在控制台输入问题 
长河君 · 2026/10/27 1:18:54 · 阅读 1.2k 基于 GME-Qwen2-VL-7B 实现多模态语义检索方案
一、GME-Qwen2-VL-7B 模型GME-Qwen2VL系列是统一的多模态Embedding模型基于Qwen2-VL训练支持动态分辨率。模型支持三种类型的输入文本、图像、图像-文本对所有输入类型都可以生成通用的向量表示并具有优秀的检索性能。使知识向量化不再局限于文本。基于该模型可以实现文搜文、文搜图图搜文图搜图等丰富的场景。GME-Qwen2-VL-7BModelScope地址https://modelscope.cn/models/iic/gme-Qwen2-VL-7B-Instruct本文基于GME-Qwen2-VL-7B模型本地化部署并实现文搜图案例效果如下所示二、GME-Qwen2-VL-7B 部署下载模型modelscope download--modeliic/gme-Qwen2-VL-7B-Instruct--local_dirgme-Qwen2-VL-7B-Instruct然后将下载后gme_inference.py和下面的服务代码放在一起读取模型并启动api服务importtimefromfastapiimportFastAPI,Requestfromfastapi.middleware.corsimportCORSMiddlewareimportuvicorn,jsonfromgme_inferenceimportGmeQwen2VLimporttorchimportbase64fromioimportBytesIOfromPILimportImage appFastAPI()app.add_middleware(CORSMiddleware,allow_origins[*],allow_credentialsTrue,allow_methods[*],allow_headers[*],)model_pathgme-Qwen2-VL-7B-InstructgmeGmeQwen2VL(model_pathmodel_path,devicecuda:0)defbase64_to_image(image_base64:str):image_database64.b64decode(image_base64)image_fileBytesIO(image_data)imageImage.open(image_file)returnimageapp.post(/v1/embeddings)asyncdefembeddings(request:Request):globalmodel,tokenizer json_post_rawawaitrequest.json()json_postjson.dumps(json_post_raw)messagesjson.loads(json_post)textsmessages.get(texts)imagesmessages.get(images)ifnottextsandnotimages:return{code:400,message:texts 和 images 至少传一个,}ttime.time()ifimages:images[base64_to_image(b)forbinimagesifb]iftextsandimages:embedsgme.get_fused_embeddings(textstexts,imagesimages).tolist()eliftexts:embedsgme.get_text_embeddings(textstexts).tolist()else:embedsgme.get_image_embeddings(imagesimages).tolist()use_timetime.time()-tiftorch.backends.mps.is_available():torch.mps.empty_cache()return{code:200,message:success,data:embeds,use_time:use_time}if__name____main__:uvicorn.run(app,host0.0.0.0,port8848,workers1)启动后大概占用17.5G显存。三、API 调用示例importbase64importrequestsdefencode_image(image_path):withopen(image_path,rb)asimage_file:returnbase64.b64encode(image_file.read()).decode(utf-8)defembeds(texts:[]None,images:[]None):ifnottextsandnotimages:raiseException(embed content is empty)ifimages:images[encode_image(p)forpinimagesifp]responserequests.post(http://127.0.0.1:8848/v1/embeddings,json{texts:texts,images:images})returnresponse.json()[data]defmain():texts[你好呀]images[img/1.png]print(embeds(textstexts,imagesimages))if__name____main__:main()调用结果向量维度为3584维。四、实现 文搜图 案例这里我准备了一些 猫、狗的图片通过GME-Qwen2-VL-7B模型向量化并持久化到Milvus向量库中 。importjsonimportosimportbase64importrequestsfrompymilvusimportMilvusClient,DataType clientMilvusClient(http://127.0.0.1:19530)collection_namegme_vl_testdefcreate_collection():client.drop_collection(collection_namecollection_name)schemaMilvusClient.create_schema(auto_idFalse,enable_dynamic_fieldFalse,)schema.add_field(field_nameid,datatypeDataType.VARCHAR,is_primaryTrue,max_length255)schema.add_field(field_namevector,datatypeDataType.FLOAT_VECTOR,dim3584)schema.add_field(field_namecontent,datatypeDataType.VARCHAR,max_length5000)schema.verify()index_paramsclient.prepare_index_params()index_params.add_index(field_namevector,index_typeIVF_FLAT,metric_typeL2,params{nlist:1024})# 创建 collectionclient.create_collection(collection_namecollection_name,schemaschema,index_paramsindex_params)defencode_image(image_path):withopen(image_path,rb)asimage_file:returnbase64.b64encode(image_file.read()).decode(utf-8)defembeds(texts:[]None,images:[]None):ifnottextsandnotimages:raiseException(embed content is empty)ifimages:images[encode_image(p)forpinimagesifp]responserequests.post(http://127.0.0.1:8848/v1/embeddings,json{texts:texts,images:images})returnresponse.json()[data]defto_milvus():forindex,imginenumerate(os.listdir(img)):img_pathos.path.join(img,img)embedembeds(images[img_path])content{type:img,content:img}client.upsert(collection_namecollection_name,data{id:str(index),vector:embed[0],content:json.dumps(content,ensure_asciiFalse)})print(save ---- ,img_path)defmain():## 创建collectioncreate_collection()## 向量持久化to_milvus()if__name____main__:main()通过文本进行图像召回检索importbase64importrequestsfrompymilvusimportMilvusClientimportmatplotlib.pyplotaspltfromPILimportImage plt.rcParams[font.sans-serif][SimHei]clientMilvusClient(http://127.0.0.1:19530)collection_namegme_vl_testdefencode_image(image_path):withopen(image_path,rb)asimage_file:returnbase64.b64encode(image_file.read()).decode(utf-8)defembeds(texts:[]None,images:[]None):ifnottextsandnotimages:raiseException(embed content is empty)ifimages:images[encode_image(p)forpinimagesifp]responserequests.post(http://127.0.0.1:8848/v1/embeddings,json{texts:texts,images:images})returnresponse.json()[data]defmain():whileTrue:questioninput(请输入)ifnotquestion:passifquestionq:breakvecembeds(texts[question])resclient.search(collection_name,datavec,limit2,output_fields[content])plt.figure()plt.axis(off)plt.title(f输入问题{question},fontsize20,fontweightbold)forindex,iteminenumerate(res[0]):img_nameitem[entity][content]plt.subplot(1,2,index1)imageImage.open(fimg/{img_name})plt.imshow(image)plt.show()if__name____main__:main()运行后在控制台输入问题