背景在某次迁移Oracle到GaussDB时应用开发人员将改好的package在GaussDB里进行创建没有ERROR也没有WARNING但是编译无效对象的时候报错了。虽然已经找到了是哪个包编译报错但是这个包有上万行而且里面也有好几十个procedure而报错信息仅仅只有 ERROR: Failed to query the 323 type in the cache.没有上下文信息连行号都没有根本无从判断是哪里出了问题。基本排查尝试drop这个package然后重建再编译现象一样尝试重启数据库以清空全局PLSQL缓存再编译现象一样这意味着这个问题与缓存无关大概率也与其他依赖对象无关所以暂时针对这个package进行排查。对于不会gdb调试的人来说要排查这个问题只能对着这个package使用二分法来删除里面的procedure直到删到某个procedure前后报错发生变化但要注意里面的procedure的依赖。当时就这么一路删最后的确发现了原因但是耗费的时间非常久解决依赖关系时还要手动改代码。那么有没有一种方式能迅速定位是哪个procedure的问题么当然有那就是使用gdb直接进行内核级别调试因为Gauss系数据库编译package时都是会逐个对里面的每一个procedure和function进行编译。gdb调试前置准备由于已经找到了触发这个报错的package特征,因此下面就用最小化模拟用例来进行演示:测试用例create package pkg_test_4 is procedurep1(i1 in varchar2,i2 out varchar2,i3 out varchar2);end;/ create package body pkg_test_4 is procedurep1(i1 in varchar2,i2 in varchar2,i3 out varchar2)is begin null;end;end;/ alter package pkg_test_4 compile;执行效果gaussdb# alter package pkg_test_4 compile;gaussdb# create or replace package pkg_test_4 is gaussdb$# procedurep1(i1 in varchar2,gaussdb$# i2 out varchar2,gaussdb$# i3 out varchar2);gaussdb$# end;gaussdb$# / CREATE PACKAGE gaussdb# create or replace package body pkg_test_4 is gaussdb$# procedurep1(i1 in varchar2,gaussdb$# i2 in varchar2,gaussdb$# i3 out varchar2)is gaussdb$# begin gaussdb$# null;gaussdb$# end;gaussdb$# end;gaussdb$# / CREATE PACKAGE BODY gaussdb# alter package pkg_test_4 compile;ERROR:Failed to query the 323 type in the cache. gaussdb#使用gdb调试找问题有个关键就是这个问题最好是能稳定复现的否则gdb抓不到报错现场也很难分析问题。另外开始gdb调试前一定要先把对应版本的符号表准备好比较简单的方式就是直接把符号表里的bin和lib解压到GaussDB的bin和lib目录。在之前分析MogDB的问题时我们内核研发有教过我可以使用 b errstart if elevel19设置断点来断住所有 ERROR以上级别的报错但是这招在GaussDB似乎不行了(gdb)b errstart if elevel19 No symbolelevelin current context.(gdb)如果直接 b errstart,是可以断但是会老是断根本没法跑起来因为这里就算没报错也会调用进来几乎所有线程都在频繁走到这里参考openGauss源码中的错误级别代码里面连INFO/NOTICE都有/* Error level codes */#define DEBUG5 \ 10/* Debugging messages, in categories of \ * decreasing detail. */#define DEBUG4 11 #define DEBUG3 12 #define DEBUG2 13 #define DEBUG1 14/* used by GUC debug_* variables */#define LOG \ 15/* Server operational messages; sent only to \ * server log by default. */#define COMMERROR \ 16/* Client communication problems; same as LOG \ * for server reporting, but never sent to \ * client. */#define INFO \ 17/* Messages specifically requested by user (eg \ * VACUUM VERBOSE output); always sent to \ * client regardless of client_min_messages, \ * but by default not sent to server log. */#define NOTICE \ 18/* Helpful messages to users about query \ * operation; sent to client and server log by \ * default. */#define WARNING \ 19/* Warnings. NOTICE is for expected messages \ * like implicit sequence creation by SERIAL. \ * WARNING is for unexpected messages. */#define ERROR \ 20/* user error - abort transaction; return to \ * known state */#define VERBOSEMESSAGE \ 9/* indicates to show verbose info for CN and DNs; \ * for DNs means to send info back to CN *//* Save ERROR value in PGERROR so it can be restored when Win32 includes * modify it. We have to use a constant rather than ERROR because macros * are expanded only when referenced outside macros. */#ifdef WIN32 #define PGERROR 20 #endif #define FATAL 21/* fatal error - abort process */#define PANIC 22/* take down the other backends with me *//* MAKE_SQLSTATE(P, 1, 0 , 0, 0)96 */#define CUSTOM_ERRCODE_P1 96看一下 b errstart 会断到哪里(gdb)b errstart Breakpoint 1 at 0x564904871b20:errstart.(3 locations)(gdb)info b Num Type Disp Enb Address What 1 breakpoint keep y MULTIPLE 1.1 y 0x0000564904871b20 inerrstart(int,char const*,int,char const*,char const*)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/share/error/elog.cpp:4108 1.2 y 0x00007f6b9cd5e5b0 errstart(int,char const*,int,char const*,char const*)plt 1.3 y 0x00007f6b9d03eb30 errstart(int,char const*,int,char const*,char const*)plt(gdb)从这个断点信息里来看errstart在 elog.cpp的4108行这很是可疑因为无论是openGauss还是MogDB,这个errstart函数应该在更前面的位置大概是第两百多行的地方。由于没有源码就只能反汇编看下有没有能参考的信息了(gdb)disassemble /m errstart Dump of assembler code for functionerrstart(int,char const*,int,char const*,char const*):238 /usr1/GaussDBKernel/server/opengauss/src/auxiliary/share/error/elog.cpp:No such file or directory. 0x0000564904871b25 5:push %rbp 0x0000564904871b26 6:mov %rsp,%rbp 0x0000564904871b29 9:push %r15 0x0000564904871b2b 11:push %r14 0x0000564904871b2d 13:push %r13 0x0000564904871b2f 15:push %r12 0x0000564904871b31 17:mov %rsi,%r14 0x0000564904871b34 20:push %rbx 0x0000564904871b35 21:mov %edi,%ebx 0x0000564904871b37 23:sub $0x58,%rsp 0x0000564904871b3e 30:mov %edx,-0x64(%rbp)0x0000564904871b41 33:mov %rcx,-0x70(%rbp)0x0000564904871b45 37:mov %r8,-0x78(%rbp)239 in /usr1/GaussDBKernel/server/opengauss/src/auxiliary/share/error/elog.cpp 240 in /usr1/GaussDBKernel/server/opengauss/src/auxiliary/share/error/elog.cpp可以看到这个函数第一次出现其实是在第238行从这几个寄存器的操作来看应该就是函数的入口也就是说实际上这个函数的定义应该在第238行而不是前面的4108行。我观察了下4108行的elevel一直是被优化掉的看不到里面的值只有238行的elevel能观测到值。所以在GaussDB里要断ERROR及以上级别的错误断点应该设置为b elog.cpp:238 if(elevel 19)可以提前先准备好下面的命令gdb进去后直接复制粘贴减少进程中断时间b elog.cpp:238 if(elevel 19)handle SIGUSR1 nostop noprint handle SIGUSR2 nostop noprint handle SIGPIPE nostop set pagi off set print elements 300 continue正式开始gdb调试先用 ps -ef |grep gaussdb找到进程号然后gdb -p 进程号[gaussdb506ky10-sp3 ~]$ ps -ef |grep gaussdb root 426694 426551 0 09:19 pts/0 00:00:00 su - gaussdb506 gaussdb 426699 426694 0 09:19 pts/0 00:00:00 -bash gaussdb 427027 426699 0 09:19 pts/0 00:00:00 ps -ef gaussdb 427028 426699 0 09:19 pts/0 00:00:00 grep gaussdb og_last 3231792 1 1 Jul22 ? 05:58:52 /opt/og_lastest/openGauss-server/dest/bin/gaussdb og700rc1 3508544 1 1 Aug04 ? 00:33:58 /opt/og700rc1/app/bin/gaussdb -D /opt/og700rc1/data -M primary gaussdb 3864702 1 29 Aug05 ? 06:33:50 /data/gaussdb506/app/bin/gaussdb [gaussdb506ky10-sp3 ~]$ gdb -p 3864702 GNU gdb(GDB)KylinOS 9.2-3.p01.ky10 Copyright(C)2020 Free Software Foundation,Inc. License GPLv3:GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is freesoftware:you are free to change and redistribute it. There is NO WARRANTY,to the extent permitted by law. Typeshow copyingandshow warrantyfor details. This GDB was configured asx86_64-kylin-linux-gnu. Typeshow configurationfor configuration details. For bug reporting instructions,pleasesee:http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources onlineat:http://www.gnu.org/software/gdb/documentation/. For help,typehelp. Typeapropos wordto search for commands related toword. Attaching to process 3864702 [New LWP 3864703] [New LWP 3864749]...#省略 [New LWP 4081944]warning:File/usr/lib64/libthread_db-1.0.soauto-loading has been declined by your auto-load safe-path set to$debugdir:$datadir/auto-load. --Type RET for more,q to quit,c to continue without paging-- To enable execution of this file add add-auto-load-safe-path /usr/lib64/libthread_db-1.0.so line to your configuration file/home/gaussdb506/.gdbinit. To completely disable this security protection add set auto-load safe-path / line to your configuration file/home/gaussdb506/.gdbinit. For more information about this security protection see theAuto-loading safe pathsection in the GDB manual. E.g.,run from theshell:info(gdb)Auto-loading safe pathwarning:Unable to find libthread_db matching inferiors thread library,thread debugging will not be available.warning:File/usr/lib64/libthread_db-1.0.soauto-loading has been declined by your auto-load safe-path set to$debugdir:$datadir/auto-load.warning:Unable to find libthread_db matching inferiors thread library,thread debugging will not be available. 0x00007f6bbc53c849 in poll()from /usr/lib64/libc.so.6(gdb)b elog.cpp:238 if(elevel 19)Breakpoint 1 at 0x564904871b25:file /usr1/GaussDBKernel/server/opengauss/src/auxiliary/share/error/elog.cpp,line 238.(gdb)handle SIGUSR1 nostop noprint Signal Stop Print Pass to program Description SIGUSR1 No No Yes User defined signal 1(gdb)handle SIGUSR2 nostop noprint Signal Stop Print Pass to program Description SIGUSR2 No No Yes User defined signal 2(gdb)handle SIGPIPE nostop Signal Stop Print Pass to program Description SIGPIPE No Yes Yes Broken pipe(gdb)set pagi off(gdb)set print elements 300(gdb)continue Continuing. [New LWP 427599] [New LWP 427600] [LWP 427599 exited] [New LWP 427601] [LWP 427601 exited] [New LWP 427602] [LWP 427602 exited] [LWP 427600 exited]当后面不断有输出 [New LWP xxxxxx]时gaussdb就是正常运行中了。接下来可以开一个客户端连接执行上面用于模拟测试的sql会卡在 alter package pkg_test_4 compile; 这个语句上同时gdb的窗口不再连续打印 [New LWP xxxxxx]而是命中了断点[Switching to LWP 3864766] Thread 16TPLworkerhit Breakpoint 1,errstart(elevel20,filename0x564908abfcc8format_type.cpp,lineno216,funcname0x564908abfdd0 format_type_internal(unsigned int,int,bool,bool,bool)::__func__format_type_internal,domain0x5649087e1004plpgsql-9.2)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/share/error/elog.cpp:238 238 in /usr1/GaussDBKernel/server/opengauss/src/auxiliary/share/error/elog.cpp(gdb)接着输入bt查看堆栈(gdb)bt #0 errstart(elevel20,filename0x564908abfcc8format_type.cpp,lineno216,funcname0x564908abfdd0 format_type_internal(unsigned int,int,bool,bool,bool)::__func__format_type_internal,domain0x5649087e1004plpgsql-9.2)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/share/error/elog.cpp:238 #1 0x00005649043cc8ae in format_type_internal(type_oid323,typemod-1,typemod_givenoptimized out,allow_invalidoptimized out,include_nspnameoptimized out)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/share/adt/format_type.cpp:212 #2 0x00005649045b82e8 in format_procedure(procedure_oidoptimized out)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/share/adt/regproc.cpp:473 #3 0x0000564904a2d53d in do_compile(fcinfo0x7f6a520460c0,proc_tup0x7f69b7c375a0,func0x7f6a39c64050,compile_func_head_info0x7f6a52046740,for_validatortrue,hashkey0x7f6a52045d50)at /usr1/GaussDBKernel/server/opengauss/src/gausskernel/pl/plsql/pl_comp/pl_comp_func_main.cpp:921 #4 0x0000564904a35f3b in gsplsql_compile(fcinfo0x7f6a520460c0,compile_func_head_info0x7f6a52046740,for_validatortrue,isRecompilefalse,func_runtime_state0x0)at /usr1/GaussDBKernel/server/opengauss/src/gausskernel/pl/plsql/pl_comp/pl_comp_func_main.cpp:3106 #5 0x0000564906c64eeb in plpgsql_validator(fcinfooptimized out)at /usr1/GaussDBKernel/server/opengauss/src/compatibility/sql_adaptor/pl/plpgsql/src/pl_handler.cpp:1481 #6 0x00005649048a56cb in OidFunctionCall4Coll(function_id10790,collation0,arg197664,arg20,arg30,arg4140094619281216,is_null0x0)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/share/fmgr/fmgr.cpp:2512 #7 0x0000564904a55363 in gsplsql_func_in_pkg_compile(pkg0x7f6a455ec050)at /usr1/GaussDBKernel/server/opengauss/src/gausskernel/pl/plsql/pl_comp/pl_comp_pkg_main.cpp:1210 #8 0x0000564904a571fc in gsplsql_pkg_init(pkg0x7f6a455ec050,isCreatefalse,isSpecfalse,ret_pkg_runtime0x7f6a52046a18,is_need_compile_functrue,pkg_debug_query_stringoptimized out,old_pkg_runtime0x7f6a39a48050)at /usr1/GaussDBKernel/server/opengauss/src/gausskernel/pl/plsql/pl_comp/pl_comp_pkg_main.cpp:1545 #9 0x0000564904a584a3 in gsplsql_pkg_compile(pkg_oid97663,for_validatortrue,is_specfalse,is_createfalse,is_recompiletrue,pkg_runtime_state0x0)at /usr1/GaussDBKernel/server/opengauss/src/gausskernel/pl/plsql/pl_comp/pl_comp_pkg_main.cpp:956 #10 0x0000564905100dc6 in recompile_single_package(package_oid97663,is_specfalse)at /usr1/GaussDBKernel/server/opengauss/src/compatibility/sql_adaptor/commands/packagecmds.cpp:329 #11 0x0000564905101212 in recompile_package_by_oid(pkg_oid97663,recompile_invalid_pkgfalse)at /usr1/GaussDBKernel/server/opengauss/src/compatibility/sql_adaptor/commands/packagecmds.cpp:416 #12 0x0000564905101262 in recompile_package(stmt0x7f6a3f4954c0)at /usr1/GaussDBKernel/server/opengauss/src/compatibility/sql_adaptor/commands/packagecmds.cpp:437 #13 0x0000564905431abe in sqlcmd_standard_process_utility(parse_tree0x7f6a3f4954c0,query_string0x7f6a3f496050alter package pkg_test_4 compile,params0x0,is_top_leveloptimized out,dest0x56490a6e0720 donothingDR,sent_to_remoteoptimized out,completion_tag0x7f6a5204a430,isCTASfalse)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/proc/tcop/utility.cpp:6813 #14 0x00007f6b9cd9b759 in gsaudit_ProcessUtility_hook(parsetree0x7f6a3f4954c0,queryString0x7f6a3f496050alter package pkg_test_4 compile,params0x0,isTopLeveloptimized out,dest0x56490a6e0720 donothingDR,sentToRemoteoptimized out,completionTag0x7f6a5204a430,isCTASfalse)at /usr1/GaussDBKernel/server/opengauss/src/gausskernel/security/security_plugin/security_policy_plugin.cpp:856 #15 0x00005649059a0f52 in audit_process_utility(parsetree0x7f6a3f4954c0,query_string0x7f6a3f496050alter package pkg_test_4 compile,paramsoptimized out,is_top_leveloptimized out,destoptimized out,sent_to_remoteoptimized out,completion_tag0x7f6a5204a430,is_ctasfalse)at /usr1/GaussDBKernel/server/opengauss/src/gausskernel/security/audit/security_auditfuncs.cpp:1512 #16 0x000056490543c71d in sqlcmd_process_utility(parse_tree0x7f6a3f4954c0,query_string0x7f6a3f496050alter package pkg_test_4 compile,params0x0,is_top_leveloptimized out,destoptimized out,sent_to_remoteoptimized out,completion_tag0x7f6a5204a430,isCTASfalse)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/proc/tcop/utility.cpp:1974 #17 0x000056490541d83f in PortalRunUtility(portal0x7f6a48878050,utilityStmt0x7f6a3f4954c0,isTopLeveltrue,dest0x56490a6e0720 donothingDR,completionTag0x7f6a5204a430)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/proc/tcop/pquery.cpp:2140 #18 0x000056490541f0be in PortalRunMulti(portal0x7f6a48878050,isTopLeveltrue,dest0x56490a6e0720 donothingDR,altdest0x56490a6e0720 donothingDR,completionTag0x7f6a5204a430,snapshot0x0,bii_state0x0)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/proc/tcop/pquery.cpp:2326 #19 0x00005649054232dc in PortalRun(portal0x7f6a48878050,count200,isTopLeveltrue,dest0x7f6a3f4a84d0,altdest0x7f6a3f4a84d0,completionTag0x7f6a5204a430,snapshot0x0,bii_state0x0)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/proc/tcop/pquery.cpp:1501 #20 0x00005649054158af in exec_execute_message(max_rows200,portal_name0x7f6a3f4a8050)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/proc/tcop/postgres.cpp:7071 #21 gs_process_command(firstcharoptimized out,input_messageoptimized out,send_ready_for_queryoptimized out)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/proc/tcop/postgres.cpp:12314 #22 0x000056490541b9c0 in PostgresMain(argcoptimized out,argv0x7f6a49e45b20,dbnameoptimized out,usernameoptimized out)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/proc/tcop/postgres.cpp:11313 #23 0x000056490539f2df in backend_run(port0x7f6a5204a890)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/proc/postmaster/postmaster.cpp:12482 #24 0x00005649053de1b0 in gauss_db_worker_thread_main(knl_thread_role)2(argoptimized out)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/proc/postmaster/postmaster.cpp:19086 #25 0x000056490539f39a in internal_thread_func(argsoptimized out)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/proc/postmaster/postmaster.cpp:20196 #26 0x00007f6bbc60ff1b in ??()from /usr/lib64/libpthread.so.0 #27 0x00007f6bbc547320 in clone()from /usr/lib64/libc.so.6可以看到在报错的 format_type_internal里出现了type_oid323的确是在报错中出现的数字但是323这么小的数字明显不可能是用户自定义类型因为小数字的oid都是被系统保留的。这里肯定是有bug的但没有源码不方便找bug原因本次调试的主要目的是找到出错的procedure。在 format_type_internal和 format_procedure里有用的参数都显示成 了这表示内核把这些变量优化掉了不给看。于是继续看下一行 do_compile打印几个参数看看(gdb) f 3#3 0x0000564904a2d53d in do_compile (fcinfo0x7f6a520460c0, proc_tup0x7f69b7c375a0, func0x7f6a39c64050,compile_func_head_info0x7f6a52046740, for_validatortrue,hashkey0x7f6a52045d50) at/usr1/GaussDBKernel/server/opengauss/src/gausskernel/pl/plsql/pl_comp/pl_comp_func_main.cpp:921921/usr1/GaussDBKernel/server/opengauss/src/gausskernel/pl/plsql/pl_comp/pl_comp_func_main.cpp:No such file or directory. (gdb) p *fcinfo $1 {flinfo 0x7f6a52046020, context 0x0, resultinfo 0x0, fncollation 0,isnull false, nargs 0, arg 0x7f6a39a4ca50, argnull 0x0,argTypes 0x0, prealloc_arg {0 repeats 20 times},prealloc_argnull {false repeats 20 times}, prealloc_argTypes {0repeats 20 times}, argVector 0x0, refcursor_data {argCursor 0x0, returnCursor 0x0, return_number 0}, out_tmtype 0 ‘\000’,out_decimals 0 ‘\000’, udfInfo {UDFArgsHandlerPtr 0x0,UDFResultHandlerPtr 0x0, udfMsgBuf 0x0, msgReadPtr 0x0,argBatchRows 0, allocRows 0, arg 0x0, null 0x0, result 0x0,resultIsNull 0x0, valid_UDFArgsHandlerPtr false}, swinfo {sw_econtext 0x0, sw_exprstate 0x0, sw_is_flt_frame false},out_typmode 0x0, fn_typmode 0, plfunc_exec_mode 0,plfunc_exec_state 0x0, args_done 0x0, prealloc_args_done {0repeats 20 times}, arginfo {{in_tmtype 0 ‘\000’, in_decimals 0‘\000’, argTypModes 0, set_enum_typeoid 0}}} (gdb) p *proc_tupAttempt to dereference a generic pointer. (gdb) p *func $2 {type T_PLpgSQL_FUNCTION, fn_oid 97664, pkg_oid 97663, namespaceOid 2200, fn_owner 16728, fn_input_collation 0, fn_signature 0x0,fn_searchpath 0x0, namespace_searchpath 0x0, fn_hashkey 0x0,fn_cxt 0x7f6a6a9599d0, fn_rettype 0, fn_rettyplen 0,glc_func_life 1, fn_rettypioparam 0, fn_retbyval false,fn_retistuple false, fn_retset false, fn_readonly false,out_param_varno -1, found_varno 0, fn_nallargs 0, argmods 0x0,argtypes 0x0, sql_cursor_found_varno 0, sql_notfound_varno 0,sql_isopen_varno 0, sql_rowcount_varno 0,sql_bulk_exceptions_varno 0, sqlcode_varno 0, sqlstate_varno 0,sqlerrm_varno 0, new_varno 0, old_varno 0, tg_name_varno 0,tg_when_varno 0, tg_level_varno 0, tg_op_varno 0, tg_relid_varno 0, tg_relname_varno 0, tg_table_name_varno 0, tg_table_schema_varno 0, tg_nargs_varno 0, tg_argv_varno 0,retvarno 0, guc_stat 5, use_count 0, resolve_option GSPLSQL_RESOLVE_COLUMN, ndatums 0, datums 0x0, datum_need_free 0x0, action 0x0, goto_labels 0x0, invalItems 0x0,saved_unique_id 4294967295, nPlaceHolders 0, placeholders 0x0,cur_estate 0x0, tg_relation 0x0, debug 0x0, ns_top 0x0,is_private false, fn_is_trigger false, pre_parse_trig false,is_autonomous false, is_inline_handler false, is_valid true,is_plpgsql_func_with_outparam false, need_skip_process_autonm_pkg false, remembered_by_resowner false, typeList 0x0, namespace_name 0x0, expr_list 0x0, fn_retinput {fn_addr 0x0, fn_oid 0, fn_nargs 0, fn_strict false, fn_retset false, fn_extra 0x0,fn_mcxt 0x0, fn_expr 0x0, fn_rettype 0, fn_rettypemod 0,fnName ‘\000’ repeats 63 times, fnLibPath 0x0, vec_fn_addr 0x0, vec_fn_cache 0x0, genericRuntime 0x0, max_length 0,fn_languageId 0, fn_stats 0 ‘\000’, fn_fenced false, fn_volatile 0 ‘\000’, decimals 0 ‘\000’}, glc_status {m_type GLC_FUNCTION_OBJ, m_location GLC_OBJECT_IN_SESSION_WAIT_REMOVE,m_glc_object_state GLC_OBJECT_IS_VALID, m_refcount 1},expired_cell {dle_next 0x0, dle_prev 0x0, dle_val 0x7f6a39c64050, dle_list 0x7f6a48876ea0}, compiled_dlist_elem {dle_next 0x0, dle_prev 0x0, dle_val 0x7f6a39c64050, dle_list 0x0}, parent_pro_ndatum 0, subparam 0x0, fn_nargs 0,copiable_size 0, deep_datums 0x0, deep_ndatums 0, cursor_datums 0x0, cursor_ndatums 0, placeholder_datums 0x0, placeholder_ndatums 0, fn_argvarnos 0x0, depend_info_list 0x0,plan_total_mem_size 0, block_level 0x0} (gdb)可以在 func里看到fn_oid97664 这意味着是在编译pg_proc里oid为97664的对象。于是我们输入q退出gdb然后回到客户端查询gaussdb# select proname,g.pkgname from pg_proc p,gs_package g where p.oid97664 and g.oidp.propackageid;proname | pkgname --------------------- p1 | pkg_test_4(1 row)可以看到这个oid对应的就是pkg_test_4这个包里的p1于是就知道了一定是编译p1的时候出了问题。到此出问题的procedure就直接找出来了肉眼一看包头和包体的定义发现有个参数的in/out方向没匹配但GaussDB在创建这个package时竟然没有报错…其他Gauss系数据库的情况同样的这个代码在openGauss 7.0.0 RC1 是不会报错的这个package还能正常调用查了下数据字典出入参方向是按包体生效的,这同样也是个BUG没有做严格判断。openGauss# create package pkg_test_4 is openGauss$# procedurep1(i1 in varchar2,openGauss$# i2 out varchar2,openGauss$# i3 out varchar2);openGauss$# end pkg_test_4;openGauss$# /;end;end pkg_test_4;/CREATE PACKAGE openGauss# create package body pkg_test_4 is openGauss$# procedurep1(i1 in varchar2,openGauss$# i2 in varchar2,openGauss$# i3 out varchar2)is openGauss$# begin openGauss$# null;openGauss$# end;openGauss$# end pkg_test_4;openGauss$# / CREATE PACKAGE BODY openGauss# alter package pkg_test_4 compile;ALTER PACKAGE openGauss# call pkg_test_4.p1(null,null,null);i3 ----(1 row)openGauss#在MogDB 5.2.0里则是在创建package body时就报错了能正确检查到包头里的procedure在包体里没定义MogDB# create package pkg_test_4 is MogDB$# procedurep1(i1 in varchar2,MogDB$# i2 out varchar2,MogDB$# i3 out varchar2);MogDB$# end pkg_test_4;MogDB$# / CREATE PACKAGE MogDB# create package body pkg_test_4 is MogDB$# procedurep1(i1 in varchar2,MogDB$# i2 in varchar2,MogDB$# i3 out varchar2)is MogDB$# begin MogDB$# null;MogDB$# end;MogDB$# end pkg_test_4;MogDB$# /ERROR:Function definition notfound:p1 MogDB#总结本次触发 ERROR: Failed to query the 323 type in the cache.这个报错的直接原因是创建的package和package body中有个procedure的参数in/out方向不匹配导致。虽然客户代码的确有问题但根本原因还是数据库有BUG未将这种异常场景检查出来。 
长河君 · 2026/10/24 6:22:49 · 阅读 1.2k 【GaussDB】使用gdb定位GaussDB编译package报错
背景在某次迁移Oracle到GaussDB时应用开发人员将改好的package在GaussDB里进行创建没有ERROR也没有WARNING但是编译无效对象的时候报错了。虽然已经找到了是哪个包编译报错但是这个包有上万行而且里面也有好几十个procedure而报错信息仅仅只有 ERROR: Failed to query the 323 type in the cache.没有上下文信息连行号都没有根本无从判断是哪里出了问题。基本排查尝试drop这个package然后重建再编译现象一样尝试重启数据库以清空全局PLSQL缓存再编译现象一样这意味着这个问题与缓存无关大概率也与其他依赖对象无关所以暂时针对这个package进行排查。对于不会gdb调试的人来说要排查这个问题只能对着这个package使用二分法来删除里面的procedure直到删到某个procedure前后报错发生变化但要注意里面的procedure的依赖。当时就这么一路删最后的确发现了原因但是耗费的时间非常久解决依赖关系时还要手动改代码。那么有没有一种方式能迅速定位是哪个procedure的问题么当然有那就是使用gdb直接进行内核级别调试因为Gauss系数据库编译package时都是会逐个对里面的每一个procedure和function进行编译。gdb调试前置准备由于已经找到了触发这个报错的package特征,因此下面就用最小化模拟用例来进行演示:测试用例create package pkg_test_4 is procedurep1(i1 in varchar2,i2 out varchar2,i3 out varchar2);end;/ create package body pkg_test_4 is procedurep1(i1 in varchar2,i2 in varchar2,i3 out varchar2)is begin null;end;end;/ alter package pkg_test_4 compile;执行效果gaussdb# alter package pkg_test_4 compile;gaussdb# create or replace package pkg_test_4 is gaussdb$# procedurep1(i1 in varchar2,gaussdb$# i2 out varchar2,gaussdb$# i3 out varchar2);gaussdb$# end;gaussdb$# / CREATE PACKAGE gaussdb# create or replace package body pkg_test_4 is gaussdb$# procedurep1(i1 in varchar2,gaussdb$# i2 in varchar2,gaussdb$# i3 out varchar2)is gaussdb$# begin gaussdb$# null;gaussdb$# end;gaussdb$# end;gaussdb$# / CREATE PACKAGE BODY gaussdb# alter package pkg_test_4 compile;ERROR:Failed to query the 323 type in the cache. gaussdb#使用gdb调试找问题有个关键就是这个问题最好是能稳定复现的否则gdb抓不到报错现场也很难分析问题。另外开始gdb调试前一定要先把对应版本的符号表准备好比较简单的方式就是直接把符号表里的bin和lib解压到GaussDB的bin和lib目录。在之前分析MogDB的问题时我们内核研发有教过我可以使用 b errstart if elevel19设置断点来断住所有 ERROR以上级别的报错但是这招在GaussDB似乎不行了(gdb)b errstart if elevel19 No symbolelevelin current context.(gdb)如果直接 b errstart,是可以断但是会老是断根本没法跑起来因为这里就算没报错也会调用进来几乎所有线程都在频繁走到这里参考openGauss源码中的错误级别代码里面连INFO/NOTICE都有/* Error level codes */#define DEBUG5 \ 10/* Debugging messages, in categories of \ * decreasing detail. */#define DEBUG4 11 #define DEBUG3 12 #define DEBUG2 13 #define DEBUG1 14/* used by GUC debug_* variables */#define LOG \ 15/* Server operational messages; sent only to \ * server log by default. */#define COMMERROR \ 16/* Client communication problems; same as LOG \ * for server reporting, but never sent to \ * client. */#define INFO \ 17/* Messages specifically requested by user (eg \ * VACUUM VERBOSE output); always sent to \ * client regardless of client_min_messages, \ * but by default not sent to server log. */#define NOTICE \ 18/* Helpful messages to users about query \ * operation; sent to client and server log by \ * default. */#define WARNING \ 19/* Warnings. NOTICE is for expected messages \ * like implicit sequence creation by SERIAL. \ * WARNING is for unexpected messages. */#define ERROR \ 20/* user error - abort transaction; return to \ * known state */#define VERBOSEMESSAGE \ 9/* indicates to show verbose info for CN and DNs; \ * for DNs means to send info back to CN *//* Save ERROR value in PGERROR so it can be restored when Win32 includes * modify it. We have to use a constant rather than ERROR because macros * are expanded only when referenced outside macros. */#ifdef WIN32 #define PGERROR 20 #endif #define FATAL 21/* fatal error - abort process */#define PANIC 22/* take down the other backends with me *//* MAKE_SQLSTATE(P, 1, 0 , 0, 0)96 */#define CUSTOM_ERRCODE_P1 96看一下 b errstart 会断到哪里(gdb)b errstart Breakpoint 1 at 0x564904871b20:errstart.(3 locations)(gdb)info b Num Type Disp Enb Address What 1 breakpoint keep y MULTIPLE 1.1 y 0x0000564904871b20 inerrstart(int,char const*,int,char const*,char const*)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/share/error/elog.cpp:4108 1.2 y 0x00007f6b9cd5e5b0 errstart(int,char const*,int,char const*,char const*)plt 1.3 y 0x00007f6b9d03eb30 errstart(int,char const*,int,char const*,char const*)plt(gdb)从这个断点信息里来看errstart在 elog.cpp的4108行这很是可疑因为无论是openGauss还是MogDB,这个errstart函数应该在更前面的位置大概是第两百多行的地方。由于没有源码就只能反汇编看下有没有能参考的信息了(gdb)disassemble /m errstart Dump of assembler code for functionerrstart(int,char const*,int,char const*,char const*):238 /usr1/GaussDBKernel/server/opengauss/src/auxiliary/share/error/elog.cpp:No such file or directory. 0x0000564904871b25 5:push %rbp 0x0000564904871b26 6:mov %rsp,%rbp 0x0000564904871b29 9:push %r15 0x0000564904871b2b 11:push %r14 0x0000564904871b2d 13:push %r13 0x0000564904871b2f 15:push %r12 0x0000564904871b31 17:mov %rsi,%r14 0x0000564904871b34 20:push %rbx 0x0000564904871b35 21:mov %edi,%ebx 0x0000564904871b37 23:sub $0x58,%rsp 0x0000564904871b3e 30:mov %edx,-0x64(%rbp)0x0000564904871b41 33:mov %rcx,-0x70(%rbp)0x0000564904871b45 37:mov %r8,-0x78(%rbp)239 in /usr1/GaussDBKernel/server/opengauss/src/auxiliary/share/error/elog.cpp 240 in /usr1/GaussDBKernel/server/opengauss/src/auxiliary/share/error/elog.cpp可以看到这个函数第一次出现其实是在第238行从这几个寄存器的操作来看应该就是函数的入口也就是说实际上这个函数的定义应该在第238行而不是前面的4108行。我观察了下4108行的elevel一直是被优化掉的看不到里面的值只有238行的elevel能观测到值。所以在GaussDB里要断ERROR及以上级别的错误断点应该设置为b elog.cpp:238 if(elevel 19)可以提前先准备好下面的命令gdb进去后直接复制粘贴减少进程中断时间b elog.cpp:238 if(elevel 19)handle SIGUSR1 nostop noprint handle SIGUSR2 nostop noprint handle SIGPIPE nostop set pagi off set print elements 300 continue正式开始gdb调试先用 ps -ef |grep gaussdb找到进程号然后gdb -p 进程号[gaussdb506ky10-sp3 ~]$ ps -ef |grep gaussdb root 426694 426551 0 09:19 pts/0 00:00:00 su - gaussdb506 gaussdb 426699 426694 0 09:19 pts/0 00:00:00 -bash gaussdb 427027 426699 0 09:19 pts/0 00:00:00 ps -ef gaussdb 427028 426699 0 09:19 pts/0 00:00:00 grep gaussdb og_last 3231792 1 1 Jul22 ? 05:58:52 /opt/og_lastest/openGauss-server/dest/bin/gaussdb og700rc1 3508544 1 1 Aug04 ? 00:33:58 /opt/og700rc1/app/bin/gaussdb -D /opt/og700rc1/data -M primary gaussdb 3864702 1 29 Aug05 ? 06:33:50 /data/gaussdb506/app/bin/gaussdb [gaussdb506ky10-sp3 ~]$ gdb -p 3864702 GNU gdb(GDB)KylinOS 9.2-3.p01.ky10 Copyright(C)2020 Free Software Foundation,Inc. License GPLv3:GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is freesoftware:you are free to change and redistribute it. There is NO WARRANTY,to the extent permitted by law. Typeshow copyingandshow warrantyfor details. This GDB was configured asx86_64-kylin-linux-gnu. Typeshow configurationfor configuration details. For bug reporting instructions,pleasesee:http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources onlineat:http://www.gnu.org/software/gdb/documentation/. For help,typehelp. Typeapropos wordto search for commands related toword. Attaching to process 3864702 [New LWP 3864703] [New LWP 3864749]...#省略 [New LWP 4081944]warning:File/usr/lib64/libthread_db-1.0.soauto-loading has been declined by your auto-load safe-path set to$debugdir:$datadir/auto-load. --Type RET for more,q to quit,c to continue without paging-- To enable execution of this file add add-auto-load-safe-path /usr/lib64/libthread_db-1.0.so line to your configuration file/home/gaussdb506/.gdbinit. To completely disable this security protection add set auto-load safe-path / line to your configuration file/home/gaussdb506/.gdbinit. For more information about this security protection see theAuto-loading safe pathsection in the GDB manual. E.g.,run from theshell:info(gdb)Auto-loading safe pathwarning:Unable to find libthread_db matching inferiors thread library,thread debugging will not be available.warning:File/usr/lib64/libthread_db-1.0.soauto-loading has been declined by your auto-load safe-path set to$debugdir:$datadir/auto-load.warning:Unable to find libthread_db matching inferiors thread library,thread debugging will not be available. 0x00007f6bbc53c849 in poll()from /usr/lib64/libc.so.6(gdb)b elog.cpp:238 if(elevel 19)Breakpoint 1 at 0x564904871b25:file /usr1/GaussDBKernel/server/opengauss/src/auxiliary/share/error/elog.cpp,line 238.(gdb)handle SIGUSR1 nostop noprint Signal Stop Print Pass to program Description SIGUSR1 No No Yes User defined signal 1(gdb)handle SIGUSR2 nostop noprint Signal Stop Print Pass to program Description SIGUSR2 No No Yes User defined signal 2(gdb)handle SIGPIPE nostop Signal Stop Print Pass to program Description SIGPIPE No Yes Yes Broken pipe(gdb)set pagi off(gdb)set print elements 300(gdb)continue Continuing. [New LWP 427599] [New LWP 427600] [LWP 427599 exited] [New LWP 427601] [LWP 427601 exited] [New LWP 427602] [LWP 427602 exited] [LWP 427600 exited]当后面不断有输出 [New LWP xxxxxx]时gaussdb就是正常运行中了。接下来可以开一个客户端连接执行上面用于模拟测试的sql会卡在 alter package pkg_test_4 compile; 这个语句上同时gdb的窗口不再连续打印 [New LWP xxxxxx]而是命中了断点[Switching to LWP 3864766] Thread 16TPLworkerhit Breakpoint 1,errstart(elevel20,filename0x564908abfcc8format_type.cpp,lineno216,funcname0x564908abfdd0 format_type_internal(unsigned int,int,bool,bool,bool)::__func__format_type_internal,domain0x5649087e1004plpgsql-9.2)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/share/error/elog.cpp:238 238 in /usr1/GaussDBKernel/server/opengauss/src/auxiliary/share/error/elog.cpp(gdb)接着输入bt查看堆栈(gdb)bt #0 errstart(elevel20,filename0x564908abfcc8format_type.cpp,lineno216,funcname0x564908abfdd0 format_type_internal(unsigned int,int,bool,bool,bool)::__func__format_type_internal,domain0x5649087e1004plpgsql-9.2)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/share/error/elog.cpp:238 #1 0x00005649043cc8ae in format_type_internal(type_oid323,typemod-1,typemod_givenoptimized out,allow_invalidoptimized out,include_nspnameoptimized out)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/share/adt/format_type.cpp:212 #2 0x00005649045b82e8 in format_procedure(procedure_oidoptimized out)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/share/adt/regproc.cpp:473 #3 0x0000564904a2d53d in do_compile(fcinfo0x7f6a520460c0,proc_tup0x7f69b7c375a0,func0x7f6a39c64050,compile_func_head_info0x7f6a52046740,for_validatortrue,hashkey0x7f6a52045d50)at /usr1/GaussDBKernel/server/opengauss/src/gausskernel/pl/plsql/pl_comp/pl_comp_func_main.cpp:921 #4 0x0000564904a35f3b in gsplsql_compile(fcinfo0x7f6a520460c0,compile_func_head_info0x7f6a52046740,for_validatortrue,isRecompilefalse,func_runtime_state0x0)at /usr1/GaussDBKernel/server/opengauss/src/gausskernel/pl/plsql/pl_comp/pl_comp_func_main.cpp:3106 #5 0x0000564906c64eeb in plpgsql_validator(fcinfooptimized out)at /usr1/GaussDBKernel/server/opengauss/src/compatibility/sql_adaptor/pl/plpgsql/src/pl_handler.cpp:1481 #6 0x00005649048a56cb in OidFunctionCall4Coll(function_id10790,collation0,arg197664,arg20,arg30,arg4140094619281216,is_null0x0)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/share/fmgr/fmgr.cpp:2512 #7 0x0000564904a55363 in gsplsql_func_in_pkg_compile(pkg0x7f6a455ec050)at /usr1/GaussDBKernel/server/opengauss/src/gausskernel/pl/plsql/pl_comp/pl_comp_pkg_main.cpp:1210 #8 0x0000564904a571fc in gsplsql_pkg_init(pkg0x7f6a455ec050,isCreatefalse,isSpecfalse,ret_pkg_runtime0x7f6a52046a18,is_need_compile_functrue,pkg_debug_query_stringoptimized out,old_pkg_runtime0x7f6a39a48050)at /usr1/GaussDBKernel/server/opengauss/src/gausskernel/pl/plsql/pl_comp/pl_comp_pkg_main.cpp:1545 #9 0x0000564904a584a3 in gsplsql_pkg_compile(pkg_oid97663,for_validatortrue,is_specfalse,is_createfalse,is_recompiletrue,pkg_runtime_state0x0)at /usr1/GaussDBKernel/server/opengauss/src/gausskernel/pl/plsql/pl_comp/pl_comp_pkg_main.cpp:956 #10 0x0000564905100dc6 in recompile_single_package(package_oid97663,is_specfalse)at /usr1/GaussDBKernel/server/opengauss/src/compatibility/sql_adaptor/commands/packagecmds.cpp:329 #11 0x0000564905101212 in recompile_package_by_oid(pkg_oid97663,recompile_invalid_pkgfalse)at /usr1/GaussDBKernel/server/opengauss/src/compatibility/sql_adaptor/commands/packagecmds.cpp:416 #12 0x0000564905101262 in recompile_package(stmt0x7f6a3f4954c0)at /usr1/GaussDBKernel/server/opengauss/src/compatibility/sql_adaptor/commands/packagecmds.cpp:437 #13 0x0000564905431abe in sqlcmd_standard_process_utility(parse_tree0x7f6a3f4954c0,query_string0x7f6a3f496050alter package pkg_test_4 compile,params0x0,is_top_leveloptimized out,dest0x56490a6e0720 donothingDR,sent_to_remoteoptimized out,completion_tag0x7f6a5204a430,isCTASfalse)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/proc/tcop/utility.cpp:6813 #14 0x00007f6b9cd9b759 in gsaudit_ProcessUtility_hook(parsetree0x7f6a3f4954c0,queryString0x7f6a3f496050alter package pkg_test_4 compile,params0x0,isTopLeveloptimized out,dest0x56490a6e0720 donothingDR,sentToRemoteoptimized out,completionTag0x7f6a5204a430,isCTASfalse)at /usr1/GaussDBKernel/server/opengauss/src/gausskernel/security/security_plugin/security_policy_plugin.cpp:856 #15 0x00005649059a0f52 in audit_process_utility(parsetree0x7f6a3f4954c0,query_string0x7f6a3f496050alter package pkg_test_4 compile,paramsoptimized out,is_top_leveloptimized out,destoptimized out,sent_to_remoteoptimized out,completion_tag0x7f6a5204a430,is_ctasfalse)at /usr1/GaussDBKernel/server/opengauss/src/gausskernel/security/audit/security_auditfuncs.cpp:1512 #16 0x000056490543c71d in sqlcmd_process_utility(parse_tree0x7f6a3f4954c0,query_string0x7f6a3f496050alter package pkg_test_4 compile,params0x0,is_top_leveloptimized out,destoptimized out,sent_to_remoteoptimized out,completion_tag0x7f6a5204a430,isCTASfalse)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/proc/tcop/utility.cpp:1974 #17 0x000056490541d83f in PortalRunUtility(portal0x7f6a48878050,utilityStmt0x7f6a3f4954c0,isTopLeveltrue,dest0x56490a6e0720 donothingDR,completionTag0x7f6a5204a430)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/proc/tcop/pquery.cpp:2140 #18 0x000056490541f0be in PortalRunMulti(portal0x7f6a48878050,isTopLeveltrue,dest0x56490a6e0720 donothingDR,altdest0x56490a6e0720 donothingDR,completionTag0x7f6a5204a430,snapshot0x0,bii_state0x0)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/proc/tcop/pquery.cpp:2326 #19 0x00005649054232dc in PortalRun(portal0x7f6a48878050,count200,isTopLeveltrue,dest0x7f6a3f4a84d0,altdest0x7f6a3f4a84d0,completionTag0x7f6a5204a430,snapshot0x0,bii_state0x0)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/proc/tcop/pquery.cpp:1501 #20 0x00005649054158af in exec_execute_message(max_rows200,portal_name0x7f6a3f4a8050)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/proc/tcop/postgres.cpp:7071 #21 gs_process_command(firstcharoptimized out,input_messageoptimized out,send_ready_for_queryoptimized out)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/proc/tcop/postgres.cpp:12314 #22 0x000056490541b9c0 in PostgresMain(argcoptimized out,argv0x7f6a49e45b20,dbnameoptimized out,usernameoptimized out)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/proc/tcop/postgres.cpp:11313 #23 0x000056490539f2df in backend_run(port0x7f6a5204a890)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/proc/postmaster/postmaster.cpp:12482 #24 0x00005649053de1b0 in gauss_db_worker_thread_main(knl_thread_role)2(argoptimized out)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/proc/postmaster/postmaster.cpp:19086 #25 0x000056490539f39a in internal_thread_func(argsoptimized out)at /usr1/GaussDBKernel/server/opengauss/src/auxiliary/proc/postmaster/postmaster.cpp:20196 #26 0x00007f6bbc60ff1b in ??()from /usr/lib64/libpthread.so.0 #27 0x00007f6bbc547320 in clone()from /usr/lib64/libc.so.6可以看到在报错的 format_type_internal里出现了type_oid323的确是在报错中出现的数字但是323这么小的数字明显不可能是用户自定义类型因为小数字的oid都是被系统保留的。这里肯定是有bug的但没有源码不方便找bug原因本次调试的主要目的是找到出错的procedure。在 format_type_internal和 format_procedure里有用的参数都显示成 了这表示内核把这些变量优化掉了不给看。于是继续看下一行 do_compile打印几个参数看看(gdb) f 3#3 0x0000564904a2d53d in do_compile (fcinfo0x7f6a520460c0, proc_tup0x7f69b7c375a0, func0x7f6a39c64050,compile_func_head_info0x7f6a52046740, for_validatortrue,hashkey0x7f6a52045d50) at/usr1/GaussDBKernel/server/opengauss/src/gausskernel/pl/plsql/pl_comp/pl_comp_func_main.cpp:921921/usr1/GaussDBKernel/server/opengauss/src/gausskernel/pl/plsql/pl_comp/pl_comp_func_main.cpp:No such file or directory. (gdb) p *fcinfo $1 {flinfo 0x7f6a52046020, context 0x0, resultinfo 0x0, fncollation 0,isnull false, nargs 0, arg 0x7f6a39a4ca50, argnull 0x0,argTypes 0x0, prealloc_arg {0 repeats 20 times},prealloc_argnull {false repeats 20 times}, prealloc_argTypes {0repeats 20 times}, argVector 0x0, refcursor_data {argCursor 0x0, returnCursor 0x0, return_number 0}, out_tmtype 0 ‘\000’,out_decimals 0 ‘\000’, udfInfo {UDFArgsHandlerPtr 0x0,UDFResultHandlerPtr 0x0, udfMsgBuf 0x0, msgReadPtr 0x0,argBatchRows 0, allocRows 0, arg 0x0, null 0x0, result 0x0,resultIsNull 0x0, valid_UDFArgsHandlerPtr false}, swinfo {sw_econtext 0x0, sw_exprstate 0x0, sw_is_flt_frame false},out_typmode 0x0, fn_typmode 0, plfunc_exec_mode 0,plfunc_exec_state 0x0, args_done 0x0, prealloc_args_done {0repeats 20 times}, arginfo {{in_tmtype 0 ‘\000’, in_decimals 0‘\000’, argTypModes 0, set_enum_typeoid 0}}} (gdb) p *proc_tupAttempt to dereference a generic pointer. (gdb) p *func $2 {type T_PLpgSQL_FUNCTION, fn_oid 97664, pkg_oid 97663, namespaceOid 2200, fn_owner 16728, fn_input_collation 0, fn_signature 0x0,fn_searchpath 0x0, namespace_searchpath 0x0, fn_hashkey 0x0,fn_cxt 0x7f6a6a9599d0, fn_rettype 0, fn_rettyplen 0,glc_func_life 1, fn_rettypioparam 0, fn_retbyval false,fn_retistuple false, fn_retset false, fn_readonly false,out_param_varno -1, found_varno 0, fn_nallargs 0, argmods 0x0,argtypes 0x0, sql_cursor_found_varno 0, sql_notfound_varno 0,sql_isopen_varno 0, sql_rowcount_varno 0,sql_bulk_exceptions_varno 0, sqlcode_varno 0, sqlstate_varno 0,sqlerrm_varno 0, new_varno 0, old_varno 0, tg_name_varno 0,tg_when_varno 0, tg_level_varno 0, tg_op_varno 0, tg_relid_varno 0, tg_relname_varno 0, tg_table_name_varno 0, tg_table_schema_varno 0, tg_nargs_varno 0, tg_argv_varno 0,retvarno 0, guc_stat 5, use_count 0, resolve_option GSPLSQL_RESOLVE_COLUMN, ndatums 0, datums 0x0, datum_need_free 0x0, action 0x0, goto_labels 0x0, invalItems 0x0,saved_unique_id 4294967295, nPlaceHolders 0, placeholders 0x0,cur_estate 0x0, tg_relation 0x0, debug 0x0, ns_top 0x0,is_private false, fn_is_trigger false, pre_parse_trig false,is_autonomous false, is_inline_handler false, is_valid true,is_plpgsql_func_with_outparam false, need_skip_process_autonm_pkg false, remembered_by_resowner false, typeList 0x0, namespace_name 0x0, expr_list 0x0, fn_retinput {fn_addr 0x0, fn_oid 0, fn_nargs 0, fn_strict false, fn_retset false, fn_extra 0x0,fn_mcxt 0x0, fn_expr 0x0, fn_rettype 0, fn_rettypemod 0,fnName ‘\000’ repeats 63 times, fnLibPath 0x0, vec_fn_addr 0x0, vec_fn_cache 0x0, genericRuntime 0x0, max_length 0,fn_languageId 0, fn_stats 0 ‘\000’, fn_fenced false, fn_volatile 0 ‘\000’, decimals 0 ‘\000’}, glc_status {m_type GLC_FUNCTION_OBJ, m_location GLC_OBJECT_IN_SESSION_WAIT_REMOVE,m_glc_object_state GLC_OBJECT_IS_VALID, m_refcount 1},expired_cell {dle_next 0x0, dle_prev 0x0, dle_val 0x7f6a39c64050, dle_list 0x7f6a48876ea0}, compiled_dlist_elem {dle_next 0x0, dle_prev 0x0, dle_val 0x7f6a39c64050, dle_list 0x0}, parent_pro_ndatum 0, subparam 0x0, fn_nargs 0,copiable_size 0, deep_datums 0x0, deep_ndatums 0, cursor_datums 0x0, cursor_ndatums 0, placeholder_datums 0x0, placeholder_ndatums 0, fn_argvarnos 0x0, depend_info_list 0x0,plan_total_mem_size 0, block_level 0x0} (gdb)可以在 func里看到fn_oid97664 这意味着是在编译pg_proc里oid为97664的对象。于是我们输入q退出gdb然后回到客户端查询gaussdb# select proname,g.pkgname from pg_proc p,gs_package g where p.oid97664 and g.oidp.propackageid;proname | pkgname --------------------- p1 | pkg_test_4(1 row)可以看到这个oid对应的就是pkg_test_4这个包里的p1于是就知道了一定是编译p1的时候出了问题。到此出问题的procedure就直接找出来了肉眼一看包头和包体的定义发现有个参数的in/out方向没匹配但GaussDB在创建这个package时竟然没有报错…其他Gauss系数据库的情况同样的这个代码在openGauss 7.0.0 RC1 是不会报错的这个package还能正常调用查了下数据字典出入参方向是按包体生效的,这同样也是个BUG没有做严格判断。openGauss# create package pkg_test_4 is openGauss$# procedurep1(i1 in varchar2,openGauss$# i2 out varchar2,openGauss$# i3 out varchar2);openGauss$# end pkg_test_4;openGauss$# /;end;end pkg_test_4;/CREATE PACKAGE openGauss# create package body pkg_test_4 is openGauss$# procedurep1(i1 in varchar2,openGauss$# i2 in varchar2,openGauss$# i3 out varchar2)is openGauss$# begin openGauss$# null;openGauss$# end;openGauss$# end pkg_test_4;openGauss$# / CREATE PACKAGE BODY openGauss# alter package pkg_test_4 compile;ALTER PACKAGE openGauss# call pkg_test_4.p1(null,null,null);i3 ----(1 row)openGauss#在MogDB 5.2.0里则是在创建package body时就报错了能正确检查到包头里的procedure在包体里没定义MogDB# create package pkg_test_4 is MogDB$# procedurep1(i1 in varchar2,MogDB$# i2 out varchar2,MogDB$# i3 out varchar2);MogDB$# end pkg_test_4;MogDB$# / CREATE PACKAGE MogDB# create package body pkg_test_4 is MogDB$# procedurep1(i1 in varchar2,MogDB$# i2 in varchar2,MogDB$# i3 out varchar2)is MogDB$# begin MogDB$# null;MogDB$# end;MogDB$# end pkg_test_4;MogDB$# /ERROR:Function definition notfound:p1 MogDB#总结本次触发 ERROR: Failed to query the 323 type in the cache.这个报错的直接原因是创建的package和package body中有个procedure的参数in/out方向不匹配导致。虽然客户代码的确有问题但根本原因还是数据库有BUG未将这种异常场景检查出来。