Skynet is a lightweight online game framework

我之前在空闲时间一直对ssm分支做测试，优化，在昨天最后一次提交 #1026 后。压测了一天后，内存还是稳定的，之前的内存错误和泄漏已经没有了，这个版本已经趋近稳定。

测试过程发现ssm版本有一个最重要的问题是回收效率（可能有之前的bug影响），我尝试了一个优化方案去减少SSM短字符串产生，有很多临时字符串是不需要进入SSM的，如

• 比如说二进制字符串（skynet.packstring，sproto.encode，string.pack，...)，
• 不会用于比较的字符串(skynet.error传递的字符串)

方法是单独提供一个接口使用luaS_createlngstrobj（对外的接口是lua_pushtmpstring）创建字符串，如果字符串长度大于LUAI_MAXSHORTLEN 那么就和LUA_TLNGSTR表现一致，小于等于LUAI_MAXSHORTLEN就会有问题，但是这些字符串不用于比较所以也没有问题，主要是这些字符串不会进入SSM。

• 最开始的没有统计

• 使用了lua_pushtmpstring替换了lua-sproto中打包，压缩，解压代码中使用的lua_pushlstring 1000机器人压测SSM字符串新增是80万/分钟

• 然后继续优化了 lua-seri中的packstring，string库中添加了一些列接口tpack，tformat（替换对应的接口中的lua_pushlstring） 1000机器人压测SSM字符串新增是18万/分钟

初步达到了效果，但是我今天中午整理代码的我觉得改动的有掉多了，坑肯定更多。不小心用错了都很难发现。但是可不可以支持lua_pushtmpstring和LUA_TLNGSTR的兼容呢。

我能想到的就是修改luaV_equalobj，但是有了这个思路，那么我可以不可以直接支持不同虚拟机产生的LUA_TLNGSTR，那么sharedtable就可以直接使用了，SSM可以不用了，那么就不需要lua_pushtmpstring，上面的优化都不需要了。

中午我试了一下这个想法，把

#define eqshrstr(a,b)	check_exp((a)->tt == LUA_TSHRSTR, (a) == (b))

替换成

#define eqshrstr(a,b)	check_exp((a)->tt == LUA_TSHRSTR, (a) == (b)?1:((isshared(a)||isshared(b))?luaS_eqshrstr(a, b):0))

luaS_eqshrstr直接实现不同指针的短字符串比较 把LUA_TSHRSTR字符串加入了共享来支持sharetable 删除了SSM相关的代码

初步实现了该想法 提交(31bc54c3af413f3ee673aab1062f703b5619ed7f) 分支(https://github.com/hongling0/skynet/tree/sharedshrstr)

@cloudwu 看要不要采纳这个修改，还有没有什么问题，如果采纳的话我再提交pr

skynet更新到最新代码遇到了一个问题。

之前版本我经常使用 ehco "help" | nc localhost port 这样的命令来执行一些 debug 命令。因为 nc 运行完会自动退出，所以socket也就断开了，skynet那边也能正常的关闭掉这条连接。但新的skynet代码运行后，发现通过这样的命令开启的socket连接无法正确断开了。nc的端口这边处于 FIN_WAIT2 状态，skynet的端口处于 CLOSE_WAIT 状态，说明 nc 这边发送了 FIN，skynet 收到了FIN，但是 nc 这边没有收到 skynet 回发的 FIN。怀疑是新加的半连接状态哪里处理的有问题。我是用 debug_console 测试的。其他类型服务开启的端口应该也会有类似问题。

I have seen that no loop is used around a call of the function "pthread_cond_wait". Would you like to reuse anything from my message on the topic "spurious wakeup"?

在网络收发量较大的时候（MMO大量场景广播消息），skynet socket线程那里表面看起来反应缓慢，accept和正常网络收发的反应都有较大延迟，下面描述下：

服务器负荷： us 用户态占比 sy 系统态占比 id 空闲态占比 hi 硬中断 si 软中断 wa 等待IO 16核16G，每核心大致都是 (5-25us) + (5-30sy)，且每核基本都能保持50id，偶尔会有一两个核有5-15si，没发现hi和wa 服务器出入口带宽使用： 出：30Mb/s 入：20Mb/s 内存使用： 较为平稳，关闭了swap

在出现卡点的时候，在socket线程epoll抛ACCEPT事件那里打log，发现在服务器本地nc上去，大概等了5秒才有反应并成功出log，如果是nc其他进程的端口则正常，证明有某些原因导致skynet epoll对事件的侦测出现了延缓，暂不确定是内核提交事件的延缓（系统抑或进程的部分缓冲区的限制？），还是skynet网络收发线程过于繁忙。但如上所述，如果是skynet线程问题，不应该出现16个核心有稳定的空闲态，这里比较费解。

1,发现有一条线程占用CPU到90多左右 PID为21042 2，用pstack看了一下该进程，发现PID为21042的线程的堆栈为 write() _IO_new_file_write() _GI__IO_file_write() buffered_vfprintf() vfprintf() fprintf() send_list_udp() send_list() send_buffer() send_buffer() socket_server_poll() skynet_socket_poll() thread_socket() start_thread() ... 3,用strace查看了一下，发现该网络线程一直在做下面事情 write(2, "socket-server":udp(3974) sendt"..., 58) select(6, [5], NULL, NULL, {0,0}) = 0 (TimeOut) sendto(14, '\0\4\1\0\0\1\0\0\0\0\0\0\3api\10music440\3com\0\0\1".... 34, 0, 0x7fbbfc3ff590, 0) = -1 EINVAL (Invalid argument) 重复上面的输出

4,用strace查出一直在重复的调用sendto失败了 static int send_list_udp(struct socket_server *ss, struct socket *s, struct wb_list *list, struct socket_message result) { while (list->head) { struct write_buffer * tmp = list->head; union sockaddr_all sa; socklen_t sasz = udp_socket_address(s, tmp->udp_address, &sa); int err = sendto(s->fd, tmp->ptr, tmp->sz, 0, &sa.s, sasz); if (err < 0) { switch(errno) { case EINTR: case AGAIN_WOULDBLOCK: return -1; } fprintf(stderr, "socket-server : udp (%d) sendto error %s.\n",s->id, strerror(errno)); return -1; / // ignore udp sendto error

		result->opaque = s->opaque;
		result->id = s->id;
		result->ud = 0;
		result->data = NULL;

		return SOCKET_ERR;

*/ }

	s->wb_size -= tmp->sz;
	list->head = tmp->next;
	write_buffer_free(ss,tmp);
}
list->tail = NULL;

return -1;

} 5，理论上sendto(14, '\0\4\1\0\0\1\0\0\0\0\0\0\3api\10music440\3com\0\0\1".... 34, 0, 0x7fbbfc3ff590, 0)最后一个参数不应为0的，但是调用时最后一个参数却是0，但是一直报Invalid argument错误

6，业务层用有用了dns解析，所以这里用了udp

7,没有更多的信息了，因为是在线上运行，发现有死循环后大致看了一下就把服务器重启了，在线下又很难重现出来

什么东西都自己造个轮子，这样真的好么？代码还没有严格的工程化风格。各种不同含义的int不能用typedef重命名一下？看的不是一般的难受。c里面还硬编码了lua代码。用c++可以不超过2000行代码，用go可以不超过500行代码搞定的事，结果整出数万行代码。

c调用那么多lua vm的代码好么？lua vm和c互相调用，简直和机房的布线一样乱。连个流程文档都没有。

现在为止，没有一个ide支持makefile，该换了。要不要我pr一个cmake？

我有一个 db 服务，服务中有一个连接 mysql 的连接和一个 redis 连接（可忽略），mysql 设置了 interactive_timeout=1800, wait_timeout=1800, 超时连接断开时，服务有打印 socket closed by peer : 127.0.0.1 3306, 在连接断开的情况下，db 服务收到了连续两条消息，都执行 skynet.db.mysql.query

function _M.query(self, query)
    local querypacket = _compose_query(self, query)
    local sockchannel = self.sockchannel
    if not self.query_resp then
        self.query_resp = _query_resp(self)
    end
    return sockchannel:request(querypacket, self.query_resp)
end

服务的协程列表

task :0000001d
thread: 0x7fa9700bc7e8 session: 51482	stack traceback:
	[C]: in function 'coroutine.yield'
	.../framework_one_world/skynet_ext/skynet/lualib/skynet.lua:390: in upvalue 'suspend_sleep'
	.../framework_one_world/skynet_ext/skynet/lualib/skynet.lua:416: in function 'skynet.wait'
	..._world/skynet_ext/skynet/lualib/skynet/socketchannel.lua:127: in upvalue 'pop_response'
	..._world/skynet_ext/skynet/lualib/skynet/socketchannel.lua:177: in function <..._world/skynet_ext/skynet/lualib/skynet/socketchannel.lua:175>
	[C]: in function 'skynet.pcall'
	..._world/skynet_ext/skynet/lualib/skynet/socketchannel.lua:324: in upvalue 'f'
	.../framework_one_world/skynet_ext/skynet/lualib/skynet.lua:280: in function <.../framework_one_world/skynet_ext/skynet/lualib/skynet.lua:252>
thread: 0x7fa974106018 session: 24167	stack traceback:
	[C]: in function 'coroutine.yield'
	.../framework_one_world/skynet_ext/skynet/lualib/skynet.lua:390: in upvalue 'suspend_sleep'
	.../framework_one_world/skynet_ext/skynet/lualib/skynet.lua:416: in function 'skynet.wait'
	..._world/skynet_ext/skynet/lualib/skynet/socketchannel.lua:422: in upvalue 'block_connect'
	..._world/skynet_ext/skynet/lualib/skynet/socketchannel.lua:480: in function <..._world/skynet_ext/skynet/lualib/skynet/socketchannel.lua:479>
	(...tail calls...)
	[C]: in function 'skynet.pcall'
	...p_server/framework_one_world/skynet_ext/lualib/mysql.lua:26: in method 'execute'
	...p_server/framework_one_world/skynet_ext/lualib/mysql.lua:147: in method 'query'
	..._server/framework_one_world/service/dbd/mysqlHandler.lua:26: in function 'mysqlHandler.query'
	.../pp_server/framework_one_world/service/dbd/dbStorage.lua:41: in upvalue 'readFromMysql'
	.../pp_server/framework_one_world/service/dbd/dbStorage.lua:118: in function 'dbStorage.get'
	...er/framework_one_world/service/dbd/strings/stringsDB.lua:75: in method 'get'
	...amework_one_world/service/dbd/strings/stringsHandler.lua:54: in function 'strings.stringsHandler.get'
	...p/pp_server/framework_one_world/service/dbd/luaproto.lua:40: in function <...p/pp_server/framework_one_world/service/dbd/luaproto.lua:31>
	(...tail calls...)
	[C]: in function 'xpcall'
	...d/skynet_ext/lualib/serviceFramework/serviceLauncher.lua:151: in upvalue 'f'
	.../framework_one_world/skynet_ext/skynet/lualib/skynet.lua:253: in function <.../framework_one_world/skynet_ext/skynet/lualib/skynet.lua:252>
thread: 0x7fa984151128 session: 51485	stack traceback:
	[C]: in function 'coroutine.yield'
	.../framework_one_world/skynet_ext/skynet/lualib/skynet.lua:390: in upvalue 'suspend_sleep'
	.../framework_one_world/skynet_ext/skynet/lualib/skynet.lua:416: in function 'skynet.wait'
	..._world/skynet_ext/skynet/lualib/skynet/socketchannel.lua:127: in upvalue 'pop_response'
	..._world/skynet_ext/skynet/lualib/skynet/socketchannel.lua:177: in function <..._world/skynet_ext/skynet/lualib/skynet/socketchannel.lua:175>
	[C]: in function 'skynet.pcall'
	..._world/skynet_ext/skynet/lualib/skynet/socketchannel.lua:324: in upvalue 'f'
	.../framework_one_world/skynet_ext/skynet/lualib/skynet.lua:280: in function <.../framework_one_world/skynet_ext/skynet/lualib/skynet.lua:252>

mysql 对象:

{
	"mysqlObj" = {
		"_server_status" = 2,
		"packet_no" = 1,
		"_server_capabilities" = 3254779903,
		"_server_ver" = "5.7.30-33-log",
		"protocol_ver" = 10,
		"query_resp" = function: 0x7fa9841a70d0,
		"_max_packet_size" = 1048576,
		"sockchannel" = {
			"__dispatch_thread" = thread: 0x7fa9700bc7e8,
			"__authcoroutine" = false,
			"__auth" = function: 0x7fa968001040,
			"__thread" = {},
			"__request" = {},
			"__overload" = false,
			"__sock" = false,
			"__closed" = false,
			"__result" = {},
			"__connecting" = {
				2 = thread: 0x7fa974106018,
			},
			"__host" = "127.0.0.1",
			"__port" = 3306,
			"__wait_response" = thread: 0x7fa9700bc7e8,
			"__result_data" = {},
		},
		"_server_lang" = 45,
	},
}

__connecting 中的协程没有被 wakeup

我在本地把 mysql 连接超时设置为 10s，连接断开后注入一个测试脚本到 db 服，连续 fork 两个协程调用 mysql.query 不是必现，有概率出现

这篇文章的方式好像不适用lua5.4, https://blog.codingnow.com/2016/06/seri_lua_object.html 这个table用is_rawarray判断会认为是数组

local t = {
    [1] = "a",
    [2] = "b",
    [100] = "c",
}

发现 luaH_next函数5.4有所改动: unsigned int asize = luaH_realasize(t);. 上面示例执行时直接从hash part开始了

lua5.4

int luaH_next (lua_State *L, Table *t, StkId key) {
  unsigned int asize = luaH_realasize(t);
  unsigned int i = findindex(L, t, s2v(key), asize);  /* find original key */
  for (; i < asize; i++) {  /* try first array part */
    if (!isempty(&t->array[i])) {  /* a non-empty entry? */
      setivalue(s2v(key), i + 1);
      setobj2s(L, key + 1, &t->array[i]);
      return 1;
    }
  }
  for (i -= asize; cast_int(i) < sizenode(t); i++) {  /* hash part */
    if (!isempty(gval(gnode(t, i)))) {  /* a non-empty entry? */
      Node *n = gnode(t, i);
      getnodekey(L, s2v(key), n);
      setobj2s(L, key + 1, gval(n));
      return 1;
    }
  }
  return 0;  /* no more elements */
}

lua5.3

int luaH_next (lua_State *L, Table *t, StkId key) {
  unsigned int i = findindex(L, t, key);  /* find original element */
  for (; i < t->sizearray; i++) {  /* try first array part */
    if (!ttisnil(&t->array[i])) {  /* a non-nil value? */
      setivalue(key, i + 1);
      setobj2s(L, key+1, &t->array[i]);
      return 1;
    }
  }
  for (i -= t->sizearray; cast_int(i) < sizenode(t); i++) {  /* hash part */
    if (!ttisnil(gval(gnode(t, i)))) {  /* a non-nil value? */
      setobj2s(L, key, gkey(gnode(t, i)));
      setobj2s(L, key+1, gval(gnode(t, i)));
      return 1;
    }
  }
  return 0;  /* no more elements */
}

配置好testmongodb.lua 然后用skynet拉起来 一直连接失败，已确认网络无问题。延迟也可以接受 然跟代码发现， lualib/skynet/db/mongo.lua 162

obj.__sock = socketchannel.channel {
		host = obj.host,
		port = obj.port,
		response = dispatch_reply,
		auth = mongo_auth(obj),
		backup = backup,
		nodelay = true,
	}
	setmetatable(obj, client_meta)
	obj.__sock:connect(true)	-- try connect only	once
	return obj

是不是mongo_auth(obj)在没有连接的情况下就被调用了。 因为obj.__sock:connect(true)在后面才被执行。

说明：1.3 版本的没有这个问题。同样都有用nginx做转发业务。

MSG.close 事件由 lua-netpack.c 中的 lfilter 函数触发，类型是 SKYNET_SOCKET_TYPE_CLOSE 而该事件，是由 skynet_socket_poll 触发，类型是 SOCKET_CLOSE

而 Commit: 56c70dc7ace1fb668debc254b30adddd711f7e48 这次修复 fix socket half close (#1263) 问题有改动到代码。 如下 if (n==0) 的情况

static int
forward_message_tcp(struct socket_server *ss, struct socket *s, struct socket_lock *l, struct socket_message * result) {
	int sz = s->p.size;
	char * buffer = MALLOC(sz);
	int n = (int)read(s->fd, buffer, sz);
	if (n<0) {
		FREE(buffer);
		switch(errno) {
		case EINTR:
			break;
		case AGAIN_WOULDBLOCK:
			skynet_error(NULL, "socket-server: EAGAIN capture.");
			break;
		default:
			// close when error
			force_close(ss, s, l, result);
			result->data = strerror(errno);
			return SOCKET_ERR;
		}
		return -1;
	}
	if (n==0) {
		FREE(buffer);
		if (nomore_sending_data(s)) {
			force_close(ss,s,l,result); 
			return SOCKET_CLOSE;
		} else { 
			s->type = SOCKET_TYPE_HALFCLOSE;
			shutdown(s->fd, SHUT_RD);
			return -1;  // 怀疑此处返回值有问题
		}
	}

	if (s->type == SOCKET_TYPE_HALFCLOSE) {
		// discard recv data
		FREE(buffer);
		return -1;
	}

	stat_read(ss,s,n);

	if (n == sz) {
		s->p.size *= 2;
	} else if (sz > MIN_READ_BUFFER && n*2 < sz) {
		s->p.size /= 2;
	}

	result->opaque = s->opaque;
	result->id = s->id;
	result->ud = n;
	result->data = buffer;
	return SOCKET_DATA;
}

#1053 使用之前加的的ltls实现了websocket ws和wss协议。提供接口如下：

websocket.accept(socket_id, handle, protocol) 接受客户端socket_id发送过来的websocket协议。 handle为通知接口，protocol为ws和wss，默认为ws。

websocket.connect(url, header) 发起一个websocket连接，返回socket id。

websocket.read(id) 以websocket协议读取一个socket id。

websocket.write(id, data, fmt) 以websocket协议写入一个socket id。data为写入的内容，fmt为text或者binary。

websocket.ping(id) 以websocket协议发送ping frame。

websocket.close(id, code, reason) 以websocket协议关闭socket id，同时可选发送错误码code和错误原因reason。

具体的使用示例可以在simplewebsocket.lua代码中查看。

2022-09-27 10:44:02.957961 SKYNET - [:00000012] lua call [0 to :12 : 0 msgsz = 24] error : ./3rd/skynet/lualib/skynet.lua:859: ./3rd/skynet/lualib/skynet.lua:330: ./3rd/skynet/lualib/http/httpc.lua:118: ./3rd/skynet/lualib/http/tlshelper.lua:56: SSL_read error:1

线上游戏服，今早玩家反馈卡顿，上服务器查看情况，发现cpu、负载很低，查看日志发现某服务出现大量的endless,并且当前服务没有任何其他日志输出（其他服务正常）。基于这些情况，执行 “pstack PID“ 查看skynet进程的线程情况发现 Thread 2 异常，具体如下：

Thread 2 (Thread 0x7f19cf5f7700 (LWP 27881)):
0  0x00007f19dad6989c in __lll_lock_wait_private () from /lib64/libc.so.6
1  0x00007f19dacdb45d in _L_lock_121 () from /lib64/libc.so.6
2  0x00007f19dacd9023 in __GI__IO_un_link () from /lib64/libc.so.6
3  0x00007f19dacd7fe8 in __GI__IO_file_close_it () from /lib64/libc.so.6
4  0x00007f19dacd55a9 in freopen64 () from /lib64/libc.so.6
5  0x00000000004263b9 in luaL_loadfilex_ ([email protected]=0x7f189487af88, [email protected]=0x7f18ace7bac0 "scripts/apis/army.lua", [email protected]=0x7f19bb5d5be0 "bt") at lauxlib.c:794
6  0x00000000004271fa in luaL_loadfilex ([email protected]=0x7f189487af88, [email protected]=0x7f18ace7bac0 "scripts/apis/army.lua", [email protected]=0x7f19bb5d5be0 "bt") at lauxlib.c:1232
7  0x000000000042b3d6 in luaB_loadfile (L=0x7f189487af88) at lbaselib.c:322
8  0x0000000000415763 in luaD_precall ([email protected]=0x7f189487af88, [email protected]=0x7f17a5598360, nresults=2) at ldo.c:532
9  0x0000000000422576 in luaV_execute ([email protected]=0x7f189487af88, ci=<optimized out>) at lvm.c:1626
10 0x00000000004158e0 in ccall ([email protected]=0x7f189487af88, func=<optimized out>, [email protected]=-1, [email protected]=1) at ldo.c:577
11 0x00000000004159ca in luaD_call ([email protected]=0x7f189487af88, func=<optimized out>, [email protected]=-1) at ldo.c:587
12 0x0000000000412a40 in lua_pcallk ([email protected]=0x7f189487af88, [email protected]=1, [email protected]=-1, [email protected]=2, [email protected]=2, [email protected]=0x42afd0 <finishpcall>) at lapi.c:1071
13 0x000000000042b07f in luaB_xpcall (L=0x7f189487af88) at lbaselib.c:473
14 0x0000000000415763 in luaD_precall ([email protected]=0x7f189487af88, [email protected]=0x7f17a55981b0, nresults=3) at ldo.c:532
15 0x0000000000422576 in luaV_execute ([email protected]=0x7f189487af88, ci=<optimized out>, [email protected]=0x7f17dc692500) at lvm.c:1626
16 0x0000000000415423 in unroll (L=0x7f189487af88, ud=<optimized out>) at ldo.c:685
17 0x0000000000414b9a in luaD_rawrunprotected ([email protected]=0x7f189487af88, [email protected]=0x415910 <resume>, [email protected]=0x7f19cf5f4f7c) at ldo.c:144
18 0x0000000000415a54 in lua_resume ([email protected]=0x7f189487af88, [email protected]=0x7f19a53d4a08, nargs=<optimized out>, [email protected]=5, [email protected]=0x7f19cf5f4fbc) at ldo.c:788
19 0x00007f19d8ffd05d in lua_resumeX (nresults=0x7f19cf5f4fbc, nargs=5, from=0x7f19a53d4a08, L=0x7f189487af88) at service-src/service_snlua.c:90
20 auxresume (narg=5, co=0x7f189487af88, L=0x7f19a53d4a08) at service-src/service_snlua.c:146
21 timing_resume ([email protected]=0x7f19a53d4a08, [email protected]=1, n=5) at service-src/service_snlua.c:198
22 0x00007f19d8ffd530 in luaB_coresume (L=0x7f19a53d4a08) at service-src/service_snlua.c:217
23 0x0000000000415763 in luaD_precall ([email protected]=0x7f19a53d4a08, [email protected]=0x7f17d8bf8410, [email protected]=-1) at ldo.c:532
24 0x000000000042227b in luaV_execute ([email protected]=0x7f19a53d4a08, ci=<optimized out>) at lvm.c:1656
25 0x00000000004158e0 in ccall (L=0x7f19a53d4a08, func=<optimized out>, nResults=<optimized out>, inc=65537) at ldo.c:577
26 0x0000000000414b9a in luaD_rawrunprotected ([email protected]=0x7f19a53d4a08, [email protected]=0x411370 <f_call>, [email protected]=0x7f19cf5f52c0) at ldo.c:144
27 0x0000000000415c5e in luaD_pcall ([email protected]=0x7f19a53d4a08, [email protected]=0x411370 <f_call>, [email protected]=0x7f19cf5f52c0, old_top=192, ef=<optimized out>) at ldo.c:892
28 0x00000000004129c7 in lua_pcallk ([email protected]=0x7f19a53d4a08, nargs=<optimized out>, [email protected]=-1, [email protected]=0, [email protected]=0, [email protected]=0x42afd0 <finishpcall>) at lapi.c:1059
29 0x000000000042b0f0 in luaB_pcall (L=0x7f19a53d4a08) at lbaselib.c:456
30 0x0000000000415763 in luaD_precall ([email protected]=0x7f19a53d4a08, [email protected]=0x7f17d8bf82a0, nresults=2) at ldo.c:532
31 0x0000000000422576 in luaV_execute ([email protected]=0x7f19a53d4a08, ci=<optimized out>) at lvm.c:1626
32 0x00000000004158e0 in ccall (L=0x7f19a53d4a08, func=<optimized out>, nResults=<optimized out>, inc=65537) at ldo.c:577
33 0x0000000000414b9a in luaD_rawrunprotected ([email protected]=0x7f19a53d4a08, [email protected]=0x411370 <f_call>, [email protected]=0x7f19cf5f5590) at ldo.c:144
34 0x0000000000415c5e in luaD_pcall ([email protected]=0x7f19a53d4a08, [email protected]=0x411370 <f_call>, [email protected]=0x7f19cf5f5590, old_top=48, ef=<optimized out>) at ldo.c:892
35 0x00000000004129c7 in lua_pcallk ([email protected]=0x7f19a53d4a08, [email protected]=5, [email protected]=0, [email protected]=1, [email protected]=0, [email protected]=0x0) at lapi.c:1059
36 0x00007f19ce6115b8 in _cb (context=0x7f19b6d38580, ud=0x7f19a53d4a08, type=20, session=2481, source=136, msg=0x7f19b8021900, sz=241) at lualib-src/lua-skynet.c:75
37 0x00000000004095d6 in dispatch_message ([email protected]=0x7f19b6d38580, [email protected]=0x7f19cf5f5650) at skynet-src/skynet_server.c:276
38 0x000000000040a1ac in skynet_context_message_dispatch ([email protected]=0x7f19da80b300, q=0x7f19290a86c0, [email protected]=1) at skynet-src/skynet_server.c:336
39 0x000000000040a95e in thread_worker (p=<optimized out>) at skynet-src/skynet_start.c:163
40 0x00007f19db955e25 in start_thread () from /lib64/libpthread.so.0
41 0x00007f19dad5bbad in clone () from /lib64/libc.so.6

看线程堆栈，像是线程调用luaL_loadfilex_ 造成的死锁，比较难复现。 skynet 版本1.5，lua版本是5.4.3

skynet版本使用的是2022年5月份的，出现过两次crash。使用最新的skynet版本还是会出现crash

1. 第一次crash，gdb调试结果如下：

Program terminated with signal 11, Segmentation fault. 0 sweep2old (p=0x7f34a6c2ca00, L=0x7f34a5a1d508) at lgc.c:1059 1059 lgc.c: No such file or directory. Missing separate debuginfos, use: debuginfo-install glibc-2.17-326.el7_9.x86_64 libgcc-4.8.5-44.el7.x86_64 libuuid-2.23.2-65.el7_9.1.x86_64 openssl11-libs-1.1.1k-4.el7.x86_64 zlib-1.2.7-20.el7_9.x86_64 (gdb) bt 0 sweep2old (p=0x7f34a6c2ca00, L=0x7f34a5a1d508) at lgc.c:1059 1 atomic2gen (g=0x7f34a5bbc5d0, L=0x7f34a5a1d508) at lgc.c:1276 2 stepgenfull (g=, L=) at lgc.c:1389 3 genstep (g=, L=) at lgc.c:1423 4 luaC_step (L[email protected]=0x7f34a5a1d508) at lgc.c:1692 5 0x00000000004447d9 in luaV_execute ([email protected]=0x7f34a5a1d508, ci=, [email protected]=0x7f34a5a61d00) at lvm.c:1356 6 0x000000000042a7fb in unroll (L=0x7f34a5a1d508, ud=) at ldo.c:717 7 0x000000000042a21a in luaD_rawrunprotected (L=0x7f34a5a1d508, f=0x42c530 , ud=0x7f34ac7d9d7c) at ldo.c:144 8 0x000000000042d870 in lua_resume (L=0x7f34a5a1d508, from=, nargs=4, nresults=0x7f34ac7d9dcc) at ldo.c:822 9 0x00007f34adfe17ec in ?? () 10 0x00007f34a5a61a40 in ?? () 11 0x00007f34a5a1d428 in ?? () 12 0x0000000000000001 in ?? () 13 0x0000000000000000 in ?? ()

2. 第二次crash， gdb调试结果如下：

Program terminated with signal 7, Bus error. 0 0x00007f6aca24afe7 in OPENSSL_sk_dup () from /lib64/libcrypto.so.1.1 Missing separate debuginfos, use: debuginfo-install glibc-2.17-326.el7_9.x86_64 libgcc-4.8.5-44.el7.x86_64 libuuid-2.23.2-65.el7_9.1.x86_64 openssl11-libs-1.1.1k-4.el7.x86_64 zlib-1.2.7-20.el7_9.x86_64 (gdb) bt 0 0x00007f6aca24afe7 in OPENSSL_sk_dup () from /lib64/libcrypto.so.1.1 1 0x00007f6aca582838 in SSL_new () from /lib64/libssl.so.1.1 2 0x00007f6aca7d8994 in ?? () 3 0x00007f6ac7bcebf8 in ?? () 4 0x00007f6ac81c2a88 in ?? () 5 0x00007f6ac7bcebf8 in ?? () 6 0x00007f6aca7d8c72 in ?? () 7 0x0000000000000000 in ?? ()

线上新项目玩家下线后agent出现内存泄漏，稳定4k。精简服务到比较小之后情况变为偶发。 直接拿最新版本skynet默认的agent来测试发现也会偶现(如下01000036、0100003e等服务)，当然概率比我的环境小很多。修改examples/main.lua仅增加启动100个agent服务并立即关闭。

local skynet = require "skynet"
local sprotoloader = require "sprotoloader"

local max_client = 64

skynet.start(function()
	skynet.error("Server start")
	skynet.uniqueservice("protoloader")
	if not skynet.getenv "daemon" then
		local console = skynet.newservice("console")
	end
	skynet.newservice("debug_console",8000)
	skynet.newservice("simpledb")
	local watchdog = skynet.newservice("watchdog")
	skynet.call(watchdog, "lua", "start", {
		port = 8888,
		maxclient = max_client,
		nodelay = true,
	})
	skynet.error("Watchdog listen on", 8888)
	for i = 1, 100 do
		local agent = skynet.newservice("agent")
		skynet.send(agent, "debug", "EXIT")
	end
	skynet.exit()
end)

debug_console查看c内存如下：

bash-4.1# telnet 127.0.0.1 8000
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
Welcome to skynet console
cmem
:01000002	213920
:01000003	47600
:01000004	64240
:01000005	95344
:01000006	144
:01000007	15488
:01000008	76704
:01000009	104848
:0100000a	19344
:0100000b	12048
:0100000c	137360
:0100000d	12800
:0100000e	15712
:0100000f	40352
:01000010	21056
:0100001f	64
:01000036	64
:0100003e	64
:01000078	64
:0100009c	64
:fffffffe	32
:ffffffff	12618368
block	7675
total	13496640
<CMD OK>

根据 https://github.com/cloudwu/skynet/issues/323 中云大指导的方案我在如下函数中增加调试输出，比对结果看着仿佛没有异常

// hook : malloc, realloc, free, calloc

void *
skynet_malloc(size_t size) {
	void* ptr = je_malloc(size + PREFIX_SIZE);
	if(!ptr) malloc_oom(size);
	uint32_t handle = skynet_current_handle();
	fprintf(stderr, "malloc [:%08x] %p %ld\n", handle, ptr, size);
	return fill_prefix(ptr);
}

void *
skynet_realloc(void *ptr, size_t size) {
	if (ptr == NULL) return skynet_malloc(size);

	void* rawptr = clean_prefix(ptr);
	void *newptr = je_realloc(rawptr, size+PREFIX_SIZE);
	uint32_t handle = skynet_current_handle();
	fprintf(stderr, "realloc [:%08x] %p %ld\n", handle, newptr, size);
	if(!newptr) malloc_oom(size);
	return fill_prefix(newptr);
}

void
skynet_free(void *ptr) {
	if (ptr == NULL) return;
	void* rawptr = clean_prefix(ptr);
	uint32_t handle = skynet_current_handle();
	fprintf(stderr, "free [:%08x] %p\n", handle, rawptr);
	je_free(rawptr);
}

void *
skynet_calloc(size_t nmemb,size_t size) {
	void* ptr = je_calloc(nmemb + ((PREFIX_SIZE+size-1)/size), size );
	uint32_t handle = skynet_current_handle();
	fprintf(stderr, "calloc [:%08x] %p %ld\n", handle, ptr, size);
	if(!ptr) malloc_oom(size);
	return fill_prefix(ptr);
}

贴了其中一个有内存问题服务日志，请查阅 https://gist.github.com/JieTrancender/b699b372e7137295ca7a7e846fefb1e2

云大侠，我在某业务场景中需要实现 cluster 组网 skynet 进程都是等价的slave（不是使用master-slave模式） 目前cluser的配置只能声明

    nodename = "nodeaddress"

我期望的输入方式

    slave = {
        "address",
        "address"
    }

loadconfig中config已经规定了cluser的配置格式

    for name,address in pairs(tmp) do
        if name:sub(1,2) == "__" then
            name = name:sub(3)
            config[name] = address
            skynet.error(string.format("Config %s = %s", name, address))
        else
            assert(address == false or type(address) == "string")
            if node_address[name] ~= address then
                -- address changed
                if node_sender[name] then
                    -- reset connection if node_sender[name] exist
                    node_channel[name] = nil 
                    table.insert(reload, name)
                end 
                node_address[name] = address
            end 
            local ct = connecting[name]
            if ct and ct.namequery and not config.nowaiting then
                skynet.error(string.format("Cluster node [%s] resloved : %s", name, address))
                skynet.wakeup(ct.namequery)
            end 
        end 
    end 

#Question: 我想问的是: 业务层不关心目标node在哪的情况下

可以对cluster进行扩展 支持 array吗？

还是自行将cluster包装一次 在wrapper中调度cluster (wrapper 中可以实现load level) ，例如： wrapper.lua:

local config = {
	slave = {
		"slave1",	--cluster config key
		"slave2"
	}
}

local wrapper = {}

function wrapper.call(name, ...)
	local slave = assert(config[name], name)
	local name = slave[math.random(1, #slave)] --load level
	cluserd.call(name, ...)
end

wrapper.call( slave,  ...)

这样就存在一个问题 cluster配置会变得臃肿 name将变成一个无意义的key cluser

    slave1 = "address"
    slave2 = "address"