[Resolved] Server crash when players with unknown species join.

Resolved as of 1.4.3 beta 2.

Dedicated servers are rendered inoperable until restarted when a player with an unknown species attempts to join. When a player with an unknown species (i.e. not installed server-side) joins the server, MapException errors with the species' name are thrown until the client disconnects. After this event, all subsequent connections will cause the same error, regardless of player species, until the server is manually restarted. Since this occurs very early in the connection process, no built-in acess controls can be used to mitigate the issue.
TL;DR: Anyone can crash any server, even if they're banned, by simply installing one of the hundreds of species mods and joining the server.

A starbound_server.log depicting the issue is attached.
This setup was used:

• A Manjaro Linux installation. (amd64, 16.06.1)
  • Also observed on Ubuntu Xenial (amd64, 16.04.1 LTS)
• A vanilla starbound_server running on localhost. (Any version from 1.0.0 and up)
  • "allowAssetsMismatch" set to true in starbound_server.config.
• A client with the Familiars and Xbawks mods installed.
• A vanilla Glitch character. (Name: "devBot", UUID : "ff00ff00ff00ff00ff00ff00ff00ff00")
• A modded Familiar character (Name: "Bogon", UUID : "deadbeefdeadbeefdeadbeefdeadbeef")
  • The character must have completed the initial quest. Otherwise, they will simply receive a client shipworld error.

These steps were used:

1. Start the vanilla server.
2. Join with the vanilla character. ("devBot")
   
3. Disconnect from the server.
4. Attempt to join with the modded character. ("Bogon")
5. Send a SIGKILL to the client. - Could not disconnect otherwise.
6. Attempt to join with the vanilla character again. ("devBot")
7. Send a SIGKILL to the client. - Could not disconnect otherwise.
8. Send a SIGINT to the server.

Setting "allowAssetsMismatch" to false in starbound_server.config does mitigate this issue. However, this may not be an option for some servers - mine included.

Edit: When a server experiences this error, it will continue to complete TCP handshakes and respond to RCON commands. This means very little for clients but becomes fairly significant for monitoring and recovery scripts. Unless the monitoring/recovery script watches for this error, it will likely return a false reading and assume the server is functioning correctly.

Edit: I've written a patch for the StarryPy3k server wrapper allowing species whitelisting and it has since gone upstream. StarryPy3k is available here.

 

I'd say "bump". As I've also suffered the same problems with a Windows 8.1 64-bit dedicated server. It seems when ever a modded race joins a server that doesn't have one, or indeed joins a modded server that doesn't have that race, then if the "allowAssetsMismatch" is set to true. Then clients can use this to half-crash the server with the race the server doesn't have installed, as it will make the server unusable until that person leaves, but they most likely won't leave as the protectorate logo on the bottom right is still moving and they get a blank screen making people feel it takes a long time to connect, when in reality they're keeping the server frozen by trying to join. I believe it is necessary this is patched because servers that like to allow client-side mods, will otherwise be forced to only allow specific assets on the server meaning that only the mods the server has could be used to join, which is the effect of "allowAssetsMismatch" being set to false.

 

A little more info on the issue:
Although RCON commands are still processed, attempting to kick a "ghosted" player with an unknown species does not work. Using the same setup as described above and Dunto's RCON client, this is what happens when attempting to kick the problematic player:

Code:

$ python rcon_client.py list
$1 : Bogon : $$deadbeefdeadbeefdeadbeefdeadbeef
$ python rcon_client.py kick \$\$deadbeefdeadbeefdeadbeefdeadbeef
Successfully kicked user with specifier $$deadbeefdeadbeefdeadbeefdeadbeef. ConnectionId: Just (1). Reason given: Bogon
$ python rcon_client.py list
$1 : Bogon : $$deadbeefdeadbeefdeadbeefdeadbeef
$ python rcon_client.py kick \$\$deadbeefdeadbeefdeadbeefdeadbeef
RCON: Error executing: kick $$deadbeefdeadbeefdeadbeefdeadbeef: (MapException) Entry with key '1' already present.
./starbound_server(_ZN4Star13StarExceptionC1EPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x66) [0x95b556]
./starbound_server(_ZN4Star12MapExceptionC1ENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x52) [0x8f4212]
./starbound_server(_ZN4Star8MapMixinISt13unordered_mapItNS_6StringENS_4hashItvEESt8equal_toItESaISt4pairIKtS2_EEEE3addEtS2_+0x9b) [0xede6db]
./starbound_server(_ZN4Star14UniverseServer16disconnectClientEtRKNS_6StringE+0x3b) [0xec667b]
./starbound_server(_ZN4Star16CommandProcessor4kickEtRKNS_6StringE+0x14d) [0xa561ed]
./starbound_server(_ZN4Star16CommandProcessor13handleCommandEtRKNS_6StringES3_+0x4eb) [0xa5917b]
./starbound_server(_ZN4Star16CommandProcessor12adminCommandERKNS_6StringES3_+0x31) [0xa593d1]
./starbound_server(_ZN4Star14UniverseServer12adminCommandENS_6StringE+0x5a) [0xeb7f5a]
./starbound_server(_ZN4Star16ServerRconClient13handleCommandENS_6StringE+0x9d6) [0x84c116]
./starbound_server(_ZN4Star16ServerRconClient14processRequestEv+0x3b3) [0x84cc63]
./starbound_server(_ZN4Star16ServerRconClient3runEv+0x18) [0x84d498]
./starbound_server(_ZN4Star10ThreadImpl9runThreadEPv+0x18) [0x9660a8]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76fa) [0x7f78f8ca16fa]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f78f82c2b5d]
$ python rcon_client.py list
$1 : Bogon : $$deadbeefdeadbeefdeadbeefdeadbeef
$ python rcon_client.py kick \$\$deadbeefdeadbeefdeadbeefdeadbeef
RCON: Error executing: kick $$deadbeefdeadbeefdeadbeefdeadbeef: (MapException) Entry with key '1' already present.
./starbound_server(_ZN4Star13StarExceptionC1EPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x66) [0x95b556]
./starbound_server(_ZN4Star12MapExceptionC1ENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x52) [0x8f4212]
./starbound_server(_ZN4Star8MapMixinISt13unordered_mapItNS_6StringENS_4hashItvEESt8equal_toItESaISt4pairIKtS2_EEEE3addEtS2_+0x9b) [0xede6db]
./starbound_server(_ZN4Star14UniverseServer16disconnectClientEtRKNS_6StringE+0x3b) [0xec667b]
./starbound_server(_ZN4Star16CommandProcessor4kickEtRKNS_6StringE+0x14d) [0xa561ed]
./starbound_server(_ZN4Star16CommandProcessor13handleCommandEtRKNS_6StringES3_+0x4eb) [0xa5917b]
./starbound_server(_ZN4Star16CommandProcessor12adminCommandERKNS_6StringES3_+0x31) [0xa593d1]
./starbound_server(_ZN4Star14UniverseServer12adminCommandENS_6StringE+0x5a) [0xeb7f5a]
./starbound_server(_ZN4Star16ServerRconClient13handleCommandENS_6StringE+0x9d6) [0x84c116]
./starbound_server(_ZN4Star16ServerRconClient14processRequestEv+0x3b3) [0x84cc63]
./starbound_server(_ZN4Star16ServerRconClient3runEv+0x18) [0x84d498]
./starbound_server(_ZN4Star10ThreadImpl9runThreadEPv+0x18) [0x9660a8]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76fa) [0x7f78f8ca16fa]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f78f82c2b5d]

The starbound_server.log from this test is attached as well.

So much for kicking the player to restore connectivity without a full restart.
--- Post updated ---
One more thing to note: The modded character must have completed the initial quest. If the modded character is brand new, they will simply receive a client shipworld error.
--- Post updated ---
This issue appears to have gone unchanged in the most recent update, which is honestly a bit surprising... My server experiences this issue multiple times per day and, although we are a fairly specialized server, I'm sure other communities are being affected as well.

 

Once again, this issue is still present after the most recent update. (1.0.4)

 

Three weeks, at least as many minor versions, and still no fix... Well, if you want something done right, do it yourself.
I've created a patch for the StarryPy3k server wrapper. It will allow an operator to whitelist specific species and intercept unknown ones before they can crash the underlying starbound_server process.

 

This is still present in the latest update (1.1.0).

 

And in 1.1.1 as well.
Does your plugin still works? it seems that unknown species pass through the wrapper on my server (therefore making it crash)

 

Yeah species mods typically crash servers; additionally they can crash players too if the server has it but the player doesn't.

 

It seems to catch most connections on my server but the occasional connection will get through. Sadly, I haven't found a good way of combating this.

 

Yet again, this issue is still present in the latest update.

 

Do I even need to say it anymore? :c This has been an issue for the last 9 revisions, spanning nearly 5 months, and apparently that isn't going to change anytime soon.
Given how much this affects larger servers (every other day, in my case), it really sucks to see this go un-patched for so long.

 

I second this. Even a pretty modest server like mine has this issue at least once a week (resulting a 14mo log file spammed with errors).

 

Fairly late update, but I revisited the issue and it appears StarryPy3k completely mitigates it now. Medeor413 made a small change ~18 days ago that fixed everything.
If anyone still experiences this issue while running a current version of StarryPy3k, please get in touch. I'd like to make sure server operators can effectively deal with this issue... even if that should be ChuckleFish's job.

Also, this bug just reached its 9 month anniversary.

 

Yay, 1.3! Surely this bug that's been causing us grief for over 10 months will be fixed, right?

Code:

[Info] UniverseServer: Connection received from: 0000:0000:0000:0000:0000:0000:0000:0001:47472
[Info] UniverseServer: Logged in account '<anonymous>' as player 'bork?' from address 0000:0000:0000:0000:0000:0000:0000:0001
[Info] UniverseServer: Loading system world (-972429590, 405312653, 59142126) from disk storage
[Info] UniverseServer: Reviving player at ClientShipWorld:352d69c87b08a287c0097634e8b68fdb
[Info] UniverseServer: Client 'bork?' <1> (0000:0000:0000:0000:0000:0000:0000:0001) connected
[Info] UniverseServer: Loading client ship world ClientShipWorld:352d69c87b08a287c0097634e8b68fdb
[Info] Protected dungeonIds for world set to (65524)
[Error] Could not instantiate item '[valublaster, 1, {"primaryAbilityType":"chargefire"}]'. (ItemException) No such item 'valublaster'
[Error] Could not instantiate item '[invishead, 1, {}]'. (ItemException) No such item 'invishead'
[Error] Could not instantiate item '[goggleshead, 1, {}]'. (ItemException) No such item 'goggleshead'
[Error] UniverseServer: exception caught: (MapException) Key 'familiar' not found in Map::get()
./starbound_server(_ZN4Star13StarExceptionC1EPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x66) [0x9e67a6]
./starbound_server(_ZN4Star12MapExceptionC1ENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x52) [0x979422]
./starbound_server() [0x100c60c]
./starbound_server(_ZN4Star17WorldServerThread13executeActionESt8functionIFvPS0_PNS_11WorldServerEEE+0x3c) [0x112c72c]
./starbound_server(_ZN4Star14UniverseServer11updateShipsEv+0x119) [0x1000fc9]
./starbound_server(_ZN4Star14UniverseServer3runEv+0x1f6) [0x100efb6]
./starbound_server(_ZN4Star10ThreadImpl9runThreadEPv+0x15) [0x9f0f15]
/lib64/libpthread.so.0(+0x75ca) [0x7f47f2b2c5ca]
/lib64/libc.so.6(clone+0x6d) [0x7f47f21510ed]
[Error] UniverseServer: exception caught: (MapException) Key 'familiar' not found in Map::get()
./starbound_server(_ZN4Star13StarExceptionC1EPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x66) [0x9e67a6]
./starbound_server(_ZN4Star12MapExceptionC1ENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x52) [0x979422]
./starbound_server() [0x100c60c]
./starbound_server(_ZN4Star17WorldServerThread13executeActionESt8functionIFvPS0_PNS_11WorldServerEEE+0x3c) [0x112c72c]
./starbound_server(_ZN4Star14UniverseServer11updateShipsEv+0x119) [0x1000fc9]
./starbound_server(_ZN4Star14UniverseServer3runEv+0x1f6) [0x100efb6]
./starbound_server(_ZN4Star10ThreadImpl9runThreadEPv+0x15) [0x9f0f15]
/lib64/libpthread.so.0(+0x75ca) [0x7f47f2b2c5ca]
/lib64/libc.so.6(clone+0x6d) [0x7f47f21510ed]
[Error] UniverseServer: exception caught: (MapException) Key 'familiar' not found in Map::get()
./starbound_server(_ZN4Star13StarExceptionC1EPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x66) [0x9e67a6]
./starbound_server(_ZN4Star12MapExceptionC1ENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x52) [0x979422]
./starbound_server() [0x100c60c]
./starbound_server(_ZN4Star17WorldServerThread13executeActionESt8functionIFvPS0_PNS_11WorldServerEEE+0x3c) [0x112c72c]
./starbound_server(_ZN4Star14UniverseServer11updateShipsEv+0x119) [0x1000fc9]
./starbound_server(_ZN4Star14UniverseServer3runEv+0x1f6) [0x100efb6]
./starbound_server(_ZN4Star10ThreadImpl9runThreadEPv+0x15) [0x9f0f15]
/lib64/libpthread.so.0(+0x75ca) [0x7f47f2b2c5ca]
/lib64/libc.so.6(clone+0x6d) [0x7f47f21510ed]
[Error] UniverseServer: exception caught: (MapException) Key 'familiar' not found in Map::get()
./starbound_server(_ZN4Star13StarExceptionC1EPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x66) [0x9e67a6]
./starbound_server(_ZN4Star12MapExceptionC1ENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x52) [0x979422]
./starbound_server() [0x100c60c]
./starbound_server(_ZN4Star17WorldServerThread13executeActionESt8functionIFvPS0_PNS_11WorldServerEEE+0x3c) [0x112c72c]
./starbound_server(_ZN4Star14UniverseServer11updateShipsEv+0x119) [0x1000fc9]
./starbound_server(_ZN4Star14UniverseServer3runEv+0x1f6) [0x100efb6]
./starbound_server(_ZN4Star10ThreadImpl9runThreadEPv+0x15) [0x9f0f15]
/lib64/libpthread.so.0(+0x75ca) [0x7f47f2b2c5ca]
/lib64/libc.so.6(clone+0x6d) [0x7f47f21510ed]
[Error] UniverseServer: exception caught: (MapException) Key 'familiar' not found in Map::get()
./starbound_server(_ZN4Star13StarExceptionC1EPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x66) [0x9e67a6]
./starbound_server(_ZN4Star12MapExceptionC1ENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x52) [0x979422]
./starbound_server() [0x100c60c]
./starbound_server(_ZN4Star17WorldServerThread13executeActionESt8functionIFvPS0_PNS_11WorldServerEEE+0x3c) [0x112c72c]
./starbound_server(_ZN4Star14UniverseServer11updateShipsEv+0x119) [0x1000fc9]
./starbound_server(_ZN4Star14UniverseServer3runEv+0x1f6) [0x100efb6]
./starbound_server(_ZN4Star10ThreadImpl9runThreadEPv+0x15) [0x9f0f15]
/lib64/libpthread.so.0(+0x75ca) [0x7f47f2b2c5ca]
/lib64/libc.so.6(clone+0x6d) [0x7f47f21510ed]

[ad infinitum]

...Why do I even bother?

 

Is this something that can be fixed, really? It seems like the solution is to either use assets mismatch (yes, I know you said you can't) or tell players not to join if they're using a race mod not supported.

 

From ChuckleFish's side, yes; and probably quite easily. Gracefully handling the exception, or even just checking the species in the initial client_connect packet, like StarryPy3k does, should take care of the issue.

From an administrator's side, yes as well... it's just ugly and cumbersome. I know from experience that telling players not to use modded species doesn't really work though. Despite it being well-known that logging in with an unknown species would lead to an automated and permanent ban, my community would experience the crash near daily. The sad fact of the matter is that many players, especially new ones, simply don't pay enough attention for this to be effective. This may be a different story for whitelisted servers, but I expect there would still be issues.
As for asset mismatch, it's pretty much unusable for any large server. The mere act of changing config values in certain locations will change the asset digest, as Starbound stores a lot of config options in its packed assets. Even overlooking this, and assuming an administrator successfully sets a spoofed asset digest, enabling these checks will severely limit the community's freedom. For certain niche servers, this may not be a problem. But it certainly is for many of the larger ones, including mine.
The only existing solution I know of that can prevent this crash 100% of the time, while not severely impeding players, is StarryPy3k's species whitelist. However, using it can still be a pain in the butt - especially if the server uses mods. Every species has to be individually defined in StarryPy3k's config file, for every mod that includes a new species. And that's assuming the administrator is already using StarryPy3k. Otherwise, they would need to set up the whole wrapper from scratch.

Please forgive me if I sound a bit jaded and salty, but spending over 10 months dealing with a bug, that in any other context could be called a DoS vulnerability, just sucks. Moreover, this sort of problem being a common theme for Starbound dedicated servers sucks. It's just painful to see something with such potential but know that it will never come to fruition, as it will always be an afterthought.

 

It looks like this has finally been fixed - nearly a year after its introduction.

 

Apparently I spoke too soon. The bug is still present in all of its gory glory.

 

I've re-implemented the species whitelisting functionalities in StarryPy3k and the patch has since gone upstream. If anyone is having trouble with this bug, this plugin is probably your best bet - as CF have done nothing to fix this for a year and a half. Note that the plugin is disabled by default, so make sure you enable it in the config.

 

This bug is still present in 1.3.4.
It's been over two years.