1. Welcome to the Starbound support forums. Please check the support FAQs before posting: http://playstarbound.com/support

Bug/Issue SEVERE Memory Leak for Server

Discussion in 'Starbound Support' started by Relinies, Jul 23, 2016.

  1. Relinies

    Relinies King Homestuck III

    Information:
    Server box has 16GB of RAM to work with.
    Running Windows Server 2012 Datacenter
    Server usually had 5-6 players playing concurrently, often all on different planets.


    The problem:
    The server generally starts off using 1-2GB of RAM, and quickly rises up to 14GB or more taken up, in well under an hour.
    Lag spikes for players were variable, but seemed to most correlate with memory usage being overly high.

    Any help or advice?
     
  2. combatmedic02

    combatmedic02 Void-Bound Voyager

    Can confirm that this is a major problem. My Windows 2008 R2 server is also suffering from a massive memory leak. I have 6 players in there at the moment and the server is using up 7GB of my 16GB of RAM. And this number just keeps rising and rising as more players join the server. Many errors are also streaming through the consoles such as:
    Code:
    [Warn] Failed to place dungeon object: underwaterlight direction: 1 position: (3461, 933)
    [Warn] Failed to place dungeon object: underwaterlight direction: 1 position: (3546, 933)
    [Warn] Failed to place dungeon object: underwaterlight direction: 1 position: (3631, 933)
    [Warn] Failed to place dungeon object: underwaterlight direction: 1 position: (3716, 933)
    [Warn] Failed to place dungeon object: vaseclassicmedium2 direction: 1 position: (3534, 947)
    [Warn] Failed to place dungeon object: underwaterlight direction: 1 position: (3546, 960)
    [Warn] Failed to place dungeon object: underwaterlight direction: 1 position: (3631, 960)
    [Warn] Failed to place dungeon object: underwaterlight direction: 1 position: (3546, 978)
    [Warn] Failed to place dungeon object: underwaterlight direction: 1 position: (3631, 978)
    [Warn] Failed to place dungeon object: ornatetorch direction: 0 position: (3428, 994)
    [Warn] Failed to place dungeon object: ornatetorch direction: 1 position: (3463, 994)
    [Warn] Failed to place dungeon object: ornatetorch direction: 0 position: (3683, 994)
    [Warn] Failed to place dungeon object: ornatetorch direction: 1 position: (3718, 994)
    Code:
    [Info] Placing dungeon at (0, 1000)
    [Warn] Failed to place dungeon object: lunarbasecrate direction: 1 position: (551, 918)
    [Warn] Failed to place dungeon object: lunarbasecrate direction: 1 position: (551, 921)
    [Warn] Failed to place dungeon object: prisonfloodlight direction: 0 position: (402, 933)
    [Warn] Failed to place dungeon object: miningdoor direction: 0 position: (539, 948)
    [Warn] Dungeon wire endpoint not found. (354, 654)
    [Warn] Dungeon wires did not make a circuit.
    [Warn] Dungeon wire endpoint not found. (528, 830)
    [Warn] Dungeon wire endpoint not found. (531, 830)
    [Warn] Dungeon wire endpoint not found. (352, 655)
    [Warn] Dungeon wires did not make a circuit.
    [Warn] Dungeon wire endpoint not found. (545, 992)
    [Warn] Dungeon wire endpoint not found. (552, 992)
    [Warn] Dungeon wire endpoint not found. (559, 992)
    [Warn] Dungeon wire endpoint not found. (355, 654)
    [Warn] Dungeon wires did not make a circuit.
    [Warn] Dungeon wire endpoint not found. (344, 705)
    [Warn] Dungeon wire endpoint not found. (369, 705)
    [Warn] Dungeon wire endpoint not found. (800, 907)
    [Warn] Dungeon wire endpoint not found. (809, 907)
    [Info] Protected dungeonIds for world set to (0)
    Code:
    [Info] UniverseServer: Expiring temporary world InstanceWorld:outpost:-:-
    [Info] target nil {self: 5700, chatfriend: 5694, reactTarget: 5694, damageSource: 5698, combatTarget: 5698, nearNpc: 5694, filterItem: 5697}
    [Info] Stack from guard
    [Info] [parallel]:1
    [Info] [parallel]:2
    [Info] [sequence]:2
    [Info] [parallel]:3
    [Info] [dynamic]:3
    [Error] Exception while invoking lua function 'update'. (LuaException) Error code 2, [string "/scripts/behavior.lua"]:178: [group] error: [string "/scripts/behavior/bgroup.lua"]:14: (LuaConversionExce
    ption) Error converting LuaValue to type 'int'
    [0] 13f734483
    [1] 13f73320e
    [2] 13f6ff32b
    [3] 13f6fb0b6
    [4] 13f8aba89
    [5] 13f8cbed6
    [6] 13fcf7f7e
    [7] 13fcc921f
    [8] 13fcde460
    [9] 13fcd0ef9
    [10] 13fcff5db
    [11] 13f70205d
    [12] 13f7010f4
    [13] 13f68d838
    [14] 13f6a6622
    [15] 13f68db1b
    [16] 13f68de3f
    [17] 13f68ab9f
    [18] 13f68a878
    [19] 13f68d838
    [20] 13f6a65e1
    [21] 13f68d343
    [22] 13f68db1b
    [23] 13f68d550
    [24] 13f683a64
    [25] 13f7090d0
    [26] 13f8a751e
    [27] 13f8aa5e4
    [28] 13f8aa471
    [29] 13f8ae3fc
    [30] 13f8ae36c
    [31] 13f94eccc
    [32] 13fbd022e
    [33] 13f83eca7
    [34] 13fbef552
    [35] 13fbf74d2
    [36] 13fbf6f38
    [37] 13f7306ee
    [38] 772c59bd BaseThreadInitThunk
    [39] 773fa2e1 RtlUserThreadStart
    
    stack traceback:
            [C]: in ?
            [C]: in function '_ENV.error'
            [string "/scripts/behavior.lua"]:178: in method 'resume'
            [string "/scripts/behavior.lua"]:134: in method 'run'
            [string "/npcs/bmain.lua"]:143: in function <[string "/npcs/bmain.lua"]:108>
    [0] 13f734483
    [1] 13f73320e
    [2] 13f707c41
    [3] 13f8a7537
    [4] 13f8aa5e4
    [5] 13f8aa471
    [6] 13f8ae3fc
    [7] 13f8ae36c
    [8] 13f94eccc
    [9] 13fbd022e
    [10] 13f83eca7
    [11] 13fbef552
    [12] 13fbf74d2
    [13] 13fbf6f38
    [14] 13f7306ee
    [15] 772c59bd BaseThreadInitThunk
    [16] 773fa2e1 RtlUserThreadStart
    [Info] target nil {self: 5694, chatfriend: 5700, reactTarget: 5700, damageSource: 5698, combatTarget: 5698, nearNpc: 5700, filterItem: 5688}
    [Info] Stack from guard
    [Info] [parallel]:1
    [Info] [parallel]:2
    [Info] [sequence]:2
    [Info] [parallel]:3
    [Info] [dynamic]:3
    [Error] Exception while invoking lua function 'update'. (LuaException) Error code 2, [string "/scripts/behavior.lua"]:178: [group] error: [string "/scripts/behavior/bgroup.lua"]:14: (LuaConversionExce
    ption) Error converting LuaValue to type 'int'
    [0] 13f734483
    [1] 13f73320e
    [2] 13f6ff32b
    [3] 13f6fb0b6
    [4] 13f8aba89
    [5] 13f8cbed6
    [6] 13fcf7f7e
    [7] 13fcc921f
    [8] 13fcde460
    [9] 13fcd0ef9
    [10] 13fcff5db
    [11] 13f70205d
    [12] 13f7010f4
    [13] 13f68d838
    [14] 13f6a6622
    [15] 13f68db1b
    [16] 13f68de3f
    [17] 13f68ab9f
    [18] 13f68a878
    [19] 13f68d838
    [20] 13f6a65e1
    [21] 13f68d343
    [22] 13f68db1b
    [23] 13f68d550
    [24] 13f683a64
    [25] 13f7090d0
    [26] 13f8a751e
    [27] 13f8aa5e4
    [28] 13f8aa471
    [29] 13f8ae3fc
    [30] 13f8ae36c
    [31] 13f94eccc
    [32] 13fbd022e
    [33] 13f83eca7
    [34] 13fbef552
    [35] 13fbf74d2
    [36] 13fbf6f38
    [37] 13f7306ee
    [38] 772c59bd BaseThreadInitThunk
    [39] 773fa2e1 RtlUserThreadStart
    
    stack traceback:
            [C]: in ?
            [C]: in function '_ENV.error'
            [string "/scripts/behavior.lua"]:178: in method 'resume'
            [string "/scripts/behavior.lua"]:134: in method 'run'
            [string "/npcs/bmain.lua"]:143: in function <[string "/npcs/bmain.lua"]:108>
    [0] 13f734483
    [1] 13f73320e
    [2] 13f707c41
    [3] 13f8a7537
    [4] 13f8aa5e4
    [5] 13f8aa471
    [6] 13f8ae3fc
    [7] 13f8ae36c
    [8] 13f94eccc
    [9] 13fbd022e
    [10] 13f83eca7
    [11] 13fbef552
    [12] 13fbf74d2
    [13] 13fbf6f38
    [14] 13f7306ee
    [15] 772c59bd BaseThreadInitThunk
    [16] 773fa2e1 RtlUserThreadStart
    [Info] UniverseServer: Loading celestial world -991539372:63074325:-119231392:4
    [Info] UniverseServer: Warping player 31 to CelestialWorld:-991539372:63074325:-119231392:4
    [Info] UniverseServer: Stopping idle world ClientShipWorld:8576a7a801532bafb57f1c93dca7d907
    Not sure if these could be causing the issue but the problem does seem to be effecting other Windows server owners that I have spoken to. Also seems to consume more resources as more players join and leave the server as well. When it gets to the point of using all of the RAM the server has everyone is kicked with an inactivity message and worlds and ships are all unloaded at once but the RAM usage doesn't appear to fall. Respectfully asking if this could be looked into and patched if necessary please.
     
    Last edited: Jul 23, 2016
  3. Keigorw

    Keigorw Void-Bound Voyager

    I'm also having this problem. Running on a Windows 2008 R2 server.
     
  4. Wesnc

    Wesnc Scruffy Nerf-Herder

    Also having this problem on Windows 2008 R2 Server, it never unloads resources from idle worlds. I eventually had the server crash with some spammy errors and had to restart it and the only way to clear this problem is manual restarts (since theres no rcon functionality WITHOUT a wrapper)

    I'm running the server along side another, thinking that it was going to be a fairly low-mem use server since its Starbound but I was wrong and it looks like I need a dedicated box for it now.
     
  5. combatmedic02

    combatmedic02 Void-Bound Voyager

    Don't get a new box just yet, something is definitely wrong here, wait for a response from the developers before doing anything rash.
     
  6. revol

    revol Scruffy Nerf-Herder

  7. wazanator

    wazanator Void-Bound Voyager

    Similar issue on my VPS running Ubuntu. So far we've been able to recreate the crash most often by having multiple players trying to do a mission that creates a one shot dungeon (tutorial, mining facility mission, etc) at the same time.

    I didn't see any dumps/logs generated other than what showed up in the console like the others posted.

    Game runs fine with minimal resource use (2 cores and 2GB of RAM) otherwise.
     
  8. revol

    revol Scruffy Nerf-Herder

    I think I see the same thing waz. I just had a crash a few minutes ago, and somebody said it happened right when they were ending a mission, and teleporting to outpost. I can confirm in my logs that it was caused by this player. It appears it tried to send them to outpost, but that failed, so it sent them to their ship, but that errored out and caused the server to crash.
    --- Post updated ---
    I believe I have concrete proof of a severe memory leak:

    http://community.playstarbound.com/threads/major-hosted-server-stability-issues.118637/
     
  9. leracos

    leracos Subatomic Cosmonaut

    Here to confirm as well. Have 7 of us in my server at present, currently sitting at 5.5 GB of usage after only three or so hours of use. I just restarted it not that long ago to flush out the memory from the morning users. It crashed my whole system last night.
     
  10. MWicker

    MWicker Space Hobo

    I'm also experiencing lag/crashing due to a memory leak, (appears to get worse as you explore more planets) with lots of warnings similar to these:

    Running starbound_server on Ubuntu 14.04 x64. Only have 2gb of RAM, but I'm the only one playing on it. I'm only able to play for a few hours before things become so disjointed that I have to stop/restart the server. Are there any temporary solutions via the starbound config file?

    I have a general feeling that this is a huge issue that won't be easily addressed.. hopefully I'm wrong about that.
     
  11. Hey, just popping in to say that this is being looked into!
     
    Wesnc and LordBob5 like this.
  12. willisterman

    willisterman Space Hobo

    Also seeing this, 27GB of 48GB used on one instance of starbound, and the server is now sat at 100% CPU usage with just one player in it.

    I can probably set up a cron job to restart the server every morning, but still, 27GB seems over the top!
     
  13. HowlingLobo

    HowlingLobo Void-Bound Voyager

    Our server is having the same issue. Went up to 7 GB in about half a day of playing.

    I noticed that when playing in Multiplayer on a dedicated server, crops will grow and silk will be produced on my ship even if nobody is on the ship. When I play on Single Player using the same character, crops and silk will not grow unless I am actually on the ship itself.

    As much as I like my crops growing while I'm away, it seems the issue could be that areas will not unload after a player leaves them.
     
  14. NukerPsych

    NukerPsych Intergalactic Tourist

    Same on Debian 8.2 actually.
    In 5-6 hours server skyrockets RAM usage up-to 16GB of RES ram in top command.
    Maximum online is 13 players, even if no players on server RAM is not clearing.
    Also some SegFaults on world unload.
    [13:48:06.265] [Info] UniverseServer: Stopping idle world CelestialWorld:984090943:-73947600:-72728428:2
    [13:48:06.532] [Error] Fatal Error: Segfault Encountered!
    /root/Starbound/linux/starbound_server(_ZN4Star10fatalErrorEPKcb+0x43) [0x95b563]
    /lib/x86_64-linux-gnu/libc.so.6(+0x350e0) [0x7f7c030d70e0]
    /root/Starbound/linux/starbound_server(_ZNK4Star13SpatialHash2DIifSt10shared_ptrINS_6EntityEEiLm4096EE7forEachINS_10StaticListINS_3BoxIfLm2EEELm2EEERKSt8functionIFvRKS3_EEEEvRKT_OT0_+0x260) [0xac9680]
    /root/Starbound/linux/starbound_server(_ZNK4Star9EntityMap11entityQueryERKNS_3BoxIfLm2EEERKSt8functionIFbRKSt10shared_ptrINS_6EntityEEEE+0x59) [0xac1ed9]
    /root/Starbound/linux/starbound_server(_ZN4Star12WorldStorage9unloadAllEb+0x2ed) [0xfd39ed]
    /root/Starbound/linux/starbound_server(_ZN4Star11WorldServerD1Ev+0x41) [0xfa8821]
    /root/Starbound/linux/starbound_server(_ZN4Star17WorldServerThreadD1Ev+0x2f5) [0xfc8585]
    /root/Starbound/linux/starbound_server(_ZN4Star14UniverseServer22shutdownInactiveWorldsEv+0x8b5) [0xec5e55]
    /root/Starbound/linux/starbound_server(_ZN4Star14UniverseServer3runEv+0xa50) [0xecc120]
    /root/Starbound/linux/starbound_server(_ZN4Star10ThreadImpl9runThreadEPv+0x18) [0x963e78]
    /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4) [0x7f7c03b620a4]
    /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f7c0318a87d]
     
  15. Crushedice

    Crushedice Void-Bound Voyager

    Well, We just Started besides our other servers now also a starbound server on my root server

    Its usually Hard to get the server to Max usage
    Intel Xeon E3-1245 @ 3.40Ghz
    32GB DDR4
    250 GB - SSD


    well... with starbound running , the Cpu is mostly at 99%
    and After half of the day running the ram is full and our other servers get Lags too ....

    i usually started to host cause the 1.0 update amazed me .
    ...
    dont let me regret this
     
  16. NukerPsych

    NukerPsych Intergalactic Tourist

    Tried to run on windows server, looks like memory leak is more severe on windows, and perfomance is also better on linux.
    Phenom 9650 + 8 GB of ram able to run with 10 users without problems for at least 5-6 hours.
    On windows with same online and same config it died after 3 hours.
     
  17. Wesnc

    Wesnc Scruffy Nerf-Herder

    Just to add, it seems this memory leak issue is only on Windows Servers & linux based servers from whats being posted here. Hosting on Windows 7 and 10 with no such memory leak issues to speak of.
     
  18. mausterio

    mausterio Subatomic Cosmonaut

    Also having the same issue. All 8 threads on my server is maxed out at 100% with ./starbound_server using 25.5G Virtual or 12.5G Res
     
  19. wazanator

    wazanator Void-Bound Voyager

    Code:
    [Error] Exception while invoking lua function 'update'. (LuaException) Error code 2, [string "/scripts/behavior.lua"]:185: [selector] error: [string "/scripts/behavior.lua"]:185: [sequence] error: [string "/scripts/behavior.lua"]:178: [receiveClueDialog] error: [string "/scripts/actions/dialog.lua"]:56: bad argument #1 to 'random' (interval is empty)
    
    stack traceback:
            [C]: in ?
            [C]: in function '_ENV.error'
            [string "/scripts/behavior.lua"]:185: in function <[string "/scripts/behavior.lua"]:182>
            (...tail calls...)
            [string "/scripts/behavior.lua"]:134: in method 'run'
            [string "/npcs/bmain.lua"]:143: in function <[string "/npcs/bmain.lua"]:108>
    ./starbound_server(_ZN4Star13StarExceptionC1EPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x66) [0x95aa36]
    ./starbound_server(_ZN4Star12LuaExceptionC1ENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x52) [0x915d02]
    ./starbound_server(_ZN4Star9LuaEngine11handleErrorEP9lua_Statei+0x3f2) [0x90b342]
    ./starbound_server(_ZN4Star9LuaEngine12callFunctionIJfEEENS_7VariantINS2_INS_5EmptyEJbxdNS_6StringENS_8LuaTableENS_11LuaFunctionENS_9LuaThreadENS_11LuaUserDataEEEEJNS_11LuaVariadicIS9_EEEEEiDpRKT_+0xce) [0xb6fe2e]
    ./starbound_server(_ZN4Star16LuaBaseComponent6invokeINS_7VariantINS_5EmptyEJbxdNS_6StringENS_8LuaTableENS_11LuaFunctionENS_9LuaThreadENS_11LuaUserDataEEEEJfEEENS_5MaybeIT_EERKS4_DpOT0_+0xed) [0xb7017d]
    ./starbound_server(_ZN4Star25LuaActorMovementComponentINS_21LuaUpdatableComponentINS_17LuaWorldComponentINS_16LuaBaseComponentEEEEEE6updateINS_7VariantINS_5EmptyEJbxdNS_6StringENS_8LuaTableENS_11LuaFunctionENS_9LuaThreadENS_11LuaUserDataEEEEJfEEENS_5MaybeIT_EEDpOT0_+0x59) [0xb704a9]
    ./starbound_server(_ZN4Star3Npc6updateEm+0x119) [0xc00189]
    ./starbound_server() [0xfa06b8]
    ./starbound_server(_ZN4Star9EntityMap17updateAllEntitiesERKSt8functionIFvRKSt10shared_ptrINS_6EntityEEEES1_IFbS6_S6_EE+0x1f8) [0xac6818]
    ./starbound_server(_ZN4Star11WorldServer6updateEv+0x490) [0xfa5ff0]
    ./starbound_server(_ZN4Star17WorldServerThread6updateENS_19WorldServerFidelityE+0x1da) [0xfc7a2a]
    ./starbound_server(_ZN4Star17WorldServerThread3runEv+0xdc0) [0xfc8e10]
    ./starbound_server(_ZN4Star10ThreadImpl9runThreadEPv+0x18) [0x965588]
    /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f2e3ed35184]
    /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f2e3e35037d]
    
    Saw this in the log thought I would pass it along.
     
  20. Davoodinator

    Davoodinator Void-Bound Voyager

    ive tried both W2012RC2 and Ubuntu Server 16.04LTS with vm that has 16cores(xeon)+24GB of ram. this is my experience.

    on windows i can run 20-25 players cpu usage goes to 100% around 15 players and memory sits at 40% and everyone experiences severe lag

    on linux i can run 50-70 players with 0 lag but after an hour the server fails hard due to ram overload / memory leaks.

    reducing the players to 20-25 on linux seems to keep the server up longer before the memory leak kills it.
     

Share This Page