Yup, I have 4 and the whole screen goes haywire and the game crashes often. I have to leave the farm until afternoon when they're done and then it's ok. Going to remove them soon. It's unplayable with them. I'm on xbox one.
Then you should write a report in the support section of this forum. http://community.playstarbound.com/forums/stardew-support/
Yeah I am on Xbox one also, and I did a hard reset on my console and so far haven't had any more problems.