Bristleworms are a good member of your clean up crew as they are very efficient detritus eaters. Their population will rise and fall based upon how much food is available in the tank for them to eat. They will not harm the Xenia or other corals in your system either. I would put the rock back in the display tank. You are at more risk of getting bristled by picking up and moving the rock he lives in than the coral is of being damaged by the worm.
They may be a little scary to look at, but until they get to be about 2 feet long you shouldn't need to worry about them.
