for layer use raycastOptions.setLayer or alternatively setMask if you want to set the bitmask directly
raycastOptions.setLayer
setMask