Do you really need to use RS232? You may be better to use RS422 where the driver
chips are designed to be multidropped in the way you describe, and can be
tri-stated for exactly this reason.

RS232 was never designed as a multidrop system, and the only time I have seen it
done was with open collector transistors to drive the line, but it was messy.
RS422 also has the advantage it is a 5V system, but is differential lines.

Should not be a problem as if you use the pair of wires you would have used as
the transmit/receive wires in RS232 as a differential bi-directional pair in
RS422 there is no extra wires.