Meet PICcy, your friendly ghost. :)

I know you said you configured all unused pins as outputs, but this
still looks very much like you getting interference through a floating
input. Double check that. Also check RB4.
Could be your breadboard is having a bad connection.
Do you have a pullup/pulldown resistor on your switch input?
As a test, make all unused pins as input and connect them to ground.
connect the case of your Xtal to ground.
And, I don't think it is your problem, but you don't need the 74HC04 to
drive the LED, you can do that direct from the PIC pin via a 680 ohm
resistor.

Just a few things I can think of.
Quentin