If you find yourself accidentally changing objects, try one of the following:
1) Turn off Automatically Select New Objects.
2) Turn on Freeze Properties of Selected Objects.
3) Remember to press Esc or Ctrl+N before you set colors
Each of the methods is a little bit different; you may have to experiment with them to see which works best for you.