If you find yourself accidentally changing objects, try one of the following:

1) Turn off Automatically Select New Objects.

2) Turn on Freeze Properties of Selected Objects.

3) Remember to press Esc or Ctrl+N before you set colors

Each of the methods is a little bit different; you may have to experiment with them to see which works best for you.