Blot, don't rub, with your eraser. Rubbing colored pencil will do more harm than good, grinding colors into the page rather than removing them. For more control and a cleaner page, press a kneaded eraser like this one on the area and then carefully peel it off, the same way you did with Silly Putty and the Sunday comics as a kid.
Often, when I am in the mood to produce a highly finished piece I will look for a themed coloring page, like an animal, ocean or floral scene. The idea of taking that black and white scene and transforming it into a colorful piece of art motivates me to produce my best coloring.