Blot, don't rub, with your eraser. Rubbing colored pencil will do more harm than good, grinding colors into the page rather than removing them. For more control and a cleaner page, press a kneaded eraser like this one on the area and then carefully peel it off, the same way you did with Silly Putty and the Sunday comics as a kid.
As with crayons, different quality pencils will give you different results. You can also try watercolor pencils, then brush a tiny bit of water over the coloring. As an optional step to help preserve the coloring and make it more colorfast, brush a thin layer of textile medium over the colored areas. Follow the manufacturer's instructions. Place a piece of scrap fabric over the design and press with a hot iron to set the color.