By 'mask', do you mean hide the bones? If so, you should use Hide Selected Bones or Shy Bones. To invert a selection, use Select Inverse or Ctrl-I.
So, to hide 'unselected' bones:
1. Select a bone
2. Press Ctrl-I
3. Select Hide Selected Bones or enable the Shy option.
When setting up character rigs, I like to use Shy instead of Hide because it's easy to toggle the visibility of 'shy-ed' bones if you set a hotkey for it. (I use the Shy button on
my MQC panel for this.) I do this for bones that I don't want to touch during animation, but that I may occasionally need to reveal when editing the rig.