Re: more complex setups: You can have more than one mask layer in a group. That's where the other Layer Masking options after
+ Add To Mask come into play.
In the most common mask usage, you would choose
+ Add To Mask, which does just that: it adds the solid area of the mask to the solid area of other mask layers.
The next option
+ Add To Mask, But Keep Invisible does the same but makes the mask layer itself invisible.
- Subtract From Mask cuts away at other masks. For example, if the main mask at the bottom is a big circle, and this mask was a smaller circle on top, it would make the complete mask effect look like a donut.
The next two
+ Clear options let you add a mask to affect the layers above them but without affecting the Masks below.
It's possible to use groups as masks but these may not display correctly in Moho. When possible, it's usually better to simplify the setup by doing the masking inside of the sub-groups, even if it means duplicating the mask layers there.
When the mask doesn't display correctly in the Workspace, use Ctrl+R to check the render results. If this looks fine, I just ignore the display error and continue animating. In my rigs, this error most commonly occurs with eye group layers drifting outside of the head. Check my
Puss rig demo to see an example of that.
Hope this helps.