I've been trying to use masking to make it so the eyeball in this picture covers the Pupil when it changes expression. It worked fine without the switch layer, but when I moved the eyeball into the "Iris" switch layer, the masking stopped working.
Another odd thing is that if I select a layer of the eyeball where part of the pupil is covered, the masking works, but when I switch to a different layer, the Pupil is in front of the eyeball again.
When working with masks I always check the following if something like this happens:
What does the preview look like?
Does it look ok if you turn' GPU acceleration' off in the display quality?
Yes, check the Render result and GPU settings as stated above.
Here are some other things you can do:
When I need the eye to look better in the workspace preview, what I like to do is place the mask in a group inside the Switch. So basically, inside each eye group, I have the eye shape layer acting as a mask for the Iris. Additionally, I use a separate animated 'eye lids' layer to overlap the eyeball without masking. This arrangement seems to display correctly in OGL and renders the same way. Then I can still use the Head layer to mask the Eye Switch in turns, and Moho will mostly handle that properly. (This last part might only mask out the iris and eye lids in OpenGL, but it doesn't affect animation so I just ignore it.)
The downside is that I don't think you can't use interpolation between different eye shapes if each is inside its own group. I don't normally use that feature but if that's important for your setup, ignore this post.
As a general rule of thumb, Moho doesn't like nested masks but sometimes you can work around the issue by containing masks inside their own groups. This can mean making multiple references of masks. If I have to do that, I'll append the name of the references with (REF) so I'll know not to edit those layers directly. Note that masking will usually render correctly even without this additional grouping--I only do this when I really need to see the result correctly (or near-correctly) during animation.
Here's another trick: Occasionally, I'll make a second copy of the element I'm having trouble with in OGL, and I'll set up that copy to display correctly but not render, and I'll hide the render version using the appropriate options in Layer Settings. So, for example, if you need the interpolation between switch layers, you might set up one version of the switch that interpolates and renders the way you want, and a second version that doesn't interpolate but looks more correct for OGL but doesn't render.
Or...if the character renders correctly, and the display errors don't affect the ability to animate the character, I just ignore the display errors and move on.