Masking Woes: Editing and Rendering Discrepancies

[AirStyle]

I've been having a problem lately with the masking functionality for AS Pro 11. Sorry I don't have the newest version, but I'm poor now, and have to wait before I can get the upgrade. Anyway, the main problem I've been having is that the masking seems to be a little weird, in that it's not rendering what I'm seeing in the editor window. I have some pictures here describing this: what it looks like in the editor, what my layer and masking choices were, and the final render preview.


Can o' Worms: Hide All
Label: No masking in group, Mask this layer
Worm: Hide All, Don't mask this layer
Pupils: Mask this layer
Eyes: Clear mask, then add this to mask
Smile: Mask this layer
Body: Mask this layer
Can: Clear mask, then add this to mask
----------------------------------------------------------------------------------------------


Can o' Worms: Hide All
Label: No masking in group, Mask this layer
Worm: Hide All, Don't mask this layer
Pupils: Mask this layer
Eyes: Clear mask, then add this to mask
Smile: Mask this layer
Body: Don't mask this layer
Can: Add to mask (sorry, I can't remember what I did for this one)
----------------------------------------------------------------------------------------------


Can o' Worms: Hide All
Label: Hide All, Mask this layer
Worm: Hide All, Mask this layer
Pupils: Mask this layer
Eyes: Clear mask, then add this to mask
Smile: Mask this layer
Body: Don't mask this layer
Can: Clear mask, then add this to mask

I'm really not sure what I'm doing wrong, but the two problems I'm having is:
1. I just want to mask the can's label so that the worm's body doesn't bleed over it into the rest of the can, like in the editor window of the third picture (which was perfect)
2. The editor does not accurately show what happens in the render view.

So what should I do to get this when I render it? Perhaps I'm making it more complicated than it needs to be?

[synthsin75]

Masking works on a per group basis. This means that whatever masking a nested group does has its result passed to the parent group as if it were a merged image.

Can o' Worms: Hide All
- Label: No masking in group, Mask this layer
- - Worm: Hide All, Don't mask this layer
- - - Pupils: Mask this layer
- - - Eyes: Clear mask, then add this to mask
- - - Smile: Mask this layer
- - - Body: Mask this layer
- Can: Clear mask, then add this to mask
----------------------------------------------------------------------------------------------

Clearing the Can mask does nothing, unless a layer below it adds to the Can o' Worms mask.
The Label group is added, and this is all you need to make the Can layer mask everything in the Label group.
The only part of the worm added to the mask is the Eyes that mask the Pupils...so the eyes are all you see when rendered. Clearing the mask does nothing here either.

Can o' Worms: Hide All
- Label: No masking in group, Mask this layer
- - Worm: Hide All, Don't mask this layer
- - - Pupils: Mask this layer
- - - Eyes: Clear mask, then add this to mask
- - - Smile: Mask this layer
- - - Body: Don't mask this layer
- Can: Add to mask (sorry, I can't remember what I did for this one)
----------------------------------------------------------------------------------------------

By not masking the Body, that allows it to show without adding the Smile layer, and be masked by the Can acting on the Label group.
Again, the Eyes only mask in the Pupils.

What you need to do is this:

Can o' Worms: Hide All
- Label: Hide All, Mask this layer
- - Worm: Hide All, Mask this layer
- - - Pupils: Mask this layer
- - - Eyes: Clear mask, then add this to mask
- - - Smile: Mask this layer
- - - Body: add to mask
- Can: add to mask

[AirStyle]

Thanks for responding.
Ok. So I'm a really confused.

I get what you mean by the layer effect:

Masking works on a per group basis. This means that whatever masking a nested group does has its result passed to the parent group as if it were a merged image.

getting me to believe that it's a bottom-up hierarchical type of thing.

And by extension of assumption, I understand this:

Clearing the Can mask does nothing, unless a layer below it adds to the Can o' Worms mask.
The Label group is added, and this is all you need to make the Can layer mask everything in the Label group.
The only part of the worm added to the mask is the Eyes that mask the Pupils...so the eyes are all you see when rendered. Clearing the mask does nothing here either.

But where you lost me was the body part:

By not masking the Body, that allows it to show without adding the Smile layer, and be masked by the Can acting on the Label group.

So by saying "Mask this layer" I'm saying: "I want this layer/layer group to be influenced by the upper-level masks, which, in this case, would be the Can layer mask." Is that correct?

But what I don't get is why I don't say mask the body layer. Does the Worm folder, having the "Mask this layer" option set already take care of that?
And then, why doesn't the smile show? Honestly I'm not sure if I've even asked this well. This is so confusing.

By the way, I followed your instructions, and then this happened:

Can o' Worms: Hide All
- Label: Hide All, Mask this layer
- - Worm: Hide All, Mask this layer
- - - Pupils: Mask this layer
- - - Eyes: Clear mask, then add this to mask
- - - Smile: Mask this layer
- - - Body: add to mask
- Can: add to mask

Now, between doing this and posting this message I got the text to appear by not masking it, and I tested it by moving it outside the bounds of the can, and this happened:

Even if I do get this working, what will still irk me is that the editor will not show the results correctly.

Now you say that clearing the mask does nothing, but let me remind you that in the editor window, and given my circumstance, "Add to Mask" and "Clear Mask and Add" meant a world of difference, and helped mold my "understanding" of masking! It's just that, when rendered, it showed something else, and broke my understanding completely.

[synthsin75]

Thanks for responding.
Ok. So I'm a really confused.

I get what you mean by the layer effect:

Masking works on a per group basis. This means that whatever masking a nested group does has its result passed to the parent group as if it were a merged image.

getting me to believe that it's a bottom-up hierarchical type of thing.

Yep.

And by extension of assumption, I understand this:

Clearing the Can mask does nothing, unless a layer below it adds to the Can o' Worms mask.
The Label group is added, and this is all you need to make the Can layer mask everything in the Label group.
The only part of the worm added to the mask is the Eyes that mask the Pupils...so the eyes are all you see when rendered. Clearing the mask does nothing here either.

But where you lost me was the body part:

By not masking the Body, that allows it to show without adding the Smile layer, and be masked by the Can acting on the Label group.

So by saying "Mask this layer" I'm saying: "I want this layer/layer group to be influenced by the upper-level masks, which, in this case, would be the Can layer mask." Is that correct?

Not influenced by the upper-level but influencing the upper-level masks. "Mask this layer" only works within the group of that layer. So a vector layer would effect other sibling layers, but its group layer would effect sibling layers in its own parent group. Influence only happens within a group. Then the result of that group is passed up to the next parent group where it is basically treated as a single layer. Upper-level layers never influence lower ones.

But what I don't get is why I don't say mask the body layer. Does the Worm folder, having the "Mask this layer" option set already take care of that?
And then, why doesn't the smile show? Honestly I'm not sure if I've even asked this well. This is so confusing.

Each Hide All group must add its own content. No higher-level layer can do that. So if you say mask the Body layer, you're saying this layer should be effected by any mask in the group, but you haven't created an add to mask layer lower than it in that group.

By the way, I followed your instructions, and then this happened:

Can o' Worms: Hide All
- Label: Hide All, Mask this layer
- - Worm: Hide All, Mask this layer
- - - Pupils: Mask this layer
- - - Eyes: Clear mask, then add this to mask
- - - Smile: Mask this layer
- - - Body: add to mask
- Can: add to mask

Oops. Yeah, the Label layer should be Show All.

Even if I do get this working, what will still irk me is that the editor will not show the results correctly.

That is the limitation of nested masking. You could get rid of the Label and Worm group layers (and just have their sublayers all in the Can o' Worms group) and do away with most the nested masking.

Now you say that clearing the mask does nothing, but let me remind you that in the editor window, and given my circumstance, "Add to Mask" and "Clear Mask and Add" meant a world of difference, and helped mold my "understanding" of masking! It's just that, when rendered, it showed something else, and broke my understanding completely.

The render is the actual result...not the workspace.
There are ways to make the workspace appear more similar to the actual render, as you've found out, but then you just have an original set to not render and a reference set to hide in editing view...both with different masking settings.
Much better to learn masking in Moho by checking the render.

[AirStyle]

OK, dude, after a while of messing with this, and taking a multi-hour break in between, I think I got it, and the editor and render windows have reached an agreement!



Can o' Worms: Hide All, Mask this Layer
--Label: Reveal All, Mask this Layer
----Worm: No masking, Mask this Layer
------Eyes: Hide All, Mask this Layer
--------Pupils: Mask this Layer
--------Eyes: Add to Mask
------Smile/Body: NA
------Squiggle O's Text: NA
--Can: Add to Mask
--Can Bottom: Don't mask this Layer

I took another look at the book to find out some fringe-free tips that I implemented, and stared at the functionality of each of the masking choices. They, coupled with you, really helped out! That is one sexy can of Squiggle O's!

Thanks for all your help!

[synthsin75]

Glad I could help. Many people find masking in Moho difficult to learn.

[AirStyle]

Thanks for telling me that.

[Greenlaw]

The render is the actual result...not the workspace.
There are ways to make the workspace appear more similar to the actual render, as you've found out...

@AirStyle, just a quick tip: I didn't read all of the above posts but I did a quick search on this page for 'GPU Acceleration' and didn't see it mentioned.

If you have GPU Acceleration enabled, you might want to disable it. In my experience, this option messes up masking previews in the workspace. To disable the feature in the workspace, go to Display Quality and check off the box for GPU Acceleration. If that doesn't help, try disabling the option in Preferences.

Note: this doesn't fix all masking preview errors (notably, nested masks,) but it does help with most common masking setups.

[AirStyle]

Thanks for the tip. Since you mention that, you don't think that would also interfere with a final animation export, by means of preventing something from moving, would it?

[Greenlaw]

GPU Acceleration has no affect on final rendering. It's only used to speed up workspace preview performance under certain conditions. It does have an affect on Preview Animation since that movie is created from the screen buffer, not the renderer.

Basically (from my understanding,) GPU Acceleration internally converts vectors to bitmaps and stores that image in the graphics card's cache, which can improve screen redraw. But the tradeoff is that there are many features in Moho that GPU Acceleration cannot display accurately (like masking.) Just to be clear, what's stored in the graphics memory is not used for final rendering, it's just for screen previews.

I keep GPU Acceleration disabled because we rely heavily on masking techniques and the animators need to be able to see the masking with reasonable accuracy during animation. Disabling GPU Acceleration doesn't fix all masking display issues but it makes it much more usable.

If your computer is fast enough, IMO, there's no real downside to leaving GPU Acceleration disabled. Naturally, if you're not concerned about the display limitations with GPU Acceleration, you may want to leave it enabled.

[Greenlaw]

Some more tips:

Regarding nested masks, these usually don't look right in the Workspace even with GPU Acceleration disabled, but in my experience, you can usually reproduce the effect without using nesting at all. Most of the time this involves breaking elements into separate but not nested groups and duplicating masks in multiple groups, or there may be a way to draw something complex without using masks. (Tools like Weld Crossings and Hide Edges can really help here. Victor's 'mask-free' masking trick described in other threads is also very useful.) It can be a bit more work to set up but if you can do it, it's worth it.

Sometimes I'll run into a situation where I absolutely must to nest the masks, and this messes up the workspace preview even with GPU Acceleration disabled. If it's something I can't live with, like the faulty preview really messes up my ability to animate effectively, I'll make a second version of the setup that may render poorly but looks more correct for previews. Then, I'll set this version to 'Don't Render This Layer', and set the original good render version to 'Hidden In Editing View'. This way, I'll have a usable preview AND a correctly rendering mask.

Sorry for the lack of examples...really busy today...but I think the above should make more sense after you've tried it yourself.