How to Make Subtractive Mask

[FoundPebble]

I'm using Anime Studio 9.5 and I'm trying to make a subtractive mask where the mask is invisible and cuts a hole in the main shape. Is there a simple way to do this? I saw in Moho 13.5 there is an option that says mask outside bottom layer which created a subtraction mask for a mouth. There isn't anything like that in 9.5. I also saw one video on how to make a subtraction mask and it was a little complicated and confusing when I was doing it. It involved making the base shape and making another shape to cut out the base shape along with the mask and a couple of other layers. The reason I want to accomplish this is because I want to create a head that can have a head turn at the side view where the mouth cuts into the face rather than making switch layers and making the head the shape of the mouth if that makes sense.

[Lukas]

Set the group masking setting to “reveal all” and set the mask-layer as “subtract from mask”, and the to-be-masked layer set to “mask this layer”.

Both layers need to be in the group and the mask needs to be below the layer you want to apply it on.

See it as a black and white alpha channel that you’re building. ‘Reveal all’ starts with a white base as the alpha channel, ‘subtract’ adds a black shape, ‘add’ adds a white shape. ‘Hide all’ for the group starts the the alpha channel black instead of white. Considering this, note that moho renders the bottom layer first and stacks the layers subsequently (per group)

When a layer is being masked it will only render its pixels in the white parts, the black parts will be transparent. (And grey is semi transparent if you’re using opacity for certain layers)

[Greenlaw]

Reveal All should do what you want. This makes the mask layer cut out the layers above. By default, the mask will be set to Subtract From Mask (This Layer Will Be Invisible.)

An alternative trick is to place a path over another path and create a single shape from them. This produces a 'live-boolean' effect in shape, which can be useful for, say, cutting out eye holes in a face shape, all within the same layer. Sometimes this method can be easier and more efficient than using a mask.

[Daxel]

Set group masking to "reveal all," mask-layer to "subtract from mask," and to-be-masked layer to "mask this layer."

Both layers must be in the group, and the mask must be below the desired layer.

Imagine a black-and-white alpha channel. 'Reveal all' uses a white alpha channel,'subtract' adds a black shape, and 'add' adds a white shape. "Hide all" turns the alpha channel black for the group. Moho renders the lowest layer first, then stacks the layers (per group).

When a layer is masked, only white pixels are rendered; black pixels are transparent. (Grey is semi-transparent when using opacity for layers.)

Did you just copied Lukas response and changed a few words? You just said the exact same 4 things in the same order.
Is this a simulation?

[FoundPebble]

I tried it and it's still not working for me. I don't think I'm getting it. I have an odd situation. I have a head that is already in a mask. It is in a mask with a bandana so that the bandana is only seen on the head. I tried to use reveal all and use a substractive mask, which does make the layer I'm looking for to be invisible but it does not cut a hole through the face like in the videos. Not sure what I am doing wrong. I tried to take the head out of that mask so it is just the head, the mouth cutter/ invisible layer, and the inside of the mouth, but it didn't work. I think I'm confusing which layer is supposed to be masked. Can you tell me which layer; head, mouth cutter, and inside of the mouth, is supposed to be set to what setting in the mask settings, and which one is supposed to be on top of the other? Also can I put a mask inside of another like what I was originally trying to do?

[FoundPebble]

I tried it again on a new document and it worked, but I can't click exclude strokes for some reason. I saw it work on other people's videos. Is this due to the limitations of the version I have?

[Greenlaw]

Normally, Exclude Strokes is available when the mask layer is a vector layer with an active stroke, but I don't think it's available for that particular masking combination (using Subtractive.) I don't believe this is a bug, it's probably a technical limitation of Exclude Strokes.

That said, all Exclude Strokes does is it places the strokes on top of everything after masking. You can get the same effect by duplicating or referencing the layer, moving that layer to the top of the stack, and then disabling the fill. (Oh, and turn masking off for that layer of course.)

Exclude Strokes is a convenient shortcut when it works, but setting it up manually is more reliable and can give cleaner results.

[Daxel]

The switch layers don't allow to exclude strokes. I'm not sure why. Greenlaw's is a good solution.

[Greenlaw]

Hi,

I don't think it has to do with the Switch Layer...you can't have a mask directly inside the Switch because it can only display 1 layer by design, but you can use a mask and Exclude Strokes inside a Switch layer when they're part of their own group.

If you mean applying a Exclude Strokes on top of a Switch layer, this should work too. The Switch and the mask just need to be inside a shared parent group so you can enable the mask. Exclude Strokes should be available but with a couple of exceptions.

One exception is when Subtract from Mask (This layer will be invisible) is active. As mentioned earlier, I don't think this is a bug; it's probably a technical limitation of how Exclude Strokes works. There are probably other modes where it doesn't work but I'll need to check on that.

When Exclude Strokes doesn't work, using a duplicate or reference layer works fine. As mentioned, Exclude Strokes is a decent shortcut when it works, but using a modified copy on top tends to be more reliable.

I think the problem I've run into with Exclude Strokes mainly has to do with getting a proper Workspace display preview when using complicated masking techniques. Using a duplicate layer to get that 'strokes on top' effect tends behave more predictably, both for display and rendering.

Hope this clarifies a few things.

[Daxel]

I made a demo project to check if my notes were wrong or something changed on a later patch but no. Switch layers don't allow to exclude strokes, at least in this simple context and it would be nice if they would.

The viewport is also displaying the mask wrong but it's working fine when you render it.

https://1drv.ms/u/s!AiyKCw6Xey5Tgb5RBog ... Q?e=3uwLea

[lucasfranca]

I made a demo project to check if my notes were wrong or something changed on a later patch but no. Switch layers don't allow to exclude strokes, at least in this simple context and it would be nice if they would.

The viewport is also displaying the mask wrong but it's working fine when you render it.

https://1drv.ms/u/s!AiyKCw6Xey5Tgb5RBog ... Q?e=3uwLea

https://1drv.ms/u/s!Ah7UWMCgRFbGg89N-pv ... A?e=ERheyB

About the preview, turn off the GPU.

[synthsin75]

I made a demo project to check if my notes were wrong or something changed on a later patch but no. Switch layers don't allow to exclude strokes, at least in this simple context and it would be nice if they would.

The viewport is also displaying the mask wrong but it's working fine when you render it.

https://1drv.ms/u/s!AiyKCw6Xey5Tgb5RBog ... Q?e=3uwLea

https://1drv.ms/u/s!Ah7UWMCgRFbGg89N-pv ... A?e=ERheyB

About the preview, turn off the GPU.

Yep, that's the way to do it.

Daxel, it's not just switch layers. It's any grouping layer, including bone and group layers as well. There's just no way for a vector sublayer to tell the group-type layer to pass on its stroke.

[Daxel]

Oh that's right, I added that to my notes. It would be useful though.

[synthsin75]

Oh that's right, I added that to my notes. It would be useful though.

Yeah, it would save quite a bit of overhead over doing it the way Lucas demonstrated.

[Greenlaw]

Oh, did you guys mean a vector layer passing Exclude Stroke outside a group? Yeah, that doesn't work. Sorry, I misunderstood.

Nested masking (i.e., using groups to pass masking data to parent masking groups) can be unpredictable. Sometimes it's a GPU thing; I've seen situations where is looks fine on-screen and renders incorrectly and vice versa; other times it just breaks. I try to avoid nesting masking groups; there are more reliable ways to get the same effect. It might requires using more copies of layers but referencing can take care of that.