Masking problem with example

[fiziwig]

I just don't get masking at all!!! (grrrrr)

I'm trying to do a mask that lets object go through a hole in a wall. The wall is an image layer, and hole is laid in as a vector layer on top.

Heres an AS file that demonstrates it: http://fiziwig.com/mask_fall_in_hole.anme

This example uses a vector layer instead of an image layer for the background layer, but it demonstrates the problem.

If I set the mask layer to "add to mask but keep invisible" then it works perfectly in AS but the objects all become invisible when I render.

If I set the mask layer to "Clear mask then add layer" the mask itself shows up in AS, but not in the render. However, the objects that are supposed to go through the hole are still invisible in the rendered movie.

What do I need to do to get this to work when rendered?

--gary

[heyvern]

Masking is tricky but you came to the right place.

First off... you probably don't need masking for this particular example but I set up the file anyway. This is a complex example because it isn't just about masking. It involves two different things, masking is one but you also have to "split" the hole in half so that objects are visible above the hole and then covered as they pass through.

The objects have to be visible before falling into the hole then get covered up or masked as they pass the bottom edge. So when they fall in the bottom part of the hole covers them up. The mask in my example "hides" the bottom half of the hole. This could look funny if the objects pass by the sides at the "seam" but if you make sure that doesn't happen it looks fine. If you need objects to pass over the mask area you could turn visibility on and off to create that effect.

You could do this just as well by putting the "half hole" shape layer filled to match the background OVER the objects layer. the hole layer that is visible would be under everything else. It could even be part of the back ground layer. I included a duplicate bone layer with out masking to demonstrate this.

http://www.lowrestv.com/moho_stuff/foru ... v_mod.anme

The first thing I noticed was that the mask layer was set to 0 opacity... Good grief! I thought I was in bizzaro world! I couldn't figure out why that shape wouldn't show! A mask layer with 0 opacity still masks.. It really confused me. The setting in the masking options for "Add to mask but keep invisible" would be a better choice if that is what you want.

That mask shape was "backwards" or "inside out" sort of. You had the hole area "open" at the bottom and covered up around it. What i did in my version was to cover just the BOTTOM HALF of the hole. The idea of those objects bouncing into the hole is where masking is not necessary.

I also used another trick to create the mask shape to match the hole. I connected vectors to the outside points of the hole circle to create a shape. I then selected just the bottom half of the closed circle vector and the connected shapes below it to create an inverted shape for the bottom of the hole.

Here is the setup:

Bone layer masking is set to "Show All".
Objects layer is set to "Mask this layer"
Hole mask is set to "Subtract from mask".
Hole layer I set it to mask this layer. Doesn't matter since it shows up no matter what.

---------
If you read my sticky thread about masking in Tips and Techniques it kind of explains as well as possible how the "show all" and "hide all" masking options work. Show all is obvious. You subtract from show all to hide anything covered by the fill shape of that layer. If you ADD to show all nothing happens because ADDING SHOWS STUFF .

If you CLEAR THE MASK THEN ADD TO IT... you are basically REVERSING a show all mask option. Everything is "backwards" now. Show all is now hide all.

Hide all hides everything, you ADD to hide all to show something. If you SUBTRACT from hide all nothing happens because it is already hidden.
----------

-vern

[fiziwig]

Hey Vern,

Thanks a million for that help!

So "subtract" from mask really means to mask out what was subtracted. That just seems backwards to me. The English verb "to mask" means "to hide", so when you remove a part of the mask you should, according to standard English, be revealing what's behind, not hiding it. After all, you "subtract" material in the eye holes of a Halloween mask so you can see through what you've subtracted. And if you add more material to a physical mask you HIDE more, not reveal more. Yikes!

Maybe they should have used the term "hide" and "show" instead of "mask", "add", and "subtract". It must have been a programmer that decided on those terms.

The no mask option is what I've always used before, but in this case the background is an image, and cutting out a matching piece of the image seemed like a lot of trouble, what with getting the image pieces to line up perfectly and all.

Thanks again.

--gary

[human]

Maybe they should have used the term "hide" and "show" instead of "mask", "add", and "subtract". It must have been a programmer that decided on those terms.

Over the years I have noted that programmers (and other engineers) can be exceedingly careless when they choose terms.

But, since they have the skills needed to create products, what they choose gets set in stone. The users have to suffer under it afterwards.

One of the many great things about your instruction is the thoughtful care you take with terminology!

[heyvern]

Read my sticky!

viewtopic.php?t=9669

Part of why I wrote it was to cover some of the confusion about "masking".

Add, subtract, multiply... the whole thing has been around FOR AGES AND AGES. It refers to MATH not actual "hiding" or "showing".

A mask is either white (show all) or black (hide all). White is a larger number than black. When you add 2 numbers the number gets HIGHER (white more opaque). When you subtract the number gets lower (black more transparent).

If you HIDE ALL and ADD to it, you get a white hole on a black mask. If you SHOW ALL and SUBTRACT from it you get a black hole on a white mask.

Try to think of the mask in terms of black and white and anything with a fill is a value between black and white depending on where it is in the layer order and initial masking property of the group.

-vern

[fiziwig]

I see the logic of it. But it's still confusing. I think what I will do for my tutorials is something like this:

If you want an object to be hidden by the mask do "blah blah blah" and don't even bother trying to "understand" it. Think of it as a magical incantation written in an ancient language. You don't need to understand it, you just need to know how to chant the right chant when the situation calls for it.

And I think for my tutorial I will skip stationary abstract shapes and talk about "if you want the man to disappear as he walks through the door, do this." or "If you want the dog to disappear behind the tree do that."

--gary

[heyvern]

That sounds like a great idea.

I have only one issue with it... even though the terminology seems "odd" that is the terminology used for all programs that use alpha channels or masking. The use of math to create alpha channels for masking has been around for at least 20 years if not more... definately more.

Photoshop has the "add, subtract, multiply" layer ink effects. Same concepts. Photoshop still has "add, subtract, multiply" for alpha channel mixing even though it isn't even needed anymore. When you use the shift or or option key or option+shift with the selection tool in photoshop, it shows a plus, minus or multiplication sign, indicating addition, subtraction or multiplication from a selection (a selection is just a fancy mask or alpha channel).

I guess my point is you can use the simple explanations but at least include a reference to what it really means. If someone's only experience with masking is based on this they won't have the understanding of it to apply to other applications or different scenarios.

I have an idea I want use as a demonstration for my AS site (I am designing it now! WooHoo! I will definately link to yours!). Use layers filled with black and white to show what is really happening. I will create an animation using AS demonstrating in rotating "3D" all the layers and and masks and how they come together.

------

These are just my obsessive compulsive opinions and you can ignore them if you like. You're doing a GREAT JOB and providing a wonderful service to the new users... the last thing I want to do is discourage you.

-vern

[slowtiger]

Use layers filled with black and white to show what is really happening.

That's the point which is missing in the Help files right now, so it doesn't make clear that the concept derives from alpha channels.

[synthsin75]

After all, you "subtract" material in the eye holes of a Halloween mask so you can see through what you've subtracted. And if you add more material to a physical mask you HIDE more, not reveal more.

Just my two cents...A mask isn't a 'mask'. But I think I can work within this metaphor. In painting, a mask is more like a window for your work area. The 'hidden' part is the edge, outside of which you don't want your work to have an effect. I think the biggest hurdle is realizing that you are creating the Halloween mask yourself. Your vector (or image) layers are not 'behind' the mask. They are the mask. So when you 'add to mask' your not adding to a mask that covers your work, but adding your work to the mask itself.
Probably the simplest way to handle masking is whenever you see the word 'mask' think 'window'. So if I 'hide all', I'm hiding my whole window. Now anything I 'add' is adding to the size/shape of my window. 'Mask this layer' think 'window this layer'. Meaning it's 'in the window'. 'Don't mask this layer' means it's not in the window, and the window's edge will not obscure it. It's in the room with you, so to speak. 'Subtract from mask' means that it blocks the view through my window (subtract from window).
The 'reveal all' option is like saying the whole group is a window ( is not framed in ). All other options work the same as with 'hide all'.
The only thing to keep in mind is your layer order. Subtracting from the 'window' blocks every layer above it, in that group. We 'look' through our mask window from the bottom layer up.

Hope that helps.....

Vern, a 3d example is a great idea.

[heyvern]

I think the simplest explanation is this:

Black is hidden.
White is visible.
Adding creates white.
Subtracting creates black.
"Clearing" the mask makes it all black.

Group mask options:

"Hide All" is black to start
"Show All" is white to start.

Layer Mask options:

Add to mask --> layer shapes are white
Subtract from mask ---> layer shapes are black

If you can see the shapes on the layers you can picture in your head how they should mask each other.

Any other explanation is MORE confusing if you ask me. This is the REAL explanation. It is based on simple addition and subtraction but using values between black and white instead of numbers.

-vern

[synthsin75]

I'm just trying to translate it in a more accessible way. Using black and white only confuses the issue for anyone who doen't understand the idea of alpha values. Although if you present it in terms of black and white being the 'lighting' of the layer, it could still come across as more intuitive.

Hide all- nothing is 'lit up' (no window in group)
Reveal all- everything is 'lit up' ( all group is window)

Add to mask- light up ( add to window)
Subtract from mask- unlight (remove from window)

Please let me know if any newbies find this useful.

Vern, I know this isn't the REAL explaination, but understanding it in ANY terms beats not getting it at all. IMHO.

[heyvern]

I see what you mean but your "real world" examples are analogies of another analogy.

"white and black" makes sense. If there is white than you can see something. If there is black you can't.

The idea of trying to use concepts like "window with light" and "windows with out light" or "lights off" or "lights on" are just variations on the concept of black and white as values of a mask.

This is a graphic representation in black and white of the example in my masking thread. Each mask layer is in the color that it represents:
http://www.lowrestv.com/moho_stuff/foru ... t_mask.mov

Forget about what is in the past. Forget about the complex issues of alpha channels. Look at the simplest explanation, black and white. I can't imagine anything less complex. Start going on about cutting holes in paper or lights on and off just seems to overly complicate the issue.

Just an opinion.

-vern

[fiziwig]

I like your graphic. Finally I see what you've been talking about with black and white. Without seeing it in action it just didn't make any sense to me.

For the record, I've been a programmer since 1962 and I've done lot's of graphics programming with AND, OR, XOR, bit masks, so you'd think I'd get masks in AS. Just some kind of mental block, I guess.

--gary

[heyvern]

I updated the movie with text labels. I also linked to it in my thread about masking and made some other changes there as well.

This is one of those concepts that is notoriously difficult to grasp for a lot of people

-vern

[synthsin75]

Very nice example, Vern. That should help alot of people out with the masking concept.