Masking Problem unwanted lines

Gabe · Post by **Gabe** » Sun Aug 27, 2006 2:45 am

Hello Mohusers,

I’ve an opposite problem form those I looked in this forum. In my case I don’t want to render the outline. Here I’m trying to eliminate the visible outline from the eyes from the mask in my render. I don’t know how to do it. Anybody can help me?

Here's the file:

URL text

Post by **Víctor Paredes** » Sun Aug 27, 2006 4:39 am

do you talk spanish?
i download the file and tried with different options combinations, but the lines always were there.
i made a new file with similar layers, and the same happened.

sorry, i can't help you.

Bones3D · Post by **Bones3D** » Sun Aug 27, 2006 5:32 am

This seems to be an anti-aliasing issue. If you perform a file>export animation and uncheck the "anti-aliasing" option, the undesired outlines vanish. Unfortunately, this will also make the resulting rendering look a bit more jagged.

Luckily, there is a way around this...

Go into your project settings and change the height/width to twice the size needed. Then, render your clip with anti-aliasing turned off. Finally, open up an external video editor (or image editor that supports batch processing), import the clip and export it at 50% the original size. The resulting file(s) will be the correct size and anti-aliased automatically due to resolution downsampling.

I do this regularly with certain 3D animated clips to improve the output quality... particularly on cel-shaded items, where the outlines are usually pretty ugly.

Post by **Víctor Paredes** » Sun Aug 27, 2006 5:36 am

wow, that's an ugly bug.
thanks for the tip, bones.

i'm now working with constraints bones, and when i put high values to spring or damping force, the file becomes crazy. and when i render, moho closes itself.

two ugly bugs in just one day is too much for me...

heyvern · Post by **heyvern** » Sun Aug 27, 2006 5:39 am

This would appear to be a bug.

No matter how big the eye mask is... there is always a thin line the exact color of the fill or stroke on the mask.

I suggest reporting this in the bug forum... I think it may already be there as I have heard this reported before.

Strange I haven't encountered this before myself... probably the lines are so thin I haven't noticed them and I don't do eye lids this way. I usually draw the lid shapes to fit the eye shape and control them with bones.

I mask the eyeballs inside the eyes... which is a bit different as well and wouldn't reveal this problem.

-vern

heyvern · Post by **heyvern** » Sun Aug 27, 2006 7:38 am

Well...

I kind of sort of... found a sort of solution...

I know how some of you might not like my more "complex" workarounds... so...

... this is one of those.

It involves using 2 extra bone layers with two sets of eyes and masking.
It also involves using those master/control bone scripts to move the lids since there are two of those each in a separate bone layer.

The basic concept... in one bone layer group the eyelids mask the eyes (no lines... no lid line on the eye though).

The second bone layer has the lid line (the line at the base of the lid so it matches the outlines of the eyes.)

Everything is controlled by bones... very very easy to animate... works like a charm... but the set up is bit tricky. But once that is done... it's over with.

Here is a short animation and a jpg:

http://www.lowrestv.com/moho_stuff/eye_ ... emoval.mov

If anyone is really interested in this I can post the file... but... I don't want to hear a lot of Moho bashing and how this stupid bug should be fixed... and my solution is too hard and overly complicated .... blah blah balh... that's already a given in this situation... and I've heard it before and I agree this is a bug...

This is just my personal solution to overcome this limitation that doesn't involve rendering at twice up without antialising and then reducing the file size in another application... steps that may or may not be considered equally or more difficult or overly complex...

(the aliased render resize is a pretty darn good solution... if you have the tools to downsize of course.)

-vern

Bones3D · Post by **Bones3D** » Sun Aug 27, 2006 9:04 am

HeyVern,

That's a pretty clever idea, and I'm kind of kicking myself for not think of that approach to begin with. I think I saw a similar approach a while back, where someone was demonstrating how to make a 2D head move like a 3D object, by using opaque shapes to mask out certain elements as the head turned.

myles · Post by **myles** » Sun Aug 27, 2006 12:17 pm

Assuming the eyelid colour is the same colour as the skin colour, how about a somewhat simpler approach?

Duplicate the lid layer, put it above the existing eye layers but in the same group, and - here's the trick - remove the outline and don't mask it against the eyeball shape.
You may have to move it up very slightly to avoid it masking any of the edge of the eyelid.

Control it with the same bone setup that controls the masked eyelids.

The drawback is you have to be careful of where the eyelid edges go - they have to stay inside the face but outside the eyeballs.

Regards, Myles.

heyvern · Post by **heyvern** » Sun Aug 27, 2006 2:49 pm

Sorry... no simplification is possible.

I tried every combination.

When you have a shape set to "Add" (still visible) you will ALWAYS get that thin line around it. The line is whatever color the mask shape is filled with or stroked... whatever color is "at the edge" of the shape.

I assume that line is caused by the anti-aliasing of the edges. Some of the shape bleeds through the mask... because that shape is visible... even though it should be completely masked.

If you set that shape to Add but be invisible... then... no line... but then the eyes are "invisible" all you see is the line at the bottom of the lid shape.

So... to get the eyes back, and still be masked by the lids... create ANOTHER group (set to Reveal all in the masking) the lids mask the eyes... THE LIDS LAYER IS SET TO SUBTRACT (so you can use the same lid shape as the other layer).

Sounds... complicated... but not really...

uh... er... no it is complicated...

(I added the eyeballs on another layer so they would be masked by the eyeshape).

I had to look at the darn file about ten times while writing this just to remember how I did it.

-vern

myles · Post by **myles** » Mon Aug 28, 2006 12:28 am

No, I meant not masking the duplicate eyelid at all. If the skin is a different colour the cheat is immediately obvious as a rectangle, but you can get away with it if the skin is the same colour.

However, you have to be careful about the edge of the eyelids. Vern, your solution looks like it might require less care by the animator after rigging, which may be a better way to go.

Here's the setup I was thinking of:

Zipped Moho file here.

Regards, Myles.

heyvern · Post by **heyvern** » Mon Aug 28, 2006 3:05 am

ooooohhhhhhhhh!!!

I get it now... yes... that is MUCH simpler.

Good grief yes...

I... uh... I guess i didn't try ALL the combinations... <sigh>. I would have suggested that one if I had come up with it.

The assumption is... that the lids are the same color as the face... in this case anyway.

This is much easier... no need for layer scripts... multiple bone layers...

And NOT to difficult to animate... both lid layers (or the points) can be moved by one bone!

I really enjoy these little excursions... they are fun.

-vern

heyvern · Post by **heyvern** » Mon Aug 28, 2006 3:20 am

If you use point binding on a bone for the lids... and only bind the bottom points of the lid "box"....

You don't have to worry too much about the shape of that box ever extending too far.

I set it up with the "visible" lid box very tight to the eyes at the top and sides. Only the bottom two points move... so it works very nicely.

If you wanted a different color for the lids... then the ORIGINAL problem vanishes... go back to the simple first version.

That thin line is so... thin.... any color in the lids area practically hides it. It is so unnoticable as to not be an issue.

(the lids look odd in a different color without the stroke... so you might want the stroke on the lids... thus also eliminating the thin line problem)

-vern

Fazek · Post by **Fazek** » Mon Aug 28, 2006 7:49 am

The thin black outline comes from the edge of the eyeball, because the masking of the lid is partially transparent at the edge. Because of the anti-aliasing. Sometimes it comes out in other programs too and in this case I usually make the masking separately to the outline and to the paint. I think an option would be useful in the layer settings to set the masking to the fill and to the outline shapes separately.

myles · Post by **myles** » Mon Aug 28, 2006 8:11 am

Fazek wrote:I think an option would be useful in the layer settings to set the masking to the fill and to the outline shapes separately.

Yes! That would be very useful for eye and mouth shapes at the very least, and probably a lot more.

Regards, Myles.

Gabe · Post by **Gabe** » Mon Aug 28, 2006 12:36 pm

I think we’re trying to solve in a tricky way one real masking problem. Fazek has a lot of reason if we could mask separately the fill to the outline, or both at same time because to the anti-aliasing. I’m working in one very complex character and I would plainly do masking without worrying about adding more “trick fixing layers” to do a simple task of mask.

By the way, the Myles’s solution seems to be the clever trick. Thanks Myles!

Hey Selgin, gracias por tratar de ayudar… Por cierto, he visto que somos pocos los que hablamos español en este foro, deberíamos estar más en contacto.